Re: How to check validity or completeness of created checkpoint/savepoint

The existence of a _metadata file is a good indicator that Flink has finished writing the checkpoint/savepoint; IIRC we use this in our tests. I'm not aware of any other mechanism. On 06.03.2019 10:21, Parth Sarathy wrote: Hi, I am running flink 1.7.2 and working on resuming a job from a

Re: RMQSource synchronous message ack

The acknowledgement has to be synchronous since Flink assume that after notifyCheckpointComplete() all data has been persisted to external systems. For example, if record 1 to 100 were passed to the sink and a checkpoint occurs and completed, on restart Flink would continue with record 101.

Re: flink-io FileNotFoundException

Is there some periodic cleanup job running on your system for /tmp? On 11.03.2019 10:38, Alexander Smirnov wrote: Hi everybody, I am using Flink 1.4.2 and periodically my job goes down with the following exception in logs. Relaunching the job does not help, only restarting the whole cluster.

Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API

Please enable debug logging, re-submit the job, check the JobManager logs and report back the logged exception. On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote: Hi all, As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading

Re: Challenges using Flink REST API

You should get the full stacktrace if you upgrade to 1.7.2 . On 13.03.2019 09:55, Wouter Zorgdrager wrote: Hey all! I'm looking for some advice on the following; I'm working on an abstraction on top of Apache Flink to 'pipeline' Flink applications using Kafka. For deployment this means that

Re: Challenges using Flink REST API

rrors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] } Regards, Wouter Zorgdrager Op wo 13 mrt. 2019 om 10:42 schreef Chesnay Schepler mailto:ches...@apache.org>>: You should get the full stacktrace if you upgrade to 1.7.2 . On 13.03.2019 09:55, Wouter Zorg

Re: Challenges using Flink REST API

/browse/FLINK-11423 [2]: _https://jira.apache.org/jira/browse/FLINK-7715_ Op wo 13 mrt. 2019 om 12:18 schreef Chesnay Schepler mailto:ches...@apache.org>>: Can you give me the stacktrace that is logged in the JobManager logs? On 13.03.2019 10:57, Wouter Zorgdrager wrote: Hi C

Re: REST API question GET /jars/:jarid/plan

I've heard of cases where client libraries are automatically changing the HTTP method when provided with a body. To figure out what exactly is received by Flink, enable TRACE logging, try again and look for logging messages from "org.apache.flink.runtime.rest.handler.router.RouterHandler"

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

There is no (accurate) reference of included dependencies for the flink-shaded-hadoop uber jars. The contained NOTICE file is a good starting point, but for the time being we're using a generalized version that we apply to all hadoop versions (so some things may be missing). I believe for

Re: Status.JVM.Memory.Heap.Used metric shows only a few megabytes but profiling shows gigabytes (as expected)

Known issue, fixed in 1.7.3/1.8.0: https://issues.apache.org/jira/browse/FLINK-11183 On 19.03.2019 15:03, gerardg wrote: Hi, Before Flink 1.7.0 we were getting correct values in Status.JVM.Memory.Heap.Used metric. Since updating we just see a constant small value (just a few megabytes), did

Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API

uot;} Content-Type: application/json *From:*Chesnay Schepler *Sent:* Πέμπτη, 14 Μαρτίου 2019 2:24 μμ *To:* Papadopoulos, Konstantinos ; user@flink.apache.org *Subject:* Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API Please enable debu

Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API

) at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:30) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) ... 20 more *From:*Chesnay Schepler *Sent:* Παρασκευή, 15 Μαρτίου 2019 10:20 πμ *To:* Papadopoulos, Konstantinos ; user@flink.apache.org *Subject

Re: Case class field limit

It is likely that the documentation is outdated. Could open a JIRA for updating the documentation? On 22/03/2019 10:12, Wouter Zorgdrager wrote: Hey all, Since Scala 2.11 the amount of fields in a case class isn't restricted to 22 anymore [1]. I was wondering if Flink still uses this limit

Re: EOF on scraping flink metrics

Since you're using Prometheus I would recommend setting up a PrometheusReporter as described in the metrics documentation and scrape each JM/TM individually. Scraping through the REST API is more expensive and you loose out on a lot of features. The REST API calls are primarily aimed at the

Re: Flink Job and Watermarking

Have you considered using the metric system to access the current watermarks for each operator? (see https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#io) On 08.02.2019 03:19, Kaustubh Rudrawar wrote: Hi, I'm writing a job that wants to make an HTTP request once

Re: stream of large objects

Whether a LargeMessage is serialized depends on how the job is structured. For example, if you were to only apply map/filter functions after the aggregation it is likely they wouldn't be serialized. If you were to apply another keyBy they will be serialized again. When you say "small size"

Re: Help with a stream processing use case

I'll need someone else to chime in here for a definitive answer (cc'd Gordon), so I'm really just guessing here. For the partitioning it looks like you can use a custom partitioner, see FlinkKinesisProducer#setCustomPartitioner. Have you looked at the KinesisSerializationSchema described in

Re: Couldn't figure out - How to do this in Flink? - Pls assist with suggestions

You should be able to use a KeyedProcessFunction for that. Find matching elements via keyBy() on the first field. Aggregate into ValueState, send alert if necessary. Upon

Re: stream of large objects

trings of length 500 to 1000. Is there a concept of “global” state in flink? Is it possible to keep these lists in global state and only pass the list reference (by name?) in the LargeMessage? *From: *Chesnay Schepler *Date: *Friday, February 8, 2019 at 8:45 AM *To: *"Aggarwal, Aja

Re: Flink Standalone cluster - logging problem

What exactly are you expecting to happen? On 08.02.2019 15:06, simpleusr wrote: We are using standalone cluster and submittig jobs through command line client. As stated in https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html , we are editing log4j-cli.properties but

Re: Reduce one event under multiple keys

This sounds reasonable to me. I'm a bit confused by this question: "/Additionally, I am (naïevely) hoping that if a window has no events for a particular key, the memory/storage costs are zero for that key./" Are you asking whether a key that was received in window X (as part of an event)

Re: Can an Aggregate the key from a WindowedStream.aggregate()

There are also versions of WindowedStream#aggregate that accept an additional WindowFunction/ProcessWindowFunction, which do have access to the key via apply()/process() respectively. These functions are called post aggregation. On 08.02.2019 18:24, stephen.alan.conno...@gmail.com wrote: If

Re: Running single Flink job in a job cluster, problem starting JobManager

I'm afraid we haven't had much experience with Spring Boot Flink applications. It is indeed strange that the job ends up using a StreamPlanEnvironment. As a debugging step I would look into all calls to ExecutionEnviroment#initializeContextEnvironment(). This is how specific execution

Re: Flink Standalone cluster - dumps

1) Setting the slot size to 1 can be used as a work-around. I'm not aware of another solution for standalone clusters. In the YARN/Kubernetes world we support the notion of a "job cluster", which is started and run only for a single job, but this isn't supported in standalone mode. 2) None

Re: Test FileUtilsTest.testDeleteDirectory failed when building Flink

I ran into a similar issue when i looked at other CI solutions; imo we shouldn't rely on the result of setWritable but instead actually verify whether the forbidden operation (i.e. creating/writing files) throws an error. Back then also created a JIRA:

Re: Change Flink checkpoint configuration at runtime

You cannot change the checkpointing configuration at runtime. You should be able to resume the job from the last checkpoint. On 22.01.2019 19:39, knur wrote: I'm running a streaming job that uses the following config: checkpointInterval = 5 mins minPauseBetweenCheckpoints = 2 mins

Re: Back pressure within a operator chain

The behavior should be identical regardless of whether the are chained or not. On 23.01.2019 09:11, Paul Lam wrote: Hi, I would like to know if back pressure applies to operators in the same operator chain? The background is that I have a simple streaming job that consumes data from

Re: Is there a way to get all flink build-in SQL functions

Beyond the documentation I don't believe there to be a mechanism for listing all built-in functions. On 23.01.2019 04:30, yinhua.dai wrote: I would like to put this list to the our self service flink SQL web UI.

Re: Flink CEP : Doesn't generate output

Can you provide us a self-contained reproducing example? (preferably as elementary as possible) On 22.01.2019 18:58, dhanuka ranasinghe wrote: Hi All, I have used Flink CEP to filter some events and generate some alerts based on certain conditions. But unfortunately doesn't print any

Re: Version "Unknown" - Flink 1.7.0

have you compiled Flink yourself? Could you check whether the flink-dist jar contains a ".version.properties" file in the root directory? On 12/04/2019 03:42, Vishal Santoshi wrote: Hello ZILI, I run flink from the distribution as from

Re: Flink add source with Scala

There is no separate Scala SourceFunction interface or similar convenience interfaces, so you'll have to work against the Java version. On 12/04/2019 09:07, hai wrote: Hello: Is there a example or best practise code of flink’s source of Scala language, I found one example on official

Re: Event Trigger in Flink

Sounds like your describing a source function that subscribes to couch db updates. You'd usually implement this as a Co(Flat)MapFunction that has 2 inputs, 1 from kafka and one from couch db, which stores the processing parameters in state. There's no built-in way to subscribe to couch db

Re: FlinkKafkaProducer010: why is checkErroneous() at the beginning of the invoke() method

It is called at the top since exceptions can occur asynchronously when invoke() already exited. In this case the only place you can fail is if the next record is being processed. On 12/04/2019 11:00, Kumar Bolar, Harshith wrote: Hi all, I had a requirement to handle Kafka producer

Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API

] at java.lang.Thread.run(Unknown Source) [?:1.8.0_201] *From:*Chesnay Schepler *Sent:* Παρασκευή, 15 Μαρτίου 2019 10:56 πμ *To:* Papadopoulos, Konstantinos ; user@flink.apache.org *Subject:* Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API Please

Re: Job recovers from an old dangling CheckPoint in case of Job Cluster based Flink pipeline

Can you provide us the jobmanager logs? After the first restart the JM should have started deleting older checkpoints as new ones were created. After the second restart the JM should have recovered all 10 checkpoints, start from the latest, and start pruning old ones as new ones were created.

Re: Building Flink distribution with Scala2.12

https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#scala-versions On 29/05/2019 15:29, Boris Lublinsky wrote: Thanks Martijn, this was it. It would be nice to have this in documentation. Boris Lublinsky FDP Architect boris.lublin...@lightbend.com

Re: Minicluster missing in scala runtime in version 1.8 source code

It's still there: https://github.com/apache/flink/tree/release-1.8/flink-runtime/src/main/java/org/apache/flink/runtime/minicluster Please provide more details on what you believe to be missing. On 27/05/2019 01:02, Ravi Pullareddy wrote: Hi There I notice mini cluster folder missing under

Re: Minicluster missing in scala runtime in version 1.8 source code

: MiniCluster is there in java code but not in Scala. Zeppelin flink interpreter looks for LocalFlinkMiniCluster class located under scala code org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster *From:*Chesnay Schepler mailto:ches...@apache.org>> *Sent:* Monday, 27 May 2019 6:58 PM *To:

Re: [DISCUSS] Deprecate previous Python APIs

I would just remove them. As you said, there are very limited as to what features they support, and haven't been under active development for several releases. Existing users (if there even are any) could continue to use older version against newer releases. It's is slightly more involved

Re: [External] checkpoint metadata not in configured directory state.checkpoints.dir

The _metadata is always stored in the same directory as the checkpoint data. As outlined here "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ be overwritten with a

Re: Simple stateful polling source

for redistributable unit of state" but this is not very clear to me.. Best, Flavio On Wed, Jun 19, 2019 at 12:40 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: This looks fine to me. What exactly were you worried about? On 19/06/2019 12:33, Flavio Pompermaier wrote:

Re: Simple stateful polling source

a RichParallelSourceFunction and a RichSourceFunction. Is this last (RichSourceFunction) implicitly using parallelism = 1? On Wed, Jun 19, 2019 at 2:25 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: It returns a list of states so that state can be re-distributed if the pa

Re: Simple stateful polling source

This looks fine to me. What exactly were you worried about? On 19/06/2019 12:33, Flavio Pompermaier wrote: Hi to all, in my use case I have to ingest data from a rest service, where I periodically poll the data (of course a queue would be a better choice but this doesn't depend on me). So

Re: Updated Flink Documentation and tutorials

job, much like Hadoop's Application Master for each job. Please let me know which one is correct. Pankaj On Thu, Jun 20, 2019, 4:54 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: What makes you believe that they are out-dated? On 19/06/2019 19

Re: Updated Flink Documentation and tutorials

What makes you believe that they are out-dated? On 19/06/2019 19:17, Pankaj Chand wrote: Hello, Please let me know how to get the updated documentation and tutorials of Apache Flink. The stable v1.8 and v1.9-snapshot release of the documentation seems to be outdated. Thanks! Pankaj

Re: [DISCUSS] Deprecate previous Python APIs

I have created a JIRA and PR for removing the Python APIs. On 11/06/2019 15:30, Stephan Ewen wrote: Hi all! I would suggest to deprecating the existing python APIs for DataSet and DataStream API

Re: Apache Flink - Question about metric registry and reporter and context information

You cannot access context information in the metric reporter itself /in a nice way/. You can wait for the first metric to be registered, and then extract arbitrary metric variables from the metric group. On 15/06/2019 19:31, Yun Tang wrote: Hi 1) Yes, the metrics reporter was instantiated

Re: Maybe a flink bug. Job keeps in FAILING state

@Till have you see something like this before? Despite all source tasks reaching a terminal state on a TM (FAILED) it does not send updates to the JM for all of them, but only a single one. On 18/06/2019 12:14, Joshua Fan wrote: Hi All, There is a topology of 3 operator, such as, source,

Re: Side output in ProcessFunction.onTimer

ProcessFunction#onTimer provides an OnTimerContext parameter which allows you to use side-outputs. On 18/06/2019 17:41, Frank Wilson wrote: Hi, Is there a way to make side outputs in an onTimer callback in ProcessFunction? I want to side output events that belong to a session that was

Re: Maybe a flink bug. Job keeps in FAILING state

-- From:Joshua Fan Send Time:2019年6月20日(星期四) 11:55 To:zhijiang Cc:user ; Till Rohrmann ; Chesnay Schepler Subject:Re: Maybe a flink bug. Job keeps in FAILING state zhijiang I did not capture the job ui

Re: [SURVEY] Usage of flink-ml and [DISCUSS] Delete flink-ml

I believe we can remove it regardless since users could just use the 1.8 version against future releases. Generally speaking, any library/connector that is no longer actively developed can be removed from the project as existing users can always rely on previous versions, which should

Re: Flink not giving full reason as to why job submission failed

Please open a new JIRA. FLINK-11902 modified the REST API to no longer hide the exception, but the WebUI isn't handling the error response properly (it only reads and displays part of it). On 20/05/2019 16:24, Wouter Zorgdrager wrote: Hi Harshith, This is indeed an issue not resolved in 1.8.

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

This issue is another case where we have problems figuring out the boundaries and responsibilities between the ExecutionEnvironments and the ClusterClient. I believe we should figure this out first, and decide whether the ClusterClient (or anything based on it) should be made public to

Re: User Interface not showing the actual count received and produced

Flink currently does not measure incoming records for sources and outgoing records for sink, see https://issues.apache.org/jira/browse/FLINK-7286. On 15/05/2019 19:43, PoolakkalMukkath, Shakir wrote: Hi Flink team, I am developing a flow which uses ·FlinkKafkaConsumer010 to consume message

Re: Flink with HBase

Unless I'm blind Flink does not provide an HBase sink, so it depends on the implementation you're using. On 15/05/2019 20:17, Nikhil Goyal wrote: Hi, Does flink support exactly once processing using HBase as sink? I am not able to find any documentation supporting the same. Thanks Nikhil

Re: Passing a custom SourceContext to a SourceFunction

You cannot control what kind of SourceContext is passed into your function. What are you trying to achieve? On 15/05/2019 09:30, Debasish Ghosh wrote: Hi - I have a custom SourceFunction .. class MySourceFunction[T](data: Seq[T]) extends SourceFunction[T] { def run(ctx: SourceContext[T]):

Re: writeAsFormattedText sets only Unix/Linux line endings

The line-ending is hard-coded into the TextOutputFormat. You will have to either extend this class and override #writeRecord(), or convert your POJOs to a Tuple and use the CsvOutputFormat, which supports setting the line delimiter (called recordDelimiter). On 09/05/2019 08:32, Papadopoulos,

Re: Can't build Flink for Scala 2.12

You are correct, that is a typo. Very well done for spotting it, will fix it right away. We can conclude that the current SNAPSHOT version does not build with scala 2.12 at the moment; hopefully there aren't too many issues. On 01/05/2019 08:05, Visser, M.J.H. (Martijn) wrote: In the

Re: Unable to build flink from source

You're likely using Java9+, but 1.3.3 only supports Java 8 (and maybe still 7). On 06/05/2019 03:20, syed wrote: Hi I am trying to build flink 1.3.3 from source using IntelliJ IDEA Ultimate 2019.1 IDE. When I build the project, I am receiving the following error *java package sun.misc does

Re: ClassNotFoundException on remote cluster

How are you packaging the jar that you submit? Specifically, are you ensuring that all your classes are actually contained within? On 02/05/2019 13:38, Abhishek Jain wrote: Hi, I'm running into ClassNotFoundException only when I run my application on a standalone cluster (using flink cli). If

Re: Type Erasure - Usage of class Tuple as a type is not allowed. Use a concrete subclass (e.g. Tuple1, Tuple2, etc.

I'm not sure what you're asking. If you have a Deserialization schema that convert the data into a Map you're done as I understand it, what do you believe to be missing? If, for a given job, the number/types of fields are fixed you could look into using Row. On 01/05/2019 22:40, Vijay

Re: configuration of standalone cluster

Which java version are you using? On 01/05/2019 21:31, Günter Hipler wrote: Hi, For the first time I'm trying to set up a standalone cluster. My current configuration 4 server (1 jobmanger and 3 taskmanager) a) starting the cluster swissbib@sb-ust1:/swissbib_index/apps/flink/bin$

Re: Can't build Flink for Scala 2.12

You can monitor https://issues.apache.org/jira/browse/FLINK-12392 for the compile issue. On 01/05/2019 22:05, Chesnay Schepler wrote: You are correct, that is a typo. Very well done for spotting it, will fix it right away. We can conclude that the current SNAPSHOT version does not build

Re: configuration of standalone cluster

OpenJDK 64-Bit Server VM (build 11.0.2+9-Ubuntu-3ubuntu118.04.3, mixed mode, sharing) swissbib@sb-ust1:~$ Is version 8 more appropriate? Günter On 02.05.19 13:48, Chesnay Schepler wrote: Which java version are you using? On 01/05/2019 21:31, Günter Hipler wrote: Hi, For the first time

Re: ClassNotFoundException on remote cluster

, Abhishek Jain <mailto:abhijai...@gmail.com>> wrote: This is a spring boot app that I've packaged using maven (Apache Maven 3.3.9). I've verified the class is present in the jar as well. On Thu, 2 May 2019 at 17:25, Chesnay Schepler mailto:ches...@apache.org>> wrote:

Re: UnsupportedOperationException from org.apache.flink.shaded.asm6.org.objectweb.asm.ClassVisitor.visitNestHostExperimental using Java 11

Flink only supports Java 8. On 04/07/2019 15:34, Rauch, Jochen wrote: Hi all, I have implemented following code snippet with Apache Flink 1.8: flinkConfiguration.getEnvironment().readTextFile(outputFile.getAbsolutePath(), "ISO-8859-1") .flatMap(new FlatMapFunctionObject>, Integer>>() {  

Re: Apache Flink - How to find the number of window instances in an application

This is unfortunately not possible. On 04/07/2019 19:40, M Singh wrote: Hi: I wanted to find out if there is a metric to find out the the number of global or non-global window instances in a Flink application. Thanks Mans

Re: Apache Flink - Are counters reliable and accurate ?

1) None that I'm aware of. 2) You should use counters. 3) No, counters are not checkpointed, but you could store the value in state yourself. 4) None that I'm aware of that doesn't require modifications to the application logic. How long does your job run for, and how do you access metrics?

Re: Apache Flink - Are counters reliable and accurate ?

your test. On 27/06/2019 22:41, M Singh wrote: Hi Chesnay: Thanks for your response. My job runs for a few minutes and i've tried setting the reporter interval to 1 second. I will try the counter on a longer running job. Thanks again. On Thursday, June 27, 2019, 11:46:17 AM EDT, Chesnay

Re: Job Startup Arguments

The passed job arguments can not be queried via the REST API. When submitting jobs through the CLI these parameters never arrive at the cluster; in case of REST API submission they are immediately discarded after the submission has finished. On 25/04/2019 12:25, Dawid Wysakowicz wrote: Hi

Re: metric does not display on web

"No metrics available" indicates that either a) metrics have not been queried yet or b) metrics can not be transferred from the TaskManagers to the JobManager. Regarding the first option, how long have you waited for metrics to show up? It may take a bit for metrics to be available (around 10

Re: get custom gauge metric from WebMonitorEndpoint

You should be able to get the value of your custom metric. You'll have to set the log level to DEBUG and scan the logs for metric-related errors. On 23/04/2019 22:25, Georgi Stoyanov wrote: I've got custom metric -> || And I'm using them as suggested in the documentation -> || |

Re: Converting Metrics from a Reporter to a Custom Events mapping

You can configure multiple reporters, so just configured both the reporter that the app users want and your own which does the transformation and sending to kinesis. On 16/07/2019 09:37, Haibo Sun wrote: Hi, Vijay Or can you implement a Reporter that transforms the metrics and sends them

Re: Build Flink against a vendor specific Hadoop version

Yes, please open a JIRA ticket. On 30/08/2019 18:46, Elise RAMÉ wrote: Thank you all ! Classpath option works for me and is easier so I’ll do this way. About flink-shaded and vendor-repos, would it be helpful if I describe this issue in a Jira ticket ? Le 30 août 2019 à 11:43, Chesnay

Re: Configure Prometheus Exporter

This is not possible. Are you trying to limit which metrics are exposed? On 07/08/2019 06:52, Chaoran Yu wrote: Hello guys,    Does anyone know if the Prometheus metrics exported via the JMX reporter or the Prometheus reporter can be configured using a YAML file similar to this one

Re: Configure Prometheus Exporter

I’m trying to do. I think Flink by default exports all metrics. Is there anything else I can do to achieve this goal? On Aug 7, 2019, at 03:58, Chesnay Schepler <mailto:ches...@apache.org>> wrote: This is not possible. Are you trying to limit which metrics are exposed? On 07/08/2

Re: Accessing variables build within a map/flatmap function within a DS

Note that this will only work when running the the application in the IDE; specifically it will not work when running on an actual cluster, since your function isn't executed on the same machine as your (presumably) main[] function. We can give you better advice if you tell us what exactly

Re: How to handle JDBC connections in a topology

Note that in order for the static class approach to work you have to ensure that the class is loaded by the parent classloader, either by placing the class in /lib or configuring `classloader.parent-first-patterns-additional` to pick up this particular class. On 24/07/2019 10:24, Haibo Sun

Re: Does Flink support raw generic types in a merged stream?

Have you looked at org.apache.flink.types.Either? If you'd wrap all elements in both streams before the union you should be able to join them properly. On 17/07/2019 14:18, John Tipper wrote: Hi All, Can I union/join 2 streams containing generic classes, where each stream has a different

Re: Extending REST API with new endpoints

There's no built-in way to extend the REST API. You will have to create a fork and either extend the DIspatcherRestEndpoint (or parent classes), or implement another WebMonitorExtension and modify the DispatcherRestEndpoint to load that one as well. On 23/07/2019 15:51, Oytun Tez wrote:

Re: Flink sent/received bytes related metrics: how are they calculated?

Which version of Flink are you using? There were some issues at some point about double-counting. On 14/06/2019 09:49, Andrea Spina wrote: Dear Community, I'd like to ask for some details about bytes related metrics in Flink. Precisely, I'm looking at *bytes sent* and *bytes received

Re: Flink sent/received bytes related metrics: how are they calculated?

:08 Chesnay Schepler mailto:ches...@apache.org>> ha scritto: Which version of Flink are you using? There were some issues at some point about double-counting. On 14/06/2019 09:49, Andrea Spina wrote: Dear Community, I'd like to ask for some details about bytes r

Re: Apache Flink - Gauge implementation

This can't be implemented as there's no guarantee that getValue is only called once. Why do you want to reset the count? If you're interested in rates, why aren't you using a meter? On 10/07/2019 09:37, Xintong Song wrote: Hi Singh, Could your problem be solved by simply record the

Re: How can I get the backpressure signals inside my function or operator?

I don't think there is a truly sane way to do this. I could envision a separate application triggering samples via the REST API, writing the results into kafka which your operator can read. This is probably the most reasonable solution I can come up with. Any attempt at accessing the

Re: Sending custom statsd tags

Not possible, you'll have to extend the StatsDReporter yourself to add arbitrary tags. On 30/10/2019 12:52, Prakhar Mathur wrote: Hi, We are running Flink 1.6.2. We are using flink-metrics-statsd jar in order to send metrics to telegraf. In order to send custom metrics, we are using

Re: Completed job wasn't saved to archive

If the archiving fails there should be some log message, like "Failed to archive job" or "Could not archive completed job..." . If nothing of the sort is logged my first instinct would be that the operation is being slowed down, _a lot_. Where are you archiving them to? Could it be the write

Re: Building with Hadoop 3

There's no JIRA and no one actively working on it. I'm not aware of any investigations on the matter; hence the first step would be to just try it out. A flink-shaded artifact isn't a hard requirement; Flink will work with any 2.X hadoop distribution (provided that there aren't any dependency

Re: Flink 'Job Cluster' mode Ui Access

by the job. Best regards, Jatin On Fri, Nov 29, 2019 at 4:18 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: To clarify, you ran "mvn package -pl flink-dist -am" to build Fink? If so, could you run that again and provide us with the maven output? |

Re: Change Flink binding address in local mode

At this point I would suggest to file a ticket; these are the options that _should_ control the behavior but apparently aren't in all cases. On 08/12/2019 12:23, Andrea Cardaci wrote: Hi, Flink (or some of its services) listens on three random TCP ports during the local[1] execution, e.g.,

Re: Flink 'Job Cluster' mode Ui Access

ord. Can i raise this as bug ? Best Regards, Jatin On Thu, Dec 5, 2019 at 8:59 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: Ok, it's good to know that the WebUI files are there. Please enable DEBUG l

Re: Flink 'Job Cluster' mode Ui Access

. Regards, Chesnay On 13/12/2019 09:58, Jatin Banger wrote: Sure, here it is. Job Manager Logs with logging level as DEBUG On Wed, Dec 11, 2019 at 3:14 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Would it be possible for you to provide us with full debug log file?

Re: Sample Code for querying Flink's default metrics

Yes, when a cluster was started it takes a few seconds for (any) metrics to be available. On 12/12/2019 11:36, Pankaj Chand wrote: Hi Vino, Thank you for the links regarding backpressure! I am currently using code to get metrics by calling REST API via curl. However, many times the REST API

Re: Flink 'Job Cluster' mode Ui Access

being requested. On 05/12/2019 05:57, Jatin Banger wrote: I have tried that already using '$FLINK_HOME/bin/jobmanager.sh" start-foreground Ui comes fine with this one. Which means web/index.html is present. On Wed, Dec 4, 2019 at 9:01 PM Chesnay Schepler <mailto:ches...@apache.org

Re: Flink 1.9.1 allocating more containers than needed

There are some follow-up issues that are fixed for 1.9.2; release date for that is TBD. https://issues.apache.org/jira/browse/FLINK-12342 https://issues.apache.org/jira/browse/FLINK-13184 On 06/12/2019 15:08, eSKa wrote: Hello, recently we have upgraded our environment to from 1.6.4 to 1.9.1.

Re: Flink 1.9.1 allocating more containers than needed

Note that FLINK-10848 is included in 1.9.X, but it didn't fix the issue completely. On 06/12/2019 15:10, Chesnay Schepler wrote: There are some follow-up issues that are fixed for 1.9.2; release date for that is TBD. https://issues.apache.org/jira/browse/FLINK-12342 https://issues.apache.org

Re: Flink 1.9.1 allocating more containers than needed

I would expect January. With 1.8.3 release being underway, 1.10 feature freeze coming close and, of course, Christmas, it seems unlikely that we'll manage to pump out another bugfix release in December. On 06/12/2019 15:18, eSKa wrote: Thank you for quick reply. Will wait for 1.9.2 then. I

Re: flink 1.9

We will not release Flink version bundling Hadoop. The versioning for flink-shaded-hadoop-uber is entirely decoupled from Flink version. You can just use the flink-shaded-hadoop-uber jar linked on the downloads page with any Flink version. On 18/10/2019 13:25, GezimSejdiu wrote: Hi Flink

Re: Fwd: Is it possible to get Flink job name in an operator?

If you have control over the job you can modify it to use ExEnv#execute(String jobName), and üass this explicitly to your functions in some form (like the global job parameters). Beyond that there is no way to access the job name from within a function/operator. On 15/10/2019 08:53, 马阳阳

Re: Can flink 1.9.1 use flink-shaded 2.8.3-1.8.2

If you need hadoop, but the approach outlined here doesn't work for you, then you still need a flink-shaded-hadoop-jar that you can download here

Re: Can flink 1.9.1 use flink-shaded 2.8.3-1.8.2

, otherwise it may make user confused. Chesnay Schepler mailto:ches...@apache.org>> 于2019年10月25日周五 下午4:21写道： If you need hadoop, but the approach outlined here <https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/projectsetup/dependencies.html> doesn't work for y

<    1   2   3   4   5   6   7   8   9   10   >