Re: save points through REST API not supported ?

No, this is also not possible in 1.4. This will be available in 1.5 where we rework the client-cluster communication to go entirely through the REST API. This will mean that everything you can do with the command-line client can also be achieved directly through the REST API. However, from

Re: Flink Batch Performance degradation at scale

Wow thank you for the reply, you gave me a lot to look into and mess with. I'll start testing with the various memory options and env settings tomorrow. BTW the current flink cluster is launched like: yarn-session.sh -n 700 -s 2 -tm 9200 -jm 5120 with flink-conf.yaml property overrides of: # so

Re: save points through REST API not supported ?

Hi. It is not possible through REST in Flink 1.3.2 I’m looking for the feature. The only option is to use ./Flink savepoint for now Med venlig hilsen / Best regards Lasse Nedergaard > Den 6. dec. 2017 kl. 21.52 skrev Vishal Santoshi : > > I was more interested in

Re: save points through REST API not supported ?

Hi Vishal, Job cancellations can be done via a REST API: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#cancel-job-with-savepoint Thanks, Vipul On Wed, Dec 6, 2017 at 10:56 AM, Vishal Santoshi wrote: > One can submit jobs,

save points through REST API not supported ?

One can submit jobs, upload jars, kill jobs etc very strange that you can’t do a save point ? Or am I missing something obvious ? Vishal

the location of JsonRowDeserializationSchema.java

Hi Flink users, I found this very useful JsonRowDeserializationSchema is in Kafka connector. https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/util/serialization/JsonRowDeserializationSchema.java Shouldn't

the location of JsonRowDeserializationSchema.java

Hi Flink users, I found this very useful JsonRowDeserializationSchema is in Kafka connector. https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/util/serialization/JsonRowDeserializationSchema.java Shouldn't

Re: Flink Batch Performance degradation at scale

Fabian, Thank you for the reply. Yes I do watch via the ui, is there another way to see progress through the steps? I think I just figured it out, the hangup is in the sort phase (ID 4) where 2 slots take all the time. Looking in the UI most slots get less than 500MB of data to sort, these

Re: aggregate does not allow RichAggregateFunction ?

Hi Vishal, you are right, it is not possible to use state in an AggregateFunction because windows need to be mergeable. An AggregateFunction knows how to merge its accumulators but merging generic state is not possible. I am not aware of an efficient and easy work around for this. If you want to

Re: CPU Cores of JobManager

That said, metrics could still be the issue, with FLINK-7368 being the prime suspect. This was fixed for 1.3.3 and 1.4 (both unreleased though). On 06.12.2017 10:36, Chesnay Schepler wrote: This can only have an effect if

Re: CPU Cores of JobManager

This can only have an effect if "*taskmanager.net.detailed-metrics*" is enabled (by default it is disabled). Otherwise an increase in sub-channels will neither increase the number of metrics nor the number of polls by the UI. On 06.12.2017 10:22, Stephan Ewen wrote: A 'rebalance()' should no

Re: CPU Cores of JobManager

A 'rebalance()' should no impose load on the JM itself. There are more sub-channels in that case, but the JobManager is not coordinating anything for them unless there is a failure/recovery. Each subchannel generates additional metrics, though - might that be an issue? On Wed, Dec 6, 2017 at

Re: CPU Cores of JobManager

> Which version of Fllink are you using? I'm using Flink 1.3.0. > Is the high CPU usage caused by the browser or the JVM? The CPU usage of browser is not high, the JVM is not also. I found that if I remove the rebalance function in my code, the WEBUI works well. I don't know why but I think

Re: How to perform efficient DataSet reuse between iterations

Hi Miguel, if the message size would be the problem, the client should fail with an exception. What might happen, is that the client gets stuck while optimizing the program. You could take a stacktrace of the client process to identify at which part the client gets stuck. Best, Fabian

Performance of docker-flink

Hi, I wanted to explore docker-flink (using Ceph for state backend). before opting for a standalone cluster. Has there been any comparative studies on the performance of docker-flink? Would the states be consistent and performant if the docker containers go down and respawn frequently?

Re: AW: Blob server not working with 1.4.0.RC2

Hi Bernd, at least from our side I don't see a change in the default BlobServer ports between 1.3 and 1.4 - without configuration, the OS chooses the port. If you want to influence the range it is chosen from (or want to fix a specific port), you need to set the blob.server.port configuration

Re: CPU Cores of JobManager

Which version of Fllink are you using? Is the high CPU usage caused by the browser or the JVM? One thing we should investigate is how many requests the UI sends to the JobManager, but I don't know of a good way to do that without modifying the source. On 06.12.2017 01:20, Yuta Morisawa