Get Job Id programmatically for SavePoint

Hi, Is there a way I can get job id of my application programmatically or through api's for doing savepoint? Rahul Raj

Re: NoResourceAvailable exception

Update: Following other discussions I even tried to reduce memory.fraction to 10% without success. How can I set G1 as garbage collector? the key is env.java.opts but the value? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink flick cancel vs stop

On Fri, Sep 15, 2017 at 10:02 AM, Eron Wright wrote: > Aljoscha, would it be correct to characterize your idea as a 'pull' source > rather than the current 'push'? It would be interesting to look at the > existing connectors to see how hard it would be to reverse their >

Re: User-based authentication in Flink

I think the short-term approach is to place an nginx proxy in front, in combination with some form of isolation of the underlying endpoint. That addresses the authentication piece but not fine-grained authorization. Be aware that the Flink JM is not multi-user due to lack of isolation among

Re: Custom Serializers

Eventually I'll have a class named Element which holds an array of Parameter Do I need typeinfo, comparator, factory and serializer for both of them? Thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink flick cancel vs stop

Aljoscha, would it be correct to characterize your idea as a 'pull' source rather than the current 'push'? It would be interesting to look at the existing connectors to see how hard it would be to reverse their orientation. e.g. the source might require a buffer pool. On Fri, Sep 15, 2017 at

Re: Task Manager was lost/killed due to full GC

Late response, but a common reason for disappearing TaskManagers is termination by the Linux out-of-memory killer, with the recommendation to decrease the allotted memory. > On Sep 5, 2017, at 9:09 AM, ShB wrote: > > Hi, > > I'm running a Flink batch job that

Re: Dot notation not working for accessing case classes nested fields

Great, thanks! The fact that it's actually written on the documentation is really misleading. Thank you very much for your response Federico D'Ambrosio Il 15 set 2017 13:26, "Gábor Gévay" ha scritto: > Hi Federico, > > Sorry, nested field expressions are not supported in

Re: Flink flick cancel vs stop

Also relevant for this discussion: Several people (including me) by now were floating the idea of reworking the source interface to take away the responsibility of stopping/canceling/continuing from a specific source implementation and to instead give that power to the system. Currently each

Re: NoResourceAvailable exception

I tried also to set the only job manager on the first node and reconfiguring the cluster admitting just two task manager. In this way I obtain immediately a NoResourceAvailable error -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: NoResourceAvailable exception

I investigated the semantics of cpu percentage on top. I have to correct my sentence: When I start the program it has a peak at 160% (max is 200%), but after a second it falls down until the 4%. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

Hi Aljoscha, Thanks for your reply. It looks great to have hat feature. I will create a Jira issue for that and try to solve it. Best Regards, Tony Wei 2017-09-15 20:51 GMT+08:00 Aljoscha Krettek : > Hi, > > I think calling getPath() on the URL returned from getResource()

Re: Queryable State

Hi Navneeth, If you increase the timeout, everything works ok? I suppose from your config that you are running in standalone mode, right? Any other information about the job (e.g. code and/or size of state being fetched) and the cluster setup that can help us pin down the problem, would be

Re: Custom Serializers

Sorry, I was discussing this with Stephan before posting it here. Basically main wrapper holds an array with a custom object and because its size can change thoughtout the stream and users can customize their sources dynamically, it make it difficult to create a generic pojo or use tuple for this

Re: Testing recoverable job state

Hi, 1. Question: When you are throwing an exception within your user code, Flink will cancel the execution of all tasks and schedule them again (if you've configured a restart strategy). 2. Question: You'll need to configure the MiniCluster in HA mode. I believe that should be possible by

Re: Clean GlobalWidnow state

I'm using nabble and seems that it has removed the code between raw tags. Here it is again: import com.typesafe.scalalogging.LazyLogging import org.apache.flink.api.common.functions.ReduceFunction import org.apache.flink.api.common.state.{ReducingStateDescriptor, ValueStateDescriptor} import

Re: Custom Serializers

Hi Nuno, Because of this, we have a legacy structure that I showed before.  Could you probably include more information about this legacy structure you mentioned here in this mail thread? I couldn’t find any other reference to that. That could be helpful to understanding your use case more

Re: Clean GlobalWidnow state

Sure, but how does the Trigger actually work? > On 15. Sep 2017, at 12:20, gerardg wrote: > > Sure: > > > > The application is configured to use processing time. > > Thanks, > > Gerard > > > > -- > Sent from: >

Re: How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

Hi, I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to

Re: NoResourceAvailable exception

I think it might be that the computation is to CPU heavy, which makes the TaskManager unresponsive to any JobManager messages and so the JobManager thinks that the TaskManager is lost. @Till, do you have another idea about what could be going on? > On 15. Sep 2017, at 13:52, AndreaKinn

Re: NoResourceAvailable exception

the job manager log probably is more interesting: 2017-09-15 12:47:45,420 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-09-15 12:47:45,650 INFO

Re: Dot notation not working for accessing case classes nested fields

Hi Federico, Sorry, nested field expressions are not supported in these methods at the moment. I have created a JIRA issue for this: https://issues.apache.org/jira/browse/FLINK-7629 I think this should be easy to fix, as all the infrastructure for supporting this is already in place. I'll try to

Re: NoResourceAvailable exception

This is the log: 2017-09-15 12:47:49,143 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classe$ 2017-09-15 12:47:49,257 INFO org.apache.flink.runtime.taskmanager.TaskManager -

Custom Serializers

Hi, First of all, great #FF17, really enjoyed it. After attending some of the dataArtisans folks talks, realized that serialization should be optimized if there is no way to use supported objects. In my case, users can configure their source in our application online which gives them freedom to

Re: Streaming API has a long delay at the beginning of the process.

Hi Fabian, Thank you for your description. This is my understanding. 1, At the exact time execute() method called, Flink creates JobGraph, submit it to JobManager, deploy tasks to TaskManagers and DOES NOT execute each operators. 2, Operators are executed when they needed. 3,

Re: Clean GlobalWidnow state

Sure: The application is configured to use processing time. Thanks, Gerard -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: NoResourceAvailable exception

Hi, Can you check in the TaskManager logs whether there is any message that indicates why the TaskManager was lost? Also, there might be information in your machine logs, i.e. "dmesg" or /var/log/messages or some such. Best, Aljoscha > On 14. Sep 2017, at 22:28, AndreaKinn

Re: Clean GlobalWidnow state

Hi, Could you maybe show the code of your trigger? Best, Aljoscha > On 15. Sep 2017, at 11:39, gerardg wrote: > > Hi, > > I have the following operator: > > mainStream > .coGroup(coStream) > .where(_.uuid).equalTo(_.uuid) > .window(GlobalWindows.create()) >

Clean GlobalWidnow state

Hi, I have the following operator: mainStream .coGroup(coStream) .where(_.uuid).equalTo(_.uuid) .window(GlobalWindows.create()) .trigger(triggerWhenAllReceived) .apply(mergeElements) TLDR; It seems that the checkpointed state of the operator keeps growing forever

Re: Streaming API has a long delay at the beginning of the process.

Hi Yuta, when the execute() method is called, the a so-called JobGraph is constructed from all operators that have been added before by calling map(), keyBy() and so on. The JobGraph is then submitted to the JobManager which is the master process in Flink. Based on the JobGraph, the master