Re: Queryable state when key is UUID - getting Kyro Exception

Hi, Jayant The key you specified in getKvState function should be the key of the keyed stream instead of the key of the map. From what I’ve seen onhttps://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html, this feature only supports managed keyed state. By

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Till, Here is the client snippet. Here Rule is a custom POJO that I use. public static void main(String[] args) throws IOException, InterruptedException, ExecutionException { UUID uuid = UUID.fromString("2ba14b80-e6ff-11e8-908b-9bd8fd37bffb"); QueryableStateClient client = new

What if not to keep containers across attempts in HA setup?

Hi, Recently I found a bug on our YARN cluster that crashes the standby RM during a RM failover, and the bug is triggered by the keeping containers across attempts behavior of applications (see [1], a related issue but the patch is not exactly the fix, because the problem is not on recovery,

Re: Any examples on invoke the Flink REST API post method ?

Hi Henry, What you see in the API documentation is a schema definition and not a sample request. The request body should be: { "target-directory": "hdfs:///flinkDsl", "cancel-job": false } Let me know if that helps. Best, Gary On Mon, Nov 12, 2018 at 7:15 AM vino yang

Re: Any examples on invoke the Flink REST API post method ?

hi， http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/cancel-with-savepoint-404-Not-Found-td19227.html it may help you. and for flink on yarn cancel job , "yarn-cancel" work well not "cancel" the below python code for trigging savepoint work well. json = {"cancel-job": False} r

Re: How to use multiple sources with multiple sinks

Hi, If you are expressing a job that contains three pairs of source->sinks that are isolated from each other, then Flink supports this form of Job. It is not much different from a single source->sink, just changed from a DataStream to three DataStreams. For example, *DataStream ds1 = xxx*

Re: Task Manager allocation issue when upgrading 1.6.0 to 1.6.2

Hi Cliff, the TaskManger fail to start with exit code 31 which indicates an initialization error on startup. If you check the TaskManager logs via `yarn logs -applicationId ` you should see the problem why the TMs don't start up. Cheers, Till On Fri, Nov 9, 2018 at 8:32 PM Cliff Resnick wrote:

***UNCHECKED*** Table To String

I have created a project to use SQL but instead of printing the output as below I need the output in a STRING so I can write it to a dynamoDB table. How do I convert this "result" to a STRING or is there a suggestion of some other way I should sink to dynamoDB? Any example code would REALLY

Job xxx not found exception when starting Flink program in Local

Hi Experts, When I start Flink program in local, I found that the following exception throws out, I do not know why it happens because it happens in sudden, some hours ago the program can start successfully. Could anyone help to explain it? Thanks a lot! 2018-11-13

Re: Job xxx not found exception when starting Flink program in Local

After I restart my computer, the error is gone. > 在 2018年11月13日，下午2:53，徐涛 写道： > > Hi Experts, > When I start Flink program in local, I found that the following > exception throws out, I do not know why it happens because it happens in > sudden, some hours ago the program can

Job xxx not found exception when starting Flink program in Local

Hi Experts, When I start Flink program in local, I found that the following exception throws out, I do not know why it happens because it happens in sudden, some hours ago the program can start successfully. Could anyone help to explain it? Thanks a lot! 2018-11-13

***UNCHECKED*** Job xxx not found exception when starting Flink program in Local

Hi Experts, When I start Flink program in local, I found that the following exception throws out, I do not know why it happens because it happens in sudden, some hours ago the program can start successfully. Could anyone help to explain it? Thanks a lot!2018-11-13 14:48:45

Re: Understanding checkpoint behavior

Hi, do you observe such long checkpoint times also without performing external calls? If not, I guess the communication to the external system is flaky. Maybe you have to rethink how you perform such calls in order to make the pipeline more robust against these latencies. Flink also offers

Re: FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

Hi Steve, We've encountered this also. We have way more than enough shards, but were still getting exceptions. We think we know what is the reason, we would love for someone to approve/reject. What we suspect is happening is as follows: The KPL's RateLimit parameter is tracking the amount of

Linkage error when using DropwizardMeterWrapper

java.lang.LinkageError: loader constraint violation: when resolving method "org.apache.flink.dropwizard.metrics.DropwizardMeterWrapper.(Lcom/codahale/metrics/Meter;)V" the class loader (instance of org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader) of

Re: Report failed job submission

Hi Flavio, I'm not entirely sure if I get your question correct but what you are looking for is more information (like categorization) why the submission failed right? Regards, Timo Am 06.11.18 um 14:33 schrieb Flavio Pompermaier: Any idea about how to address this issue? On Tue, Oct 16,

Re: Report failed job submission

Let's say that my job needs to do some check before running (like existence of a file or some other condition): at the moment I can only throw an Exception but on the client side you get only something like: {"errors":["org.apache.flink.client.program.ProgramInvocationException: The main method

Re: Multiple operators to the same sink

Hi, I'm not quite sure if I understand your problem correctly. But your use case sounds like a typical application of a union operation. What do you mean with "knowledge of their destination sink"? The operators don't need to be aware of the destination sink. The only thing that needs to be

Re: How to run Flink 1.6 job cluster in "standalone" mode?

Hi, a session cluster does not imply that JM + TM are always executed in the same JVM. Debugging a job running on different JVMs might be a bit more difficult to debug but it should still be straightforward. Maybe you can tell us what wrong behavior you observe? Btw. Flink's metrics can

Re: Run a Flink job: REST/ binary client

I will loop in Chesnay. He might know more about the REST service internals. Timo Am 07.11.18 um 16:15 schrieb Flavio Pompermaier: After a painful migration to Flink 1.6.2 we were able to run one of the jobs. Unfortunately we faced the same behaviour: all the code after the first

Re: Error with custom `SourceFunction` wrapping `InputFormatSourceFunction` (Flink 1.6)

Hi Aljoscha, Thanks! I will look into this. Best, Aaron Levin On Fri, Nov 9, 2018 at 5:01 AM, Aljoscha Krettek wrote: > Hi, > > I think for this case a model that is similar to how the Streaming File > Source works should be good. You can have a look at > ContinuousFileMonitoringFunction and

Re: Implementation error: Unhandled exception - "Implementation error: Unhandled exception."

Hi Richard, this sounds like a bug to me. I will loop in Till (in CC) who might know more about this. Regards, Timo Am 07.11.18 um 20:35 schrieb Richard Deurwaarder: Hello, We have a flink job / cluster running in kubernetes. Flink 1.6.2 (but the same happens in 1.6.0 and 1.6.1) To

Flink with parallelism 3 is running locally but not on cluster

Hi, may be i just missing smth, but i just have no more ideas where to look. here is an screen of the failed state i read messages from 2 sources, make a join based on a common

Re: flink-1.6.2 in standalone-job mode | Cluster initialization failed.

jepp, that was the Issue. tnx a lot. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Flink auth against Zookeeper with MD5-Digest

Hi, I was wondering if there is any plans in the near future to include support for another authentication mechanism different than Kerberos? such as MD5-Digest ? Cheers,

Re: Report failed job submission

I assume you are using the REST API? Flink's RestClusterClient is able to deserialize the exception including its cause that might be more helpful in your case as well. The entire exception should be queryable via execution result [1]. At least we get a better error using the SQL Client [2]

Re: flink run from savepoint

Hi Franck, as a first hint: paths are hard-coded in the savepoint's metadata so you should make sure that the path is still the same and accessible by all JobManagers and TaskManagers. Can you share logs with us to figure out what caused the internal server error? Thanks, Timo Am

Re: InterruptedException when async function is cancelled

Hi Anil, if I researched correctly we are talking about these changes [1]. I don't know if you can back port it, but I hope this helps. Regards, Timo [1] https://issues.apache.org/jira/browse/FLINK-9304 Am 07.11.18 um 17:41 schrieb Anil: Hi Till, Thanks for the reply. Is there

Auto/Dynamic scaling in Flink

Dear Flink Contributors and users, I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in

Kinesis Shards and Parallelism

Looking at the doc https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kinesis.html "When the number of shards is larger than the parallelism of the consumer, then each consumer subtask can subscribe to multiple shards; otherwise if the number of shards is smaller than the

Re: flink job restarts when flink cluster restarts?

Hi, by default all the metadata is lost when shutting down the JobManager in a non high available setup. Flink uses Zookeeper together with a distributed filesystem to store the required metadata [1] in a persistent and distributed manner. A single node setup is rather uncommon, but you can

Re: Rich variant for Async IO in Scala

Hi Bruno, `org.apache.flink.streaming.api.functions.async.RichAsyncFunction` should also work for the Scala API. `RichMapFunction` or `RichFilterFunction` are also shared between both APIs. Is there anything that blocks you from using it? Regards, Timo Am 09.11.18 um 01:38 schrieb Bruno

Re: How to run Flink 1.6 job cluster in "standalone" mode?

Hi Tim, I am trying to debug this issue http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/java-io-IOException-NSS-is-already-initialized-td24247.html And in general, how to debug code in a distributed JM/TM architecture is very interesting topic to me. Any hints will be

Re: Linkage error when using DropwizardMeterWrapper

Nevermind. Relocating the dropwizard packages using maven shade plugin fixed it.

Re: Task Manager allocation issue when upgrading 1.6.0 to 1.6.2

Hi Till, Yes, it turns out the problem was having flink-queryable-state-runtime_2.11-1.6.2.jar in flink/lib. I guess Queriable State bootstraps itself and, in my situation, it brought the task manager down when it found no available ports. What's a little troubling is that I had not configured