Re: How to output a Table to Kafka?

Hi chi ma, A query with "group by" produces an updated table. Both UpsertStreamTableSink and RetractStreamTableSink can be used to emit an updated table. The main difference between UpsertStreamTableSink and RetractStreamTableSink is RetractStreamTableSink will encode an UPDATE change as a

Re: java.lang.Exception: TaskManager was lost/killed

Hi Chesnay, Don’t know if this helps, but I’d run into this as well, though I haven’t hooked up YourKit to analyze exactly what’s causing the memory problem. E.g. after about 3.5 hours running locally, it failed with memory issues. In the TaskManager logs, I start seeing exceptions in my

Re: java.lang.Exception: TaskManager was lost/killed

Same story here, 1.3.2 on K8s. Very hard to find reasons on why a TM is killed. Not likely caused by memory leak. If there is a logger I have turn on please let me know. On Mon, Apr 9, 2018, 13:41 Lasse Nedergaard wrote: > We see the same running 1.4.2 on Yarn hosted

Re: java.lang.Exception: TaskManager was lost/killed

We see the same running 1.4.2 on Yarn hosted on Aws EMR cluster. The only thing I can find in the logs from are SIGTERM with the code 15 or -100. Today our simple job reading from Kinesis and writing to Cassandra was killed. The other day in another job I identified a map state.remove command

Re: java.lang.Exception: TaskManager was lost/killed

We will need more information to offer any solution. The exception simply means that a TaskManager shut down, for which there are a myriad of possible explanations. Please have a look at the TaskManager logs, they may contain a hint as to why it shut down. On 09.04.2018 16:01, Javier Lopez

Re: Flink Uploaded JAR Filename

Why don’t you parse the response from curl and use it to trigger the second request? That is easy automatable using Bash commands - or do I overlook something here? > On 9. Apr 2018, at 18:49, Pavel Ciorba wrote: > > Hi everyone > > I make 2 cURL POST requests to upload

Flink Uploaded JAR Filename

Hi everyone I make 2 cURL POST requests to upload and run a Flink job. The issue is that I do it manually: First I do the upload, then using the returned name of the jar, I make a follow up run request. $ curl -X POST upload jar ... {status:success, filename: 12jhgjasd123asd12-ACTUALNAME.jar} $

Re: RocksDBMapState example?

Hi, Have you looked at the ctor call in : flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java around line 1261 ? Cheers

RocksDBMapState example?

Hi, I'd like to use RocksDB to store a key-value mapping table (with 45 million keys). Can someone please point me to an example of RocksDBMapState() constructor invocation? Or an explanation of constructor arguments?.. Thanks, Alex Nekrassov nekras...@att.com

Re: Re: java.lang.Exception: TaskManager was lost/killed

Hi, "are you moving the job jar to the ~/flink-1.4.2/lib path ? " -> Yes, to every node in the cluster. On 9 April 2018 at 15:37, miki haiat wrote: > Javier > "adding the jar file to the /lib path of every task manager" > are you moving the job jar to the*

Re: Re: java.lang.Exception: TaskManager was lost/killed

Javier "adding the jar file to the /lib path of every task manager" are you moving the job jar to the* ~/flink-1.4.2/lib path* ? On Mon, Apr 9, 2018 at 12:23 PM, Javier Lopez wrote: > Hi, > > We had the same metaspace problem, it was solved by adding the jar file to >

Re: Flink override config params (Docker)

Miki, So something like following will do the trick? *volumes:- /host/path/to/custom-flink-conf.yaml:/opt/flink/conf/flink-conf.yaml* 2018-04-09 14:11 GMT+03:00 miki haiat : > You can mount the conf folder and ovride the conf file. > > On Mon, 9 Apr 2018, 14:04 Pavel

Re: Flink 1.4.2 in Zeppelin Notebook

The error message is: org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Communication with JobManager failed: Lost connection to the JobManager. Am 09.04.2018 um 14:12 schrieb kedar mhaswade: > Hmm. What error do you see on the Zeppelin console when you

Re: Flink 1.4.2 in Zeppelin Notebook

Hmm. What error do you see on the Zeppelin console when you click the run (flink code) button after making these changes for flink interpreter config (I assume you restart the interpreter)? Regards, Kedar On Mon, Apr 9, 2018 at 12:50 AM, Rico Bergmann wrote: > Hi. > >

Re: Flink override config params (Docker)

You can mount the conf folder and ovride the conf file. On Mon, 9 Apr 2018, 14:04 Pavel Ciorba, wrote: > Hi everyone > > Is there a way to override the *conf/flink-conf.yaml* of the Flink Docker > container? > > I need to specify some params such as: > state.backend >

Flink override config params (Docker)

Hi everyone Is there a way to override the *conf/flink-conf.yaml* of the Flink Docker container? I need to specify some params such as: state.backend state.backend.fs.checkpointdir state.checkpoints.dir etc. Thanks

Re: Re: java.lang.Exception: TaskManager was lost/killed

Hi, We had the same metaspace problem, it was solved by adding the jar file to the /lib path of every task manager, as explained here https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/debugging_classloading.html#avoiding-dynamic-classloading. As well we added these java

Re: Re: java.lang.Exception: TaskManager was lost/killed

I've seen similar problem, but it was not a heap size, but Metaspace. It was caused by a job restarting in a loop. Looks like for each restart, Flink loads new instance of classes and very soon in runs out of metaspace. I've created a JIRA issue for this problem, but got no response from the

Re:Re: java.lang.Exception: TaskManager was lost/killed

thanks a lot,i will try it 在 2018-04-09 00:06:02，"TechnoMage" 写道： I have seen this when my task manager ran out of RAM. Increase the heap size. flink-conf.yaml: taskmanager.heap.mb jobmanager.heap.mb Michael On Apr 8, 2018, at 2:36 AM, 王凯 wrote:

Re: Flink 1.4.2 in Zeppelin Notebook

Hi. Thanks for your reply. But this also didn’t work for me. In the JM log I get an akka Error („dropping message for non-local recipient“). My setup: I have Flink running on Kubernetes cluster, version 1.4.2. zeppelin is version 0.8 using the flink interpreter compiled against flink 1.1.3.

How to output a Table to Kafka?

Hi, I'm a newbie to Flink. I'm trying to load data from HDFS and analyze it using Flink Table APIs. I created a TableSource, and registered it into the StreamTableEnvironment as a table, and ran a SQL on that table through streamTableEnvironment.sqlQuery, and finally I called writeToSink to