Re: flink-1.6.1 :: job deployment :: detached mode

Hi Till,Thank you for your reply.Yes, I’ve upgraded to the latest Flink-1.6.2 and the problem is still there, please find the log file attached.  flink-job-1.6.2.log Description: Binary data Kind Regards,Mike Pryakhin On 7 Nov 2018, at 18:46, Till Rohrmann wrote:Hi

Implementation error: Unhandled exception - "Implementation error: Unhandled exception."

Hello, We have a flink job / cluster running in kubernetes. Flink 1.6.2 (but the same happens in 1.6.0 and 1.6.1) To upgrade our job we use the REST API. Every so often the jobmanager seems to be stuck in a crashing state and the logs show me this stack trace: 2018-11-07 18:43:05,815

RE: Kubernetes Job Cluster - Checkpointing with Parallelism > 1

Upgrading to Flink 1.6.2 from 1.6.0 appears to fix this, after making sure each job is writing checkpoints to a unique directory since the jobid's all match. Thad Truman | Software Engineer | Neovest, Inc. A: T: E: 1145 S 800 E, Ste 310 Orem, UT 84097 +1 801 900 2480

Re: Error after upgrading to Flink 1.6.2

Hi Till, we are not using HBase at the moment. We managed to run successfully the job but it was a pain to find the right combination of dependencies, library shading and the right HADOOP_CLASSPATH. The problem was the combination of parquet, jaxrs, hadoop and jackson. Moreover we had to run the

Re: FlinkCEP, circular references and checkpointing failures

Really good finding Stefan! On Wed, Nov 7, 2018 at 5:28 PM Stefan Richter wrote: > Hi, > > I think I can already spot the problem: LockableTypeSerializer.duplicate() > is not properly implemented because it also has to call duplicate() on the > element serialiser that is passed into the

Re: Error after upgrading to Flink 1.6.2

Hi Flavio, I haven't seen this problem before. Are you using Flink's HBase connector? According to similar problems with Spark one needs to make sure that the hbase jars are on the classpath [1, 2]. If not, then it might be a problem with the MR1 version 2.6.0-mr1-cdh5.11.2 which caused problems

Re: flink-1.6.2 in standalone-job mode | Cluster initialization failed.

Hi Zavalit, the AbstractMethodError indicates that there must be some kind of version conflict. From Flink 1.6.1 to 1.6.2 we modified the signature of `ClusterEntrypoint#createResourceManager` which causes the problem if you mix up versions. Could you check that you don't mix Flink 1.6.1 and

Re: InterruptedException when async function is cancelled

Hi Till, Thanks for the reply. Is there any particular patch I can use as upgrading to Flink 1.6 is not an option for me at the moment. Regards, Anil. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Unbalanced Kafka consumer consumption

Hi Gerard, the behaviour you are describing sounds odd to me. I have a couple of questions: 1. Which Flink and Kafka version are you using? 2. How many partitions do you have? --> Try to set the parallelism of your job to the number of partitions. That way, you will have one partition per source

flink run from savepoint

Hi, I'm working with Flink 1.5.0 and I try to run a job from a savepoint. My jobmanager is dockerized and I try to run my flink job in another container. The command : flink run -m jobmanager:8081 myJar.jar works fine, but when I try to run a job from a savepoint, I got an Internal server

Re: FlinkCEP, circular references and checkpointing failures

Hi, I think I can already spot the problem: LockableTypeSerializer.duplicate() is not properly implemented because it also has to call duplicate() on the element serialiser that is passed into the constructor of the new instance. I will open an issue and fix the problem. Best, Stefan > On 7.

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Jayant, could you check that the UUID key on the TM is actually serialized using a Kryo serializer? You can do this by setting a breakpoint in the constructor of the `AbstractKeyedStateBackend`. Cheers, Till On Tue, Oct 30, 2018 at 9:44 AM bupt_ljy wrote: > Hi, Jayant > > Your code

Re: FlinkCEP, circular references and checkpointing failures

Hi Shailesh, could you maybe provide us with an example program which is able to reproduce this problem? This would help the community to better debug the problem. It looks not right and might point towards a bug in Flink. Thanks a lot! Cheers, Till On Tue, Oct 30, 2018 at 9:10 AM Dawid

Re: InterruptedException when async function is cancelled

Hi Anil, as Stephan stated, the fix is not included in Flink 1.4.2 but in the later version of Flink. Can you upgrade to Flink 1.5.5 or Flink 1.6.2 to check whether the problem still occurs? Cheers, Till On Sun, Oct 28, 2018 at 8:55 AM Anil wrote: > I do see the same error but in case

Re: RocksDB checkpointing dir per TM

This is a very good point Elias. We actually forgot to add these options to the configuration documentation after a refactoring. I will fix it. Cheers, Till On Fri, Oct 26, 2018 at 8:27 PM Elias Levy wrote: > There is also state.backend.rocksdb.localdir. Oddly, I can find the > documentation

Re: 答复: Flink1.6.0 submit job and got "No content to map due to end-of-input" Error

Hi Jeroen, could you maybe share the Flink logs with us to further debug the problem? Cheers, Till On Fri, Oct 26, 2018 at 3:56 PM Jeroen Steggink | knowsy wrote: > Hi, > > I'm running Flink 1.5.4 and all dependencies in the job rely on 1.5.4. > However, I still get this error. According to

Re: Flink Task Allocation on Nodes

Hi Sayat, at the moment it is not possible to control the scheduling behaviour of Flink. In the future, we plan to add some kind of hints which controls whether tasks of a job get spread out or will be packed on as few nodes as possible. Cheers, Till On Fri, Oct 26, 2018 at 2:06 PM Kien Truong

Re: Run a Flink job: REST/ binary client

After a painful migration to Flink 1.6.2 we were able to run one of the jobs. Unfortunately we faced the same behaviour: all the code after the first env.execute() is not execute if the job is called from the REST services or from the web UI, while everything works fine if running the job using

Re: HA jobmanagers redirect to ip address of leader instead of hostname

Hi Jeroen, this sounds like a bug in Flink that we return sometimes IP addresses instead of hostnames. Could you tell me which Flink version you are using? In the current version, the redirect address and the address retrieved from ZooKeeper should actually be the same. In the future, we plan to

Re: Counting DataSet in DataFlow

Hi Fabian, Thanks for the response, I am going to use the second solution ! Regards, Bastien -- Bastien DINE Data Architect / Software Engineer / Sysadmin bastiendine.io Le mer. 7 nov. 2018 à 14:16, Fabian Hueske a écrit : > Another option for certain tasks is to work with

Re: RichInputFormat working differently in eclipse and in flink cluster

Hi Teena, which Flink version are you using? Have you tried whether this happens with the latest release 1.6.2 as well? Cheers, Till On Fri, Oct 26, 2018 at 1:17 PM Teena Kappen // BPRISE < teena.kap...@bprise.com> wrote: > Hi all, > > > > I have implemented RichInputFormat for reading result

State processing and testing utilities (Bravo)

Hey all! I just wanted to give you a quick update on the bravo project. Bravo contains a bunch of useful utilities for processing the checkpoint/savepoint state of a streaming job as Flink Datasets (batch). The end goal of the project is to be contributed to Flink once we are happy with it but

Re: Counting DataSet in DataFlow

Another option for certain tasks is to work with broadcast variables [1]. The value could be use to configure two filters. DataSet input = DataSet count = input.map(-> 1L).sum() DataSet input.filter(if cnt == 0).withBroadcastSet("cnt", count).doSomething DataSet input.filter(if cnt !=

How to run Flink 1.6 job cluster in "standalone" mode?

"Standalone" here I mean job-mananger + taskmanager on the same JVM. I have an issue to debug on our K8S environment, I can not reproduce it in local docker env or Intellij. If JM and TM are running in different VMs, it makes things harder to debug. Or is there a way to debug a job running on JM

Re: Counting DataSet in DataFlow

Hi, Counting always requires a job to be executed. Not sure if this is what you want to do, but if you want to prevent to get an empty result due to an empty cross input, you can use a mapPartition() with parallelism 1 to emit a special record, in case the MapPartitionFunction didn't see any

Counting DataSet in DataFlow

Hello, I would like to a way to count a dataset to check if it is empty or not.. But .count() throw an execution and I do not want to do separe job execution plan, as hthis will trigger multiple reading.. I would like to have something like.. Source -> map -> count -> if 0 -> do someting

Error after upgrading to Flink 1.6.2

Hi to all, we tried to upgrade our jobs to Flink 1.6.2 but now we get the following error (we saw a similar issue with spark that was caused by different java version on the cluster servers so we checked them and they are all to the same version - oracle-8-191): Caused by:

RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

FYI, the code below ends with version 1.6.0, do not end in 1.6.1. I suspect it’s a bug instead of a new feature. De : LINZ, Arnaud Envoyé : mercredi 7 novembre 2018 11:14 À : 'user' Objet : RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6 Hello, This has

RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

Hello, This has nothing to do with HA. All my unit tests involving a streaming app now fail in “infinite execution” This simple code never ends : @Test public void testFlink163() throws Exception { // get the execution environment final StreamExecutionEnvironment env =

flink job restarts when flink cluster restarts?

Hi, I have a question regarding whether the current running job will restart if I stop and start the flink cluster? 1. Let’s say I am just having a Standalone one node cluster. 2. I have several Flink jobs already running on the cluster. 3. If I do a bin/cluster-stop.sh and then do a

Re: Always trigger calculation of a tumble window in Flink SQL

Hi, You would have to register timers (probably based on event time). Your operator would be a vastly simplified window operator, where for given window you keep emitted record from your SQL, sth like: MapState emittedRecords; // map window start -> emitted record When you process elements,

Re: Flink Kafka to BucketingSink to S3 - S3Exception

Thank you Addison and Ravi for the detailed info. Hi Addison, it sounds like StreamingFileSink is promising and will be available in Flink 1.7. From the mailing list, it looks like Flink 1.7 RC is now available for use. Some questions for you... in your use case, is your source Kafka and is