Running job in "dry mode"?

Hello, I'd like to be able to see if new version of my job is compatible with the old one. I can make a savepoint and run new version from that, but I'd like to be able to do it without actually starting sources and so on - so that e.g. it won't start to read from my kafka topics. Of

Re: Fink: KafkaProducer Data Loss

I tried to build Flink with hadoop cdc 5.8.3 and test it, but it was unable to come up. When I ran the yarn-session script, it exited with something like "Can't get the url for job manager". I didn't spend much time figuring out what's wrong, and went straight to Flink 1.3.0. I ran several

Re: Flink and swapping question

Hi Stephan, I also think that the error is more related to netty. The only suspicious library I use are parquet or thrift. I'm not using off-heap memory. What do you mean for "crazy high number of concurrent network shuffles"?how can I count that? We're using java 8. Thanks a lot, Flavio On 6

Re: Issue With Configuration File

Hi Will, I’m afraid that will simply run your program in Flink local mode, where the configuration settings are ignored because an in-process Flink Cluster is being started. For running a Beam pipeline on YARN you have two options right now: 1. Start a Flink YARN session as described here

Re: AsyncCollector Does not release the thread (1.2.1)

Hi Aljoscha , thank you for your reply, yes the queue being filled up and no more elements are being processed.(In relation to the limit defined at the "orderedWait" function call). To add additional information, if I run the test on a local cluster I can see that the job never ends because the

Re: Fink: KafkaProducer Data Loss

Not yet. Planning to do that today. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13544.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

Hi, From the logs and the description of your test scenarios where data loss is observed and not observed, it seems like the differentiating factor here is whether or not the session windows trigger was first fired when the checkpoint occurred. It doesn’t however explain the case where your

Re: Flink and swapping question

Hi! I would actually be surprised if this is an issue in core Flink. - The MaxDirectMemory parameter is pretty meaningless, it really is a max and does not have an impact on how much is actually allocated. - In most cases we had reported so far, the leak was in a library that was used in

Re: Issue With Configuration File

Aljoscha, You're correct that I'm using Beam. Here's the shell script I'm using to start the job: hadoop jar **path to jar file** org.apache.beam.examples.complete.TfIdf \--runner=FlinkRunner \--input= **path to directory** \--output=tfIdf There are two lines before that to set the Flink config

Re: Flink cluster : Client is not connected to any Elasticsearch nodes!

Hi Dhinesh, Could it be that you didn’t configure the network binding address of the ES installation properly? You need to make sure it isn’t binded to 127.0.0.1, which I think in some Elasticsearch versions is the default binding. Also, just a reminder if you haven’t done so, please make sure

RE: KryoException: Encountered unregistered class ID

Hi Shai, Your suggestion makes a lot of sense. I did not realize Kryo allows changing that, thanks for correcting! It’s definitely reasonable to provide a way to proxy that setting through the `StreamExecutionEnvironment`, if Kryo itself has the functionality already. I’ve filed a JIRA for this

RE: KryoException: Encountered unregistered class ID

Actually Kryo does allow overriding that by calling kryo.setDefaultSerializer(), it would be nice if Flink would provide a way to override getKryoInstance() or to subscribe a callback for after Kryo is initialized, or simply let me define a default serializer and then call

Re: Methods that trigger execution

*Thank so much Aljoscha* :) I was stucked in this point. I didn't know that the print or collect method collecting all the data in one place. The execution time has dropped a lot. However, I still get that Flink is slower (just for 7 seconds). I really think I'm not getting all the performance

RE: KryoException: Encountered unregistered class ID

Ah, I see what you’re trying to achieve. Then I don’t think that is possible then, simply because Kryo doesn’t allow overriding that. But wouldn’t you be able to just, through Flink’s `StreamExecutionEnvironment`, register the TaggedFieldSerializer for your to-be-migrated specific class? The

RE: KryoException: Encountered unregistered class ID

No. This way you can set default serializers for specific classes, I want to change the default serializer that Kryo uses when it can't find a registered serializer for a class. See Kryo.getDefaultSerializer(), notice the last line calls newDefaultSerializer() which is hardcoded to be

Re: Streaming use case: Row enrichment

Hi Aljosha, thanks for getting back to me on this! I'll try to simplify the thread starting from what we want to achieve. At the moment we execute some queries to a db and we store the data into Parquet directories (one for each query). Let's say we then create a DataStream from each dir, what we

Re: Async I/O & async-http-client & Netty

Hi Philipp, Thanks for the heads-up. I think it fails because Flink uses a different version of Netty that has an incompatible method signature or is missing that method completely. By placing it in the lib folder the different Netty version is in the user code class loader, which means that

Re: Queryable State Client with 1.3.0-rc0

Hi Claudio, Quick follow up: querying a locally started cluster does not work out-of-box anymore in Flink 1.3. You can manually start a mini cluster that has the required settings, though. You would do something like this: Configuration configuration = new Configuration();

RE: KryoException: Encountered unregistered class ID

StreamExecutionEnvironment.addDefaultKryoSerializer(YourClass.class, TaggedFieldSerializer.class) should work. You can also specify it directly: StreamExecutionEnvironment.registerTypeWithKryoSerializer(YourClass.class, TaggedFieldSerializer.class). Does the above work? On 6 June 2017 at

RE: KryoException: Encountered unregistered class ID

I understand that my problem arises from Kryo using FieldSerializer as the default serializer. Looking at Kryo's documentation (https://github.com/EsotericSoftware/kryo#compatibility), this could be easily solved by setting the default serializer to TaggedFieldSerializer. Flink, however,

Re: Flink with Hibernate ORM doesn't release memory after job is canceled

Hi Sendoh, Could you please go into some details about your job. Maybe show some representative code snippets or how the ORM is used exactly. Best, Aljoscha > On 5. May 2017, at 18:10, Sendoh wrote: > > Hi Flink users, > > Does anyone use Flink with Hibernate ORM

Re: AsyncCollector Does not release the thread (1.2.1)

Hi, As far as I know calling collect(Throwable) should also finish the promise that would otherwise fulfilled by successfully collecting a result. If not then you might have found a bug. What makes you think that the Thread is not being released? Is your queue being filled up and no more

Re: Fink: KafkaProducer Data Loss

Hi Till, Attaching the logs to this post again. Thanks. jobManager.log tmOne.log tmTwo.log

Re: Use Single Sink For All windows

because of parallelism i am seeing db contention. Wondering if i can merge sink of multiple windows and insert in batch. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Use-Single-Sink-For-All-windows-tp13475p13525.html Sent from the Apache

Re: Queryable State Client with 1.3.0-rc0

Hi Claudio, The documentation for this was recently updated: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/queryable_state.html#querying-state . Could you see

Re: Queries regarding FlinkCEP

Hi Dawid, What you wrote is exactly correct, it wouldn't generate a new waatermark (and subsequently throw events) unless maxOutOfOrderness time is elapsed. Thus, I was expecting for alerts to be raised as the stream was out of order but not out of maxOutOfOrderness. Nevertheless I tried your

Re: Queryable state in a keyed stream not querying properly

Hi Philip, The JobManager should figure out the correct TaskManager for your query based on the key. You mentioned that you get the result 1/3 of the time, is this 1/3 of the time for queries with exactly the same key or for queries with different keys? Also, could it be that the state

Re: State in Custom Tumble Window Class

Hi, Yes I think you can manually store the latest watermark by using the OperatorStateStore that you get if your user function implements the CheckpointedFunction interface. Best, Aljoscha > On 30. May 2017, at 13:43, rhashmi wrote: > > Thanks Aljoscha Krettek, > > So

Re: Amazon Athena

Hi, I don’t have any experience with Athena but this sounds doable. It seems that you only need to have some way of writing into S3 and then Athena will pick up the data in S3 when running queries. Multiple folks have used Flink to write data from Kafka into S3, the most recent case I know

Re: Keys distribution insights

Thanks Aljoscha. As I was suspecting, currently there's no unobtrusive way for that, but I can live with it.. Best, Flavio On Tue, Jun 6, 2017 at 3:48 PM, Aljoscha Krettek wrote: > Hi, > > There is no way of doing it with any Flink UI but you could try and do it >

Re: Batch Side Outputs

There are no plans to add side output to the DataSet API. The reason is that the optimizer would need to be adapted as well which would be a major change. Cheers, Fabian 2017-06-06 14:47 GMT+02:00 Aljoscha Krettek : > Hi Flavio, > > As far as I am aware no one is currently

Re: Cluster Memory Error

Hi Will, You you please share some details regarding how you start your cluster and how you submit the program? Best, Aljoscha > On 3. Jun 2017, at 00:26, Will Walters wrote: > > Hello, > > I am attempting to use the Flink runner to execute a data pipeline on a >

Re: Queries regarding FlinkCEP

Hi Biplop, CEP library internally must ensure order of the incoming events. It sorts events upon Watermark arrival. At that time it sorts events with timestamp < Watermark. With BoundedOutOfOrdernessTimestampExtractor a Watermark with time* t* is generated if there arrives event with timestamp

Re: Use Single Sink For All windows

Hi, Do you mean to write all the windows to the sink (MySQL) at the same time or simply that you want to write windows, as they come, using the one single sink instance? Best, Aljoscha > On 4. Jun 2017, at 08:47, rhashmi wrote: > > Is it possible to return all windows

Re: Keys distribution insights

Hi, There is no way of doing it with any Flink UI but you could try and do it manually: in your job, instead of doing the actual computation just count how many elements you have per key (in your GroupReduce). Then put a MapPartition right after the GroupReduce (which should preserve the same

Re: Queries regarding FlinkCEP

Sorry to bombard with so many messages , but one last thing is the example would produce alert if the line specifying Event Time is commented out. More specifically, this one: env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); Only with event time, there is no alert. Thanks,

Re: Queries regarding FlinkCEP

Also, my test environment was Flink 1.4-Snapshot with Kafka 0.10.0 on HDP 2.5. And I sent my test messages via the console producer. Thanks, Biplob -- View this message in context:

Re: Fink: KafkaProducer Data Loss

Hi Ninad, the logs for the data loss case weren't attached to the mails. Maybe you could attach them again in the same way as you did for the no data loss logs. Cheers, Till On Sun, Jun 4, 2017 at 2:55 PM, ninad wrote: > Yeah, this seems like a problem with flink

Re: Latency on Flink

Hi Gabriele, There are a couple of things you can try to speed up your pipeline: 1. Avoid JSON, parsing it and creating it are both slow. You can try using a binary format for your data, for example. If you can’t avoid JSON look into JSON parsing libraries and compare performance. 2. Avoid

Re: Queries regarding FlinkCEP

Thanks a lot, Till and Dawid for such detailed reply. I tried to check and wait what both of you suggested and I still have no events. Thus as pointed out by till, I created a self-contained example to reproduce the issue and the behaviour is the same as was in my original case. Please find the

Re: Methods that trigger execution

Hi, Using collect() (or print()) on a DataSet is almost never a good idea in Flink because this requires collecting all the data in one central place and sending it to the client. What you normally would do is write the data out to some file system (for example HDFS) and use env.execute() for

Re: Batch Side Outputs

Hi Flavio, As far as I am aware no one is currently working on adding side outputs for the DataSet API. The workaround is to output one common type from a function have several parallel filters after that for filtering out the elements of the correct type for the respective stream. Best,

Re: Issue With Configuration File

Hi Will, How are you starting your cluster/executing your program? From what I can gather you are using Beam, is that right? The line about the FlinkMiniCluster seems strange because this would hint at the fact that the Runner is trying to execute in local mode. Best, Aljoscha > On 6. Jun

Re: FileSystem vs RocksDb backend

Hi Fritz, the FSStateBackend is a lot faster than the RocksDBStateBackend. I don't know exact numbers but if I remember correctly somebody mentioned a 10x difference. Delta checkpointing is only supported by RocksDB since Flink 1.3. Given that the FSStateBackend is only suitable for small

Re: Flink and swapping question

Hi Flavio, can you post the all memory configuration parameters of your workers? Did you investigate which whether the direct or heap memory grew? Thanks, Fabian 2017-05-29 20:53 GMT+02:00 Flavio Pompermaier : > Hi to all, > I'm still trying to understand what's going on

Batch Side Outputs

Hi to all, will side outputs [FLINK-4460 ] be eventually available also for batch API? Best, Flavio

Re: Unable to use Flink RocksDB state backend due to endianness mismatch

Hi, RocksDB is a native library with JNI binding. It is included as a dependency and does not build from source when you build Flink. The included jar provides native code for Linux, OSX, and Windows on x86-64. From the Exception, I would conclude you are using a different CPU architecture

Re: Exception in Flink 1.3.0

AFAIK, the release process for 1.3.1 should hopefully start within the next days. There’s already a few critical fixes on the waiting list. On 6 June 2017 at 9:34:40 AM, rhashmi (rizhas...@hotmail.com) wrote: Thanks for sharing ticket reference. Is there any time line as this is blocker?

Re: Exception in Flink 1.3.0

Thanks for sharing ticket reference. Is there any time line as this is blocker? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Exception-in-Flink-1-3-0-tp13493p13500.html Sent from the Apache Flink User Mailing List archive. mailing list

Re: ProcessFunction broke : Flink 1.3 upgrade

Hi, Sorry for my misunderstanding here. Yes, the RichProcessFunction variant was removed in 1.3. Now, ProcessFunction has been changed to always be a RichFunction, so access to the runtime context / lifecycle methods is always available through the ProcessFunction interface. You should simply

Re: ProcessFunction broke : Flink 1.3 upgrade

yes i see that so shall i update code from RichProcessFunction to RichFunction(based object)? The implementation throw compile exception as RichProcessFunction has been removed from 1.3. -- View this message in context:

Re: ProcessFunction broke : Flink 1.3 upgrade

Hi, The RichProcessFunction variant gives access to the functionality provided by the RichFunction interface (e.g. allows overriding the `open` function lifecycle method, access to the runtime context, etc.). Whether or not to use it depends on your use case. You should be able to just

Re: Exception in Flink 1.3.0

Hi, Thanks for reporting this. This is a know issue to be fixed in the upcoming 1.3.1:  https://issues.apache.org/jira/browse/FLINK-6844. Regards, Gordon On 6 June 2017 at 1:47:53 AM, rhashmi (rizhas...@hotmail.com) wrote: After upgrade i started getting this exception, is this a bug?