Re: Slow flink checkpoint

Hi, Yes, you cannot start a separate thread to cleanup the state. State is managed by Flink and can only be accessed at certain points in time when the user code is called. If you are using event time, another trick you could play is to only register all timers on (currentWatermark + 1). That

Re: Custom Processing per window

Task 1: I implemented it using a custom Trigger (see attached file). Looks like it is doing what I want it to. I copied the code from EventTimeTrigger.java and overwrote the onElement method. Task 2: I will need to maintain the state (this will be the LRU cache) for multiple keys in the same data

Re: How to correct use timeWindow() with DataStream?

Hi, The timestamps of the stream records should be increasing (strict monotonicity is not required, a bit out of orderness can be handled due to watermarks). So, the events should also be generated with increasing timestamps. It looks like your generator generates random dates. I'd also generate

Re: Partial aggregation result sink

Hi, Yes there are plans to support early results and control the result latency / result completeness trade off. However, I cannot say when these features will be available. Best, Fabian 2018-03-19 8:14 GMT+01:00 LiYue : > Hi , > > Is there any plan to adding these

Re: Strange behavior on filter, group and reduce DataSets

Hi, Union is actually a very simple operator (not even an operator in Flink terms). It just merges to inputs. There is no additional logic involved. Therefore, it should also not emit records before either of both ReduceFunctions sorted its data. Once the data has been sorted for the

Re: Queryable State

Those are understandable. I am more interested in a few things ( and may be more that could be added ) * As far as I can understand JM is the SPOF. Does HA become a necessity ? * If there are 2 or more JM could we theoretically have a LB fronting them ? Thus it is a peer to peer access (

Re: How to correct use timeWindow() with DataStream?

If you don't want to partition by key, i.e., have a single result for each time window, you should not use keyBy and an allWindow. However, this will only be executed with a parallelism of 1. 2018-03-19 13:54 GMT+01:00 Felipe Gutierrez : > thanks a lot Fabian, > >

Re: Custom Processing per window

Hi, Data is partitioned by key across machines and state is kept per key. It is not possible to interact with two keys at the same time. Best, Fabian 2018-03-19 14:47 GMT+01:00 Dhruv Kumar : > In other words, while using the Flink streaming APIs, is it possible to > take

Re: HDFS data locality and distribution

Hi Chesnay, Thanks for responding. I managed to resolve the problem last Friday; I had a single datasource for each file, instead of one big datasource for all the files. The reading of the one or two HDFS blocks within each datasource was then distributed to a small percentage of slots

Re: Incremental checkpointing performance

Hi Miyuru, Indeed, the behaviour you observed sounds strange and kind of go against the results Stefan presented in [1]. To see what is going on, can you also share your changes to Flink's configuration, i.e. flink-conf.yaml? Let's first make sure you're really comparing RocksDBStateBackend with

Re: Strange behavior on filter, group and reduce DataSets

Ah, thanks for the update! I'll have a look at that. 2018-03-19 15:13 GMT+01:00 Fabian Hueske : > HI Simone, > > Looking at the plan, I don't see why this should be happening. The pseudo > code looks fine as well. > Any chance that you can create a minimal program to reproduce

Re: Strange behavior on filter, group and reduce DataSets

HI Simone, Looking at the plan, I don't see why this should be happening. The pseudo code looks fine as well. Any chance that you can create a minimal program to reproduce the problem? Thanks, Fabian 2018-03-19 12:04 GMT+01:00 simone : > Hi Fabian, > > reuse is not

Re: Strange behavior on filter, group and reduce DataSets

Hmmm, I still don't see the problem. IMO, the result should be correct for both plans. The data is replicated, filtered, reduced, and unioned. There is nothing in between the filter and reduce, that could cause incorrect behavior. The good thing is, the optimizer seems to be fine. The bad thing

Re: Queryable State

Queryable state is "just" an additional feature on regular keyed state. i.e., the only difference is that you can read the state from an outside application. Besides that it behaves exactly like regular application state Queryable state is (at the moment) designed to be accessible if a job runs.

Re: flink on mesos

Can you elaborate a bit more on what is not working? (please provide a log file or the standard output/error). Also, can you try a newer flink checkount? The start scripts have been merged into a single one for 'flip6' and 'old' - I guess, mesos-appmaster.sh should be the right script for you now.

Re: Queryable State

AFAIK, there have been discussions to replicate state among TMs to speed up recovery (and improve availability). However, I'm not aware of plans to implement that. I don't think serving state while a job is down has been considered yet. 2018-03-19 15:17 GMT+01:00 Vishal Santoshi

Re: Strange behavior on filter, group and reduce DataSets

Hi Fabian, This simple code reproduces the behavior -> https://github.com/xseris/Flink-test-union Thanks, Simone. On 19/03/2018 15:44, Fabian Hueske wrote: Hmmm, I still don't see the problem. IMO, the result should be correct for both plans. The data is replicated, filtered, reduced, and

Re: Custom Processing per window

In other words, while using the Flink streaming APIs, is it possible to take a decision on emitting a particular key based on the state of some other key present in the same window? Thanks! -- Dhruv Kumar PhD Candidate Department of Computer

Re: Queryable State

Are there plans to address all or few of the above apart from the "JM LB not possible" which seems understandable ? On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske wrote: > Queryable state is "just" an additional feature on regular keyed state. > i.e., the only difference is

Re: Flink web UI authentication

Hi Nico Flink does not support username/password authentication as of now. we can do something like this: Here you got a guide from the guys of DigitalOcean about protecting an url with basic authentication:

Re: Restart hook and checkpoint

currently there is only time based way to trigger a checkpoint. based on this discussion, I think flink need to introduce event based way to trigger checkpoint such as restart a task manager should be count as a event. -- Sent from:

Re: Queryable State

Hi Vishal, In general, Queryable State should be ready to use. There are a few things to consider though: - State queries are not synchronized with the application code, i.e., they can happen at the same time. Therefore, the Flink application should not modify objects that have been put into or

Re: Checkpoint is not triggering as per configuration

Hi, Please analyse what was going on the TaskManager and JobManager before this “task is not being executed at the moment”. What is the reason why it is not being executed? Was there some exception? Depending on your setup, you might need to check your stdout/stderr files (if your code is

Re: How to correct use timeWindow() with DataStream?

thanks a lot Fabian, It clarified my way to developing. I am using keyBy, timeWindow, and apply monad operator at the EventTimeStreamExampleJava now. I am generating

Re: Out of the blue: "Cannot use split/select with side outputs"

Update: Even weirder, I stopped Flink (jobmanager and taskmanager) to increase the number of slots and, upon restart, it crashed again and then processed everything just fine. On Mon, Mar 19, 2018 at 3:01 PM, Julio Biason wrote: > Hey guys, > > I got a weird problem

Re: Custom Processing per window

Is there a way I can leverage OperatorState (instead of KeyState) to solve my issue? > On Mar 19, 2018, at 09:00, Fabian Hueske wrote: > > Hi, > > Data is partitioned by key across machines and state is kept per key. It is > not possible to interact with two keys at the

Out of the blue: "Cannot use split/select with side outputs"

Hey guys, I got a weird problem with my pipeline. The pipeline process lines from our logs and generate different metrics based on it (I mean, quite the standard procedure). It uses side outputs for dead letter queues, in case it finds something wrong with the logs and a metric can't be

Re: Flink kafka connector with JAAS configurations crashed

Hi, I'm no expert on Kafka here, but as the tasks are run on the worker nodes (where the TaskManagers are run), please double-check whether the file under /data/apps/spark/kafka_client_jaas.conf on these nodes also contains the same configuration as on the node running the JobManager, i.e. an

Re: CsvSink

Hi Karim, when I was trying to reproduce your code, I got an exception with the name 'table' being used - by replacing it and completing the job with some input, I did see the csv file popping up. Also, the job was crashing when the file 1.txt already existed. The code I used (running Flink

Re: Calling close() on Failure

Hi Gregory, I tried to reproduce the behaviour you described but in my case (Flink 1.5-SNAPSHOT, using the SocketWindowWordCount adapted to let the first flatmap be a RichFlatMapFunction with a close() method), the close() method was actually called on the task manager I did not kill. Since the

Re: Submiting jobs via UI/Rest API

Thanks for reporting these issues, 1. This behaviour is actually intended since we do not spawn any thread that is waiting for the job completion (which may or may not occur eventually). Therefore, the web UI always submits jobs in detached mode and you could not wait for job completion anyway.

Re: Queryable State

Thank you. These do look like show stoppers for us. But again thank you. On Mon, Mar 19, 2018 at 10:31 AM, Fabian Hueske wrote: > AFAIK, there have been discussions to replicate state among TMs to speed > up recovery (and improve availability). > However, I'm not aware of

Re: Flink web UI authentication

Hi Sampath, aside from allowing only certain origins via the configuration parameter "web.access-control-allow-origin", I am not aware of anything like username/password authentication. Chesnay (cc'd) may know more about future plans. You can, however, wrap a proxy like squid around the web UI if

Re: Incremental checkpointing performance

Hi Nico, Thanks for the detailed explanation. The only change I have made in my flink-conf.yaml file is the following. state.backend.fs.checkpointdir: file:///home/ubuntu/tmp-flink-rocksdb The default "state.backend" value is set to filesystem. Removing the env.setStateBackend() method code or

Re: Strange Kafka consumer behaviour

Did you changed the kafka group id? On Mon, Mar 19, 2018 at 9:54 PM, Aljoscha Krettek wrote: > I think it's because it's just taking the state for the existing topics > and starts reading from that. > > A workaround could be to change the uid of the operator to avoid

Running FlinkML ALS with more than two features

Hello Flink experts, I am new to FlinkML and currently playing around with using ALS in a recommender system. In our dataset, we have more than 2 features. When I tried running the example towards the bottom of this page:

Re: CsvSink

Hi Nico, I tried to reproduce your code but registerDataStream keeps failing to register the fields even though i am following your code and the Docs. here is the error [error] found : Symbol [error] required: org.apache.flink.table.expressions.Expression [error]

ListCheckpointed function - what happens prior to restoreState() being called?

Hi all, If I implement ListCheckpointed in a function, is there a guarantee that open() is called before restoreState()? Asking because it doesn’t seem to be the case, and I didn’t notice this being described here:

Re: Flink SSL Setup on a standalone cluster

Hi, When I set ssl.verify.hostname to true , the job fails with SSL handshake exception where it tries to match the IP address instead of the hostname in the certificates. Everything works when I set this to false. The keystore is created with FQDN. The solution of adding all the hostnames and

Let BucketingSink roll file on each checkpoint

The behavior of BucketingSink is not exactly we want. If I understood correctly, when checkpoint requested, BucketingSink will flush writer to make sure data not loss, but will not close file, nor roll new file after checkpoint. In the case of HDFS, if file length is not updated to name

Let BucketingSink roll file on each checkpoint

The behavior of BucketingSink is not exactly we want. If I understood correctly, when checkpoint requested, BucketingSink will flush writer to make sure data not loss, but will not close file, nor roll new file after checkpoint. In the case of HDFS, if file length is not updated to name

Re: Timezone offset on daily time window grouping using SQL

Hi, Thanks for the tip. My team will try to read flink table source code and maybe we clould contribute later. > 在 2018年3月19日，下午6:06，Fabian Hueske 写道： > > Hi, > > Calcite's parser supports this syntax TUMBLE(z, INTERVAL '0.004' SECOND, > TIME '08:00:00') but

Re: Strange Kafka consumer behaviour

Hi Gyula, Are you using Flink 1.4.x, and have partition discovery enabled? If yes, then both the state of previously existing topics, as well as partitions of the newly specified topics will be consumed. Cheers, Gordon On Tue, Mar 20, 2018 at 6:01 AM, Ankit Chaudhary wrote:

Re: Custom Processing per window

How would you start implementing it? Where are you stuck? Did you already try to implement this? > On 18. Mar 2018, at 04:10, Dhruv Kumar wrote: > > Hi > > I am a CS PhD student at UMN Twin Cities. I am trying to use Flink for > implementing some very specific

Re: Partial aggregation result sink

Hi , Is there any plan to adding these features to flink SQL ? Thanks LiYue tig.jd.com > 在 2018年3月14日，上午7:48，Fabian Hueske 写道： > > Hi, > > Chesnay is right. > SQL and Table API do not support early window results and no allowed lateness > to update results with late

Re: Strange behavior on filter, group and reduce DataSets

Any help on this? This thing is very strange..the "manual" union of the output of the 2 datasets is different than the flink-union of them.. Could it be a problem of the flink optimizer? Best, Flavio On Fri, Mar 16, 2018 at 4:01 PM, simone wrote: > Sorry, I

Re: cannot see monitor metrics

This is currently expected behavior. We do not measure incoming records for sources, and outgoing records for sinks as we can't do this in a generalized fashion. See FLINK-7286 . On 19.03.2018 02:37, ?? wrote: I run flink on yarn , Im