Re: Implementing a low level join

2019-08-14 Thread Felipe Gutierrez
Hi Hequn, I am implementing the broadcast and the regular join. As you said I need different functions. My question is more about if I can have an operator which decides beteween broadcast and regular join dynamically. I suppose I will have to extend the generic TwoInputStreamOperator in Flink.

Path Style Access for S3 compliant object stores

2019-08-14 Thread Achyuth Narayan Samudrala
Hi user group, I am trying to use Flink to write to an S3 object store. I am using the flink-s3-fs-hadoop as a filesystem implementation to interact with this store. How do I provide the s3 property to enable path style access instead of virtual host addressing? I tried looking around in the

Re: How many task managers can Flink efficiently scale to?

2019-08-14 Thread Chad Dombrova
Thanks for the info! It's very helpful. -chad On Sun, Aug 11, 2019 at 4:21 AM Zhu Zhu wrote: > Hi Chad, > > We have (Blink) jobs each running with over 10 thousands of TMs. > In our experience, the main regression caused by large scale TMs is the in > TM allocation stage in ResourceManager,

Re: Making broadcast state queryable?

2019-08-14 Thread Yu Li
Good to know your thoughts and the coming talk in Flink Forward Berlin Oytun, interesting topic and great job! And it's great to hear the voice from application perspective. Could you share, if possible, the reason why you need to open the state to outside instead of writing the result to sink

Re: Changing the way keys are defined breaks savepoints

2019-08-14 Thread Congxian Qiu
Hi, >From what you described, does the key have the same type before and after the change? and for using Lambda expression, maybe the doc[1] can be helpful(especially the limitation) [1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/java_lambdas.html#examples-and-limitations

Flink metrics scope for YARN single job

2019-08-14 Thread Vasily Melnik
Hi, I want to run Flink apps on YARN in single job mode and keep metrics in Graphite. But as i see, the only variable i can use for JobManager scope customization is : https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables Is there any way to

Re: Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" when trying to stop a pipeline

2019-08-14 Thread Kaymak, Tobias
You are right, my bad. We had a company internal java dependency that was referring to an older version of RocksDB. I've spotted it via running mvn dependency:tree while investigating with a colleague. Thank you! On Tue, Aug 13, 2019 at 8:01 PM Yun Tang wrote: > Hi Tobias > > First of all, I

Re: Implementing a low level join

2019-08-14 Thread Hequn Cheng
Hi Felipe, If I understand correctly, you also have to decide whether to broadcast the datastream from the right side before performing the function? One option is you can add a Util method to join dynamically, e.g., Util.joinDynamically(ds1, ds2). In the util method, you can implement your own

Re: Customize file assignments logic in flink application

2019-08-14 Thread Zhu Zhu
Hi Lu, Implementing your own *InputFormat* and *InputSplitAssigner*(which has the interface "InputSplit getNextInputSplit(String host, int taskId)") created by it should work if you want to assign InputSplit to tasks according to the task index and file name patterns. To assign 2 *InputSplit*s in

Re: External classpath

2019-08-14 Thread Biao Liu
Hi Vishwas, > Does my external class path have to be on NFS share ? Can I not have the config directory on each machine in the same location ? It can be local files on each machine. > -C file://home/was/classpathconfig/ I guess the problem is your URI format of local file is not correct. Try

Re: Flink metrics scope for YARN single job

2019-08-14 Thread Biao Liu
Hi Vasily, > Is there any way to distinguish logs from different JobManager running on same host? Do you mean "distinguish metrics from different JobManager running on same host"? I guess there is no other variable you could use for now. But I think it's reasonable to support this requirement.

Re: How can I pass jvm options to flink when started from command line

2019-08-14 Thread Zhu Zhu
Hi Vishwas, If what you want is to set JVM options for Flink client JVM when running jobs with "flink run", I think export the variable 'JVM_ARGS' does help. Thanks, Zhu Zhu Vishwas Siravara 于2019年8月15日周四 上午4:03写道: > I understand that when I run a flink job from command line it forks a jvm >

Update tables env after have been register them

2019-08-14 Thread Andres Angel
Hello everyone, My use case assume that we execute a job where we load from Redis few data and turn it into DS to register them as tables. But, it's possible that after have completed this step the data might change and we may need to read again the data to keep the tables content up to date.

External classpath

2019-08-14 Thread Vishwas Siravara
Hi guys, I m very close to deploying my application in production so I am trying to externalize some of the config files which has to be available on the classpath when I run my application via flink command line interface. >From the flink doc I can add to class path by -C,--classpath

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Bowen Li
Congratulations Andrey! On Wed, Aug 14, 2019 at 10:18 PM Rong Rong wrote: > Congratulations Andrey! > > On Wed, Aug 14, 2019 at 10:14 PM chaojianok wrote: > > > Congratulations Andrey! > > At 2019-08-14 21:26:37, "Till Rohrmann" wrote: > > >Hi everyone, > > > > > >I'm very happy to announce

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Rong Rong
Congratulations Andrey! On Wed, Aug 14, 2019 at 10:14 PM chaojianok wrote: > Congratulations Andrey! > At 2019-08-14 21:26:37, "Till Rohrmann" wrote: > >Hi everyone, > > > >I'm very happy to announce that Andrey Zagrebin accepted the offer of the > >Flink PMC to become a committer of the Flink

Re:[ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread chaojianok
Congratulations Andrey! At 2019-08-14 21:26:37, "Till Rohrmann" wrote: Hi everyone, I'm very happy to announce that Andrey Zagrebin accepted the offer of the Flink PMC to become a committer of the Flink project. Andrey has been an active community member for more than 15 months. He has

Re:[ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread chaojianok
Congratulations Andrey! At 2019-08-14 21:26:37, "Till Rohrmann" wrote: >Hi everyone, > >I'm very happy to announce that Andrey Zagrebin accepted the offer of the >Flink PMC to become a committer of the Flink project. > >Andrey has been an active community member for more than 15 months. He has

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Gary Yao
Congratulations Andrey, well deserved! Best, Gary On Thu, Aug 15, 2019 at 7:50 AM Bowen Li wrote: > Congratulations Andrey! > > On Wed, Aug 14, 2019 at 10:18 PM Rong Rong wrote: > >> Congratulations Andrey! >> >> On Wed, Aug 14, 2019 at 10:14 PM chaojianok wrote: >> >> > Congratulations

How can I pass jvm options to flink when started from command line

2019-08-14 Thread Vishwas Siravara
I understand that when I run a flink job from command line it forks a jvm and runs the main method and the flink related code run in the task manager. So when I say "flink run " the main does not run on JobManager hence it does not take env.java.options set in the flink-conf.yaml as this applies

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Zili Chen
Congratulations Andrey! Best, tison. Till Rohrmann 于2019年8月14日周三 下午9:26写道: > Hi everyone, > > I'm very happy to announce that Andrey Zagrebin accepted the offer of the > Flink PMC to become a committer of the Flink project. > > Andrey has been an active community member for more than 15

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread zhijiang
Congratulations Andrey, great work and well deserved! Best, Zhijiang -- From:Till Rohrmann Send Time:2019年8月14日(星期三) 15:26 To:dev ; user Subject:[ANNOUNCE] Andrey Zagrebin becomes a Flink committer Hi everyone, I'm very happy to

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Kostas Kloudas
Congratulations Andrey! Well deserved! Kostas On Wed, Aug 14, 2019 at 4:04 PM Yun Tang wrote: > > Congratulations Andrey. > > Best > Yun Tang > > From: Xintong Song > Sent: Wednesday, August 14, 2019 21:40 > To: Oytun Tez > Cc: Zili Chen ; Till Rohrmann ; >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Robert Metzger
Congratulations! Very happy to have you onboard :) On Wed, Aug 14, 2019 at 4:06 PM Kostas Kloudas wrote: > Congratulations Andrey! > Well deserved! > > Kostas > > On Wed, Aug 14, 2019 at 4:04 PM Yun Tang wrote: > > > > Congratulations Andrey. > > > > Best > > Yun Tang > >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Dawid Wysakowicz
Congratulations Andrey! Pleasure to have you in the community! Best, Dawid On 14/08/2019 16:06, Kostas Kloudas wrote: > Congratulations Andrey! > Well deserved! > > Kostas > > On Wed, Aug 14, 2019 at 4:04 PM Yun Tang wrote: >> Congratulations Andrey. >> >> Best >> Yun Tang >>

[ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Till Rohrmann
Hi everyone, I'm very happy to announce that Andrey Zagrebin accepted the offer of the Flink PMC to become a committer of the Flink project. Andrey has been an active community member for more than 15 months. He has helped shaping numerous features such as State TTL, FRocksDB release, Shuffle

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Oytun Tez
Congratulations Andrey! I am glad the Flink committer team is growing at such a pace! --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Wed, Aug 14, 2019 at 9:29 AM Zili Chen wrote: > Congratulations Andrey! > > Best, >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Xintong Song
Congratulations Andery~! Thank you~ Xintong Song On Wed, Aug 14, 2019 at 3:31 PM Oytun Tez wrote: > Congratulations Andrey! > > I am glad the Flink committer team is growing at such a pace! > > --- > Oytun Tez > > *M O T A W O R D* > The World's Fastest Human Translation Platform. >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Yun Tang
Congratulations Andrey. Best Yun Tang From: Xintong Song Sent: Wednesday, August 14, 2019 21:40 To: Oytun Tez Cc: Zili Chen ; Till Rohrmann ; dev ; user Subject: Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer Congratulations Andery~! Thank you~

Re: How can I pass multiple java options in standalone mode ?

2019-08-14 Thread Vishwas Siravara
When I add env.java.opts like this env.java.opts:"-Dconfig.resource=qa.conf" I see an error in the log file saying : - Error while trying to split key and value in configuration file /data/flink-1.7.2/conf/flink-conf.yaml:248: "env.java.opts:"-Dconfig.resource=qa.conf"" This is really

Re: How can I pass multiple java options in standalone mode ?

2019-08-14 Thread Aleksandar Mastilovic
It’s a YAML file, so I think you need to do something like env.java.opts: -Dconfig.resource=qa.conf > On Aug 14, 2019, at 11:58 AM, Vishwas Siravara wrote: > > When I add env.java.opts like this > env.java.opts:"-Dconfig.resource=qa.conf" > > I see an error in the log file saying : - Error

Re: How can I pass multiple java options in standalone mode ?

2019-08-14 Thread Vishwas Siravara
Thanks a lot, I fixed that, so now this works when I submit my job with the flink UI but when I submit it via flink run(command line) it does not take this env.java.opts: -Dconfig.resource=qa.conf property . How can I pass the jvm property to flink run which is running standalone without resource

Re: Flink SQL: How to tag a column as 'rowtime' from a Avro based DataStream?

2019-08-14 Thread Timo Walther
Hi Niels, if you are coming from DataStream API, all you need to do is to write a timestamp extractor. When you call: tableEnv.registerDataStream("TestStream", letterStream, "EventTime.rowtime, letter, counter"); The ".rowtime" means that the framework will extract the rowtime from the

Flink SQL: How to tag a column as 'rowtime' from a Avro based DataStream?

2019-08-14 Thread Niels Basjes
Hi, Experimenting with the StreamTableEnvironment I build something like this: DataStream> letterStream = ... tableEnv.registerDataStream("TestStream", letterStream, "EventTime.rowtime, letter, counter"); Because the "EventTime" was tagged with ".rowtime" it is now being used as the rowtime

Customize file assignments logic in flink application

2019-08-14 Thread Lu Niu
Hi, I have a data set backed by a directory of files in which file names are meaningful. folder1 +-file01 +-file02 +-file03 +-file04 I want to control the file assignments in my flink application. For example, when parallelism is 2, worker 1 get file01 and file02 to