Re: StreamingFileSink rolling callback Inbox

2019-09-12 Thread Kostas Kloudas
Hi Anton, First of all, there is this PR https://github.com/apache/flink/pull/9581 that may be interesting to you. Second, I think you have to keep in mind that the hourly bucket reporting will be per-subtask. So if you have parallelism of 4, each of the 4 tasks will report individually that

Re: Problem starting taskexecutor daemons in 3 node cluster

2019-09-12 Thread Komal Mariam
I managed to fix it however ran into another problem that I could appreciate help in resolving. it turns out that the username for all three nodes was different. having the same username for them fixed the issue. i.e same_username@slave-node2-hostname same_username@slave-node3-hostname

SIGSEGV error

2019-09-12 Thread Marek Maj
Hi everyone, Recently we decided to upgrade from flink 1.7.2 to 1.8.1. After an upgrade our task managers started to fail with SIGSEGV error from time to time. In process of adjusting the code to 1.8.1, we noticed that there were some changes around TypeSerializerSnapshot interface and its

Re: suggestion of FLINK-10868

2019-09-12 Thread Anyang Hu
Thanks Till, I will continue to follow this issue and see what we can do. Best regards, Anyang Till Rohrmann 于2019年9月11日周三 下午5:12写道: > Suggestion 1 makes sense. For the quick termination I think we need to > think a bit more about it to find a good solution also to support strict > SLA

RE: Filter events based on future events

2019-09-12 Thread Theo Diefenthal
Hi Fabian, Thank’s for sharing your thought’s. I’ll give it a try. Best regards Theo From: Fabian Hueske Sent: Mittwoch, 11. September 2019 09:55 To: theo.diefent...@scoop-software.de Cc: user Subject: Re: Filter events based on future events Hi Theo, I would

[SURVEY] How many people are using customized RestartStrategy(s)

2019-09-12 Thread Zhu Zhu
Hi everyone, I wanted to reach out to you and ask how many of you are using a customized RestartStrategy[1] in production jobs. We are currently developing the new Flink scheduler[2] which interacts with restart strategies in a different way. We have to re-design the interfaces for the new

Re: suggestion of FLINK-10868

2019-09-12 Thread Peter Huang
Hi Anyang and Till, I think we agreed on making the interval configurable in this case. Let me revise the current PR. You can review it after that. Best Regards Peter Huang On Thu, Sep 12, 2019 at 12:53 AM Anyang Hu wrote: > Thanks Till, I will continue to follow this issue and see what we

Flink web ui authentication using nginx

2019-09-12 Thread Kumar Bolar, Harshith
Hi all, I'm trying to add authentication to the web dashboard using `nginx`. Flink's `rest.port` is set to `8081`, connection to this port is disabled by firewall. I'm using `nginx` to listen to requests on port 8080 and redirect to port 8081 them with username/password authentication (Port

Re: [SURVEY] How many people are using customized RestartStrategy(s)

2019-09-12 Thread Zhu Zhu
Thanks Oytun for the reply! Sorry for not have stated it clearly. When saying "customized RestartStrategy", we mean that users implement an *org.apache.flink.runtime.executiongraph.restart.RestartStrategy* by themselves and use it by configuring like "restart-strategy:

Re: [SURVEY] How many people are using customized RestartStrategy(s)

2019-09-12 Thread Oytun Tez
Hi Zhu, We are using custom restart strategy like this: environment.setRestartStrategy(failureRateRestart(2, Time.minutes(1), Time.minutes(10))); --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Thu, Sep 12, 2019 at

Re: How to handle avro BYTES type in flink

2019-09-12 Thread Catlyn Kong
Turns out there was some other deserialization problem unrelated to this. On Mon, Sep 9, 2019 at 11:15 AM Catlyn Kong wrote: > Hi fellow streamers, > > I'm trying to support avro BYTES type in my flink application. Since > ByteBuffer isn't a supported type, I'm converting the field to an >

Re: Kafka Schema registry

2019-09-12 Thread Lasse Nedergaard
Hi Elias Thanks for letting me know. I have found it but we also need the option to register Avro Schema’s and use the registry when we write to Kafka. So we will create a serialisation version and when it works implement it into Flink and create a pull request for the community. Med venlig

Re: Kafka Schema registry

2019-09-12 Thread Elias Levy
Just for a Kafka source: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#the-deserializationschema - There is also a version of this schema available that can lookup the writer’s schema (schema which was used to write the record) in Confluent Schema

Re: How do I start a Flink application on my Flink+Mesos cluster?

2019-09-12 Thread Felipe Gutierrez
Thanks Gary, I am compiling a new version of Mesos and when I test it again I will reply here if I found an error. On Wed, 11 Sep 2019, 09:22 Gary Yao, wrote: > Hi Felipe, > > I am glad that you were able to fix the problem yourself. > > > But I suppose that Mesos will allocate Slots and Task

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

2019-09-12 Thread Zili Chen
Thanks a lot everyone for the warm welcome. Happy Mid-autumn Festival! Best, tison. Leonard Xu 于2019年9月12日周四 上午11:05写道: > Congratulations Zili Chen ! ! > > Best, > Leonard Xu > > On 2019年9月12日, at 上午11:02, Yun Tang wrote: > > > > Congratulations Zili > > > > Best > > Yun Tang > >

externalizing config flies for flink class loader

2019-09-12 Thread Vishwas Siravara
I have a standalone cluster. I have added my own library(jar file) to the lib/ folder in flink . I submit my job from cli after I start the cluster. Now I want to externalize a property file which has to be read by this library. Since this library is loaded by flink's classloader and not the

Jobsubmission fails in Flink 1.7.1 High Availability mode

2019-09-12 Thread Bajaj, Abhinav
Hi, I came across an issue during job submission via Flink Cli Client with Flink 1.7.1 in high availability mode. Setup: Flink version:: 1.7.1 Cluster:: K8s Mode:: High availability with 2 jobmanagers CLI Command ./bin/flink run -d -c MyExample /myexample.jar The CLI runs inside a K8s job and

?????? blinkSQL????????????????????state??

2019-09-12 Thread ????
tableEnv.registerDataStream("testCountTable", waterMarkStream, 'curuserid,'timelong,'rowtime.rowtime) val result = tableEnv.sqlQuery(s"SELECT COUNT(0) as pv,COUNT(distinct curuserid)" + s" as uv,TUMBLE_END(rowtime, INTERVAL '10' MINUTE) FROM

Re: blinkSQL架构会自动清理过期的state吗

2019-09-12 Thread LakeShen
Hi 守护, *可*以先确保你的watermark是否一直在更新。 然后可以把窗口时间间隔设置小一点,然后在运行程序,看看checkpoint 的状态文件是否会自动清理。 守护 <346531...@qq.com> 于2019年9月12日周四 下午2:35写道: > 下面是我要执行的代码,麻烦帮看一下: > > > tableEnv.registerDataStream("testCountTable", > waterMarkStream, 'curuserid,'timelong,'rowtime.rowtime) > > val result =

?????? blinkSQL????????????????????state??

2019-09-12 Thread ????
blinkSQL??watermarkcheckpointflinkSQLblinkSQL?? ---- ??:"LakeShen"

Re: 如何优化flink内存?

2019-09-12 Thread cai yi
你好, 没有明白"对每条数据使用slideWindow得出这条数据对应的90天的窗口结束时间"这一步是怎么做的, 滑动窗口中一条数据最大可以包含在size/slide个window中,所以这个结束时间是怎么取值的呢? 在 2019/9/5 下午4:08,“陈赋赟” 写入: HI 我在项目中有遇到过类似的情况,我说下我的想法和思路。 伊始是需要统计90天事件窗口中用户浏览事件总数,如果是在近30天内有浏览事件则累加1次,在30天内没有浏览事件但在 30天 ~ 90天内有其他浏览事件则记0次(需求比较奇葩),我们使用了滑动窗口(长度90天 步长1天