Re: assignTimestampsAndWatermarks not work after KeyedStream.process

Hi, After keyby maybe only some of BoundedOutOfOrdernessTimestampExtractors could receive the elements(trip). If that is the case BoundedOutOfOrdernessTimestampExtractor, which does not receive element would not send the WM. Since that the timeWindowAll operator could not be triggered. You could

Re: Identify orphan records after joining two streams

Thank you Hecheng. I just tried to use Table API as your suggestion, and it almost worked (it worked with two issues here below): - I only get the output when my event-time watermark goes pass the end of the tumbling window. But, because I know that there are maximum 2 records per window (one

host name in REST API

Hi everyone, I am having a problem and hope some one can provide insights here. I am trying to make use of the queryable state feature. due to https://issues.apache.org/jira/browse/FLINK-10225, I have to know on which task managers the job is deployed. My idea is to use REST API

Re: assignTimestampsAndWatermarks not work after KeyedStream.process

I don't think it is the watermark. I see the same watermarks from the two versions of code. The processing on the keyed stream doesn't change event time at all. I can simply change my code to use `map` on the keyed stream to return back the input data, so that the window operator receives the

PatternFlatSelectAdapter - Serialization issue after 1.8 upgrade

Hi all, We are just migration from 1.6 to 1.8. I encountered a serialization error which we didn't have before if memory serves: The implementation of the *PatternFlatSelectAdapter* is not serializable. The object probably contains or references non serializable fields. The method below simply

Re: Do flink have plans to support Deep Learning?

Thanks Suneel for the information. I see https://github.com/FlinkML/flink-tensorflow/wiki has support for scala api. Can you please confirm if java support is there for the same? Regards, Manju On Thu, Apr 18, 2019 at 7:36 PM Suneel Marthi wrote: > that's a very open-ended question. > >

Re: Do flink have plans to support Deep Learning?

that's a very open-ended question. There's been enough work done on using Flink for Deep Learning model inference - with TensorFlow (look at Eron Wright's Flink-Tensorflow project), with Amazon Sagemaker (i have code for that) or work from LightBend on Flink Model serving. So yes, there's enuf

Do flink have plans to support Deep Learning?

Hello, Do flink have any plans to support Deep Learning, in near future? Thanks, Manju

Submit user application jar in a programming way

Hi, Recently I met a issue which relevant to the class loader of user application and the flink ’s own class loader. I want to solve this issue[1]byfind out the right class loader of the user jar. If anyone colud show me the Flink production’s start up class loader sequence. I would

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Thanks for starting this discussion Jeff. I can see the need for additional hooks for third party integrations. The thing I'm wondering is whether we really need/want to expose a JobListener via the ExecutionEnvironment. The ExecutionEnvironment is usually used by the user who writes the code and

Re: Service discovery on YARN - find out which port was dynamically assigned to the JobManager Web Interface

Hi Olivier, since version 1.8 you can set the rest bind port via `rest.bind-port` to a single port or a range. This will now be respected by Yarn deployments. With the next bug fix release of 1.7 you can do the same with `rest.port` but this option only accepts a single port (might lead to port

Re: Fast restart of a job with a large state

Hi, If rescaling is the problem, let me clarify that you can currently rescale from savepoints and all types of checkpoints (including incremental). If that was the only problem, then there is nothing to worry about - the documentation is only a bit conservative about this because we will not

Re: Fast restart of a job with a large state

Hi, Have you tried task local recovery [1]? [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#retained-checkpoints Best, Paul Lam > 在 2019年4月17日，17:46，Sergey Zhemzhitsky 写道： > > Hi Flinkers, > > Operating different flink jobs I've discovered that job

Re: Fast restart of a job with a large state

The URL in my previous mail is wrong, and it should be: https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#task-local-recovery Best, Paul Lam >

Re: Flink Metrics

Hi Brian, You can implement a new org.apache.flink.metrics.reporter.MetricReporter as you like and register it to flink in flink conf. e.g. metrics.reporters:my_reporter metrics.reporter.my_other_reporter.class: xxx metrics.reporter.my_other_reporter.config1: yyy

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Hi Jeff, I personally like this proposal. From the perspective of programmability, the JobListener can make the third program more appreciable. The scene where I need the listener is the Flink cube engine for Apache Kylin. In the case, the Flink job program is embedded into the Kylin's

Flink Metrics

Hi, I am trying to profile my Flink job. For example I want to output the results of the TaskIOMetricGroup to a log file. Does anyone know if there is a way to access this object at runtime and execute the methods to get the data from within my user code that I submit to the Flink to start a