Re: RichAsyncFunction Timer Service

t; > > Guowei Ma mailto:guowei@gmail.com>> 于2019年4月28日周日 > 上午9:25写道： > Hi > AFAIK, TimeService is Flink could guarantee the semastics of > Best, > Guowei > > > Mikhail Pryakhin mailto:m.prya...@gmail.com>> > 于2019年4月26日周五 下午7:57写道： >

Re: RichAsyncFunction Timer Service

ees when/and where(at > which stage of the pipeline) the function will actually be executed. This > characteristic doesn't align with TimerService and timely callbacks. > > Best, > > Dawid > > On 19/04/2019 17:41, Mikhail Pryakhin wrote: >> Hello, Flink community! >>

RichAsyncFunction Timer Service

Hello, Flink community! It happens that I need to access a timer service in a RichAsyncFunction implementation. I know it's normally accomplished via StreamingRuntimeContext instance available in a RichFunction, but unfortunately, RichAsyncFunction extending RichFunction overrides

Iterator Data Sync

Hello Flink community! I've come across of employing an "Iterator Data Sync"[1] approach to test output from a streaming pipeline. The pipeline consists of a single ProcessFunction which side-outputs some events. I'd like to collect both the primary and the side-output streams in my test. I do

Side Output from AsyncFunction

Hello Flink experts! My streaming pipeline makes async IO calls via the recommended AsyncFunction. The pipeline evolves and I've encountered a requirement to side output additional events from the function. As it turned out the side output feature is only available in the following functions:

Re: flink-1.6.1 :: job deployment :: detached mode

i Mike,have you tried whether the problem also occurs with Flink 1.6.2? If yes, then please share with us the Flink logs with DEBUG log level to further debug the problem.Cheers,TillOn Fri, Oct 26, 2018 at 5:46 PM Mikhail Pryakhin <m.prya...@gmail.com> wrote:Hi community!Righ after I've upgra

Re: Flink-1.6.1 :: HighAvailability :: ZooKeeperRunningJobsRegistry

> Cheers, > Till > > On Fri, Oct 26, 2018 at 4:31 PM Mikhail Pryakhin <mailto:m.prya...@gmail.com>> wrote: > Hi Andrey, Thanks a lot for your reply! > >> What was the full job life cycle? > > 1. The job is deployed as a YARN cluster with the fo

flink-1.6.1 :: job deployment :: detached mode

Hi community! Righ after I've upgraded flink up to flink-1.6.1 I get an exception during job deployment as a YARN cluster. The job is submitted with zookeper HA enabled, in detached mode. The flink yaml contains the following properties: high-availability: zookeeper

Re: Flink-1.6.1 :: HighAvailability :: ZooKeeperRunningJobsRegistry

; cleaned from Zookeeper upon cancellation. > > Best, > Andrey > > [1] https://issues.apache.org/jira/browse/FLINK-10011 > <https://issues.apache.org/jira/browse/FLINK-10011> > >> On 25 Oct 2018, at 15:30, Mikhail Pryakhin > <mailto:m.prya...@gmail.com>> w

Flink yarn -kill

Hi Flink community, Could you please help me clarify the following question: When a streaming job running in YARN gets manually killed via yarn -kill command is there any way to make a savepoint or other clean up actions before the job manager is killed? Kind Regards, Mike Pryakhin

Flink-1.6.1 :: HighAvailability :: ZooKeeperRunningJobsRegistry

Hi Flink experts! When a streaming job with Zookeeper-HA enabled gets cancelled all the job-related Zookeeper nodes are not removed. Is there a reason behind that? I noticed that Zookeeper paths are created of type "Container Node" (an Ephemeral node that can have nested nodes) and fall back

Re: HighAvailability :: FsNegativeRunningJobsRegistry

yakhin > On 24 Oct 2018, at 16:12, Mikhail Pryakhin wrote: > > Hi guys, > I'm trying to substitute Zookeeper-based HA registry with YARN-based HA > registry. (The idea was taken from the issue > https://issues.apache.org/jira/browse/FLINK-5254 > <https://issues.apa

HighAvailability :: FsNegativeRunningJobsRegistry

Hi guys, I'm trying to substitute Zookeeper-based HA registry with YARN-based HA registry. (The idea was taken from the issue https://issues.apache.org/jira/browse/FLINK-5254) In Flink 1.6.1, there exists an org.apache.flink.runtime.highavailability.FsNegativeRunningJobsRegistry which claims

Re: FlinkKafkaProducerXX

Exactly, at least it's worth mentioning the partitioner used by default in case none was specified in the javadoc, because the default behavior might not seem obvious. Kind Regards, Mike Pryakhin > On 3 Dec 2017, at 22:08, Stephan Ewen wrote: > > Sounds like adding a

FlinkKafkaProducerXX

Hi all, I've just come across a FlinkKafkaProducer misconfiguration issue especially when a FlinkKafkaProducer is created without specifying a kafka partitioner then a FlinkFixedPartitioner instance is used, and all messages end up in a single kafka partition (in case I have a single task

Re: Add custom configuration files to TMs classpath on YARN

k the Taskmanagers > have access to all files (of any type) that are passed using the --ship > command (or in the lib/ folder). > > > On Wed, Sep 27, 2017 at 3:42 PM, Mikhail Pryakhin <m.prya...@gmail.com > <mailto:m.prya...@gmail.com>> wrote: > Hi Nico, >

Re: Add custom configuration files to TMs classpath on YARN

, Mike Pryakhin > On 21 Jun 2017, at 16:55, Mikhail Pryakhin <m.prya...@gmail.com> wrote: > > Hi Nico! > Sounds great, will give it a try and return back with results soon. > > Thank you so much for your help!! > > Kind Regards, > Mike Pryakhin > >>

Re: Add custom configuration files to TMs classpath on YARN

tedCacheTest.java#L99 > > public class MyFunction extends AbstractRichFunction { > private static final long serialVersionUID = 1L; > > @Override > public void open(Configuration conf) throws IOException { > File file = > getRuntimeContext(

Re: Can't get my job restarted on job manager failures

ts, the errors stop as well, I'd say, you > don't have to worry about the messages. > Till (cc'd) may elaborate a bit more on this. > > > Nico > > On Tuesday, 20 June 2017 17:06:00 CEST Mikhail Pryakhin wrote: >> Hi Niko, >> Thanks for your reply! >> >>

Re: Can't get my job restarted on job manager failures

y > > On Tuesday, 20 June 2017 13:23:35 CEST Mikhail Pryakhin wrote: >> Hello, >> >> I'm currently trying to check whether my job is restarted in case of Job >> Manager failure. The job is submitted as a single job on YARN with the >> following options set in th

Can't get my job restarted on job manager failures

Hello, I'm currently trying to check whether my job is restarted in case of Job Manager failure. The job is submitted as a single job on YARN with the following options set in the flink-conf.yaml: restart-strategy: fixed-delay restart-strategy.fixed-delay.attempts: 3

Add custom configuration files to TMs classpath on YARN

Hi all, I run my flink job on yarn cluster and need to supply job configuration parameters via configuration file alongside with the job jar. (configuration file can't be packaged into jobs jar file). I tried to put the configuration file into the folder that is passed via --yarnship option to

Re: RichMapFunction setup method

ature originates from the > Record API, > the predecessor of the current DataSet API. > > Even in the DataSet API you can just pass arguments through the constructor. > Feel free to open a JIRA, just make sure it is a subtask of FLINK-3957. > > On 13.06.2017 16:40, Mikhai

Re: RichMapFunction setup method

> The existing signature for open() is a remnant of the past. > > We currently recommend to pass all arguments through the constructor and > store them in fields. > You can of course also pass a Configuration containing all parameters. > > On 13.06.2017 15:4

RichMapFunction setup method

Hi all! A RichMapFunction [1] provides a very handy setup method RichFunction#open(org.apache.flink.configuration.Configuration) which consumes a Configuration instance as an argument, but this argument doesn't bear any configuration parameters because it is always passed to the method as a

Re: yarnship option

help. —— Mike Pryakhin > On 23 May 2017, at 16:04, Robert Metzger <rmetz...@apache.org> wrote: > > Hi Mike, > > I would recommend you to build a "fat jar" containing your application code > and all required dependencies. > > On Tue, May 23, 20

Re: yarnship option

Pryakhin > On 22 May 2017, at 23:06, Mikhail Pryakhin <m.prya...@cleverdata.ru> wrote: > > Hi Robert! > Thanks a lot for your reply! > > >Can you double check if the > >job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains > >org/apache/flink/streamin

Re: yarnship option

s in the log??From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.prya...@cleverdata.ru> wrote:Hi all! I'm playing with flink streaming j

yarnship option

Hi all! I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out. The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option. Here is the way I run the job: export

Implementing Flink Jobs :: Java-API or Scala-API

Hi all! I'm currently choosing which API to stick with while implementing Flink jobs (primarily streaming jobs). Could you please shed a light on in which API new features are implemented first? Do all Flink features are available in both APIs? Many thanks in advance. Best Regards, Mike