Re: Flink batch processing fault tolerance

his is indeed true. We had some plans for how to resolve this but > they never materialised because of the focus on Stream Processing. We might > unite the two in the future and then you will get fault-tolerant > batch/stream processing in the same API. > > > > Best, > > Aljoscha >

Flink batch processing fault tolerance

Hi, all: I'm learning flink's doc and curious about the fault tolerance of batch process jobs. It seems that when one of task execution fails, the whole job will be restarted, is it true? If so, isn't it impractical to deploy large flink batch jobs? -- Liu, Renjie Software Engineer, MVAD

Re: Question about the checkpoint mechanism in Flink.

Essentially you are right, but the snapshot commit process is asynchronous. That's what you have to pay for exactly once semantics. Li Wang 于2016年11月1日周二 下午3:05写道： > Hi all, > > I have a question regarding to the state checkpoint mechanism in Flink. I > find the statement

Re: Question about the checkpoint mechanism in Flink.

Sorry the incorrect reply, please ignore this. On Tue, Nov 1, 2016 at 8:47 PM Renjie Liu <liurenjie2...@gmail.com> wrote: > Essentially you are right, but the snapshot commit process is > asynchronous. That's what you have to pay for exactly once semantics. > > Li Wang <w

Re: Question about the checkpoint mechanism in Flink.

Hi, Till: By operator with multiple inputs, do you mean inputs from multiple subtasks? On Tue, Nov 1, 2016 at 8:56 PM Till Rohrmann wrote: > Hi Li, > > the statement refers to operators with multiple inputs (two in this case). > With the current implementation you will

Re: Question about the checkpoint mechanism in Flink.

Thanks for the reply. On Wed, Nov 2, 2016 at 5:19 PM Till Rohrmann <trohrm...@apache.org> wrote: > Yes you're right. Whenever you have multiple input channels which could > also be the case if you do a repartitioning between two mappers. > > On Tue, Nov 1, 2016 at 11

Flink stream job change and recovery

Hi, all: It seems that flink's checkpoint mechanism saves state per partition. However, if I want to change configuration of the job and resubmit the job, I can not recover from last checkpoint, right? Say I change the parallelism of kafka consumer and partitions assigned to each subtask will be

Re: Flink stream job change and recovery

ferent parallelism. > > 2016-11-04 11:24 GMT+01:00 Renjie Liu <liurenjie2...@gmail.com>: > > Hi, all: > It seems that flink's checkpoint mechanism saves state per partition. > However, if I want to change configuration of the job and resubmit the job, > I can not reco

Re: Question about the checkpoint mechanism in Flink.

t; > With multiple inputs I referred to a coFlatMap operator or a join operator > which have both two inputs. > > Cheers, > Till > > On Tue, Nov 1, 2016 at 3:29 PM, Renjie Liu <liurenjie2...@gmail.com> > wrote: > > Hi, Till: > By operator with multiple inputs, do you me

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

Great! On Wed, Dec 14, 2016 at 1:30 AM Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > I think it's worth to announce this via news list. :) > > On 12/13/16 7:32 AM, Robert Metzger wrote: > > The commun...@flink.apache.org

Re: Kafka 09 consumer does not commit offsets

I don’t think this should be a issue > for the 09 consumer, since we’re solely relying on the Kafka client’s own > implementation to do the auto offset committing. > > Cheers, > Gordon > > > On January 9, 2017 at 7:55:33 PM, Timo Walther (twal...@apache.org) wrote: > >

Kafka 09 consumer does not commit offsets

Hi, all: I'm using flink 1.1.3 and kafka consumer 09. I read its code and it says that the kafka consumer will turn on auto offset commit if checkpoint is not enabled. I've turned off checkpoint and it seems that kafka client is not committing to offsets to kafka? The offset is important for

Re: Task manager number mismatch container number on mesos

gt; On Thu, Mar 23, 2017 at 11:38 AM, Renjie Liu <liurenjie2...@gmail.com> > wrote: > > Permanent. I've waited for several minutes and the task manager is still > lost. > > On Thu, Mar 23, 2017 at 6:34 PM Ufuk Celebi <u...@apache.org> wrote: > > When it happ

Re: Re: Just do a survey, how many people give up the storm and turn to Flink ?

We are running both systems. Old existing systems are still running on storm and new jobs are all running on flink. On Mon, Aug 21, 2017 at 3:06 AM kaniska Mandal wrote: > Storm supports 'exactly once' - > http://storm.apache.org/releases/current/Trident-tutorial.html

Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)

Hi, does this include the FLIP6? On Tue, May 9, 2017 at 2:29 AM Stephan Ewen wrote: > Did a quick test: Simply adding the > "org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer" > helps with NOTICE files, > but does not add the required BSD licence copies.

Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)

ue, May 9, 2017 at 10:02 AM, Renjie Liu <liurenjie2...@gmail.com> > wrote: > >> Stephan: >> This means that we can use FLIP 6 for basic functionality now? Any >> document for setting up that? Will FLIP 6 be finished in release 1.3? >> >> On Tu

Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)

Stephan: This means that we can use FLIP 6 for basic functionality now? Any document for setting up that? Will FLIP 6 be finished in release 1.3? On Tue, May 9, 2017 at 3:43 PM Stephan Ewen <se...@apache.org> wrote: > @Renjie Liu: A good part of the FLIP-6 code is in there, bu

Re: ConnectedStream keyby issues

@Jason I think there's a mistake in your explanation since each task in the task manager has its own copy of an operator instance, so the tuple may not be shared. State is a great solution but I think that's not the root cause. @Tarek What's the parallelism of your data stream? I think the reason

Re: ConnectedStream keyby issues

Jason's solution is right, l'm just clarifying the mistake in the explanation. Tarek khal 于2017年5月19日周五 下午7:11写道： > Hello Renjie, > > Yes, the parallelism is 1. what should i do pls ? > > Regards, > > > > -- > View this message in context: >

Re: ConnectedStream keyby issues

Even if you increase the operator parallelism, you can still use the state operation. On Fri, May 19, 2017 at 7:47 PM Tarek khal wrote: > If I increase the parallelism operator, I risk losing shared state solution > or it has nothing to do. > And if it's going to

Re: Blobstore exceptions.

You need to check that whether you disk is full. On Thu, Jun 14, 2018 at 2:15 PM Lasse Nedergaard wrote: > Hi. > > We sometimes see job fails with a blob store exception, like the one > below. Anyone has an idea why we get them, and how to avoid them?. > In this case the job has run without any

Re: Flink(1.5.0) SQL stream non-windowed inner join

Could you please paste your sql here? On Thu, Jun 14, 2018 at 2:54 PM 徐林彬 wrote: > Hi all > I'm trying to prove flink sql stream non-windowed inner join with flink > 1.5.0, but it failed. > Then i tried flink test >

[DISCUSS] FLIP-6 Problems

Hi: We've deployed flink 1.5.0 and tested the new cluster manager, it's really great for flink to be elastic. However we've also found some problems that blocks us from deploying it to production environment. 1. Task manager isolation. Currently flink allows different jobs to execute on same

Re: not found: type CustomWatermarkEmitter

It seems that it's related with your development environment settings. On Sun, Jul 29, 2018 at 4:39 PM Mich Talebzadeh wrote: > Hi, > > I have written a simple test program as below > > import java.util.Properties > import java.util.Arrays > import

Re: Order of events in a Keyed Stream

Hi, Another way to ensure order is by adding a logical version number for each message so that earlier version will not override later version. Timestamp depends on your ntp server works correctly. On Sun, Jul 29, 2018 at 3:52 PM Niels Basjes wrote: > Hi, > > The basic thing is that you will

Re: Flink on Mesos: containers question

ly Docker container is the one that runs Job Manager. > > > > What am I missing? > > > > Thanks, > > Alex > > > > *From:* Renjie Liu [mailto:liurenjie2...@gmail.com] > *Sent:* Friday, July 20, 2018 8:56 PM > *To:* Till Rohrmann > *Cc:* NEKRASSOV, ALEXE

Re: Flink resource manager unable to connect to mesos after restart

here is an even older > ticket describing the same problem: > > https://issues.apache.org/jira/browse/FLINK-7470 > > One of them should be closed. > > Best, > Gary > > > On Thu, Jul 19, 2018 at 10:45 AM, Renjie Liu > wrote: > >> Attached is job ma

Re: Working out through individual messages in Flink

Hi, Mich: You can add write a sink function for that. On Mon, Jul 30, 2018 at 2:58 PM Mich Talebzadeh wrote: > > Hi, > > I have a Kafka topic that transmits 100 security prices ever 2 seconds. > > In Spark streaming I go through the RDD and walk through rows one by one > and check prices > In

Re: Managed Keyed state update

Hi, Alexey: It depends on the state backend you use. If you use heap memory backend, then you don't need to do put again. However, if you use rocksdb state backend, then you need to do the put again so that it will be saved by the checkpoint. On Tue, Aug 14, 2018 at 4:58 AM Alexey Trenikhun

Re: Hive Integration

Hi, yuvraj: Do you mean querying hive with sql? Or anything else? On Tue, Aug 14, 2018 at 3:52 AM yuvraj singh <19yuvrajsing...@gmail.com> wrote: > I want to know ,if FLink have support for hive . > > Thanks > Yubraj Singh > -- Liu, Renjie Software Engineer, MVAD

Re: [DISCUSS] Remove the slides under "Community & Project Info"

Hi, Stephan: Can we put project wiki in some place? I think it's a great place to find flip lists On Mon, Aug 27, 2018 at 9:56 AM vino yang wrote: > +1 > The reason is the same as Hequn, because we have given a link to > SlideShare under the "Flink Forward" section. > > Thanks, vino. > > Hequn

Re: Flink on Mesos: containers question

Hi， Alexei: What you paste is expected behavior. Jobmanager, two task managers each should run in a docker instance. 13276 is should be the process of job manager, and it's the same process as 789. They have different processes id because in show them in different namesapces(that's a concept in

Flink resource manager unable to connect to mesos after restart

Hi, all: I'm testing flink 1.5.0 and find that flink mesos resource manager unable to connect to mesos after restart. Have you seen this happenen? -- Liu, Renjie Software Engineer, MVAD

Re: Flink resource manager unable to connect to mesos after restart

are able to re-produce this reliably, can you post the jobmanager > logs? > > Best, > Gary > On Wed, Jul 18, 2018 at 10:33 AM, Renjie Liu > wrote: > >> Hi, all: >> >> I'm testing flink 1.5.0 and find that flink mesos resource manager unable >> to conne

Re: job status monitor

You can use rest api here https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/rest_api.html On Wed, Jul 25, 2018 at 5:18 PM 从六品州同 <26304...@qq.com> wrote: > dear all： > Is there a notification mechanism for Flink? When job's status changes, > such as restart, failure, notify

Re: Why FlinkKafkaConsumer doesn't subscribe to topics?

ote: > > Hi Renjie, > > 1. For what I could grasp from Kafka docs, you can subscribe and still use > poll() to capture a specific offset. But I just read the starting point of > it and didn't go deep into it. > > 2. Currently, Flink 1.4.2, Kafka 0.10.1 and the FlinkKafkaCo

Re: How to customize schedule mode and result partition type?

Hi: 1. I'm wondering in which case you need this LAZY_FROM_SROUCE schedule mode? 2. It seems that this can't be changed for streaming jobs, and it's worth a discussion for an api that changes the schedule mode On Fri, Sep 7, 2018 at 6:25 PM 陈梓立 wrote: > Sorry to attach this message here, but if

Re: Does Queryable State only support K/V queries not SQL?

Hi, kant: Full support for sql is not easy tuo support. A simple KV storage is not enough to build a RDBMS. On Sun, Mar 4, 2018 at 1:42 AM kant kodali wrote: > Hi Fabian, > > Does it make sense to have it on the roadmap? that way external > applications can do ad-hoc queries

Re: Partitions vs. Subpartitions

Hi, Chris: Where are these words from? Are they from flink source code? On Wed, Oct 10, 2018 at 10:18 PM Chris Miller wrote: > Hi, > > what's the difference between partitions and subpartitions? > > > > Thanks. > > > > CM > > -- Renjie Liu Software Engineer, MVAD

Re: Why FlinkKafkaConsumer doesn't subscribe to topics?

Hi, Julio: 1. Flink doesn't use subscribe because it needs to control partition assignment itself, which is important for implementing exactly once. 2. Can you share the versions you are using, including kafka, kafka client, flink? We are also use flink kafka consumer and we can monitor it

Re: Trying to figure out why a slot takes a long time to checkpoint

Hi, Julio: This happens frequently? What state backend do you use? The async checkpoint duration and sync checkpoint duration seems normal compared to others, it seems that most of the time are spent acking the checkpoint. On Sun, Sep 16, 2018 at 9:24 AM vino yang wrote: > Hi Julio, > > Yes, it

Re: Task managers run on separate nodes in a cluster

Hi, Martin: I think a better solution would be to set the number of cores of each container equals to that of a physical server if this mesos cluster is dedicated to your flink cluster. On Mon, Sep 17, 2018 at 5:28 AM Martin Eden wrote: > Hi Till, > > I was able to use

Re: Trying to figure out why a slot takes a long time to checkpoint

ing so long? > > The checkpoint size of task #8 is twice as big as the second biggest > checkpoint. But this should not cause an increase in checkpoint time of a > factor of 8. > > Cheers, > Till > > On Mon, Sep 17, 2018 at 5:25 AM Renjie Liu > wrote: > >>