Re: [POLL] Who still uses Java 7 with Flink ?

[Jark Wu]

+1 for dropping Java 7

2017-07-13 9:34 GMT+08:00 ☼ R Nair (रविशंकर नायर) <
ravishankar.n...@gmail.com>:

> +1 for dropping Java 1.7.
>
> On Wed, Jul 12, 2017 at 9:10 PM, Kurt Young  wrote:
>
>> +1 for droppint Java 7, we have been using Java 8 for more than one year
>> in Alibaba and everything work fine.
>>
>> Best,
>> Kurt
>>
>> On Thu, Jul 13, 2017 at 2:53 AM, Bowen Li 
>> wrote:
>>
>>> +1 for dropping Java 7
>>>
>>> On Wed, Jul 12, 2017 at 9:04 AM, Gyula Fóra 
>>> wrote:
>>>
>>> > +1 for dropping 1.7 from me as well.
>>> >
>>> > Gyula
>>> >
>>> > On Wed, Jul 12, 2017, 17:53 Ted Yu  wrote:
>>> >
>>> > > +1 on dropping support for Java 1.7
>>> > >
>>> > >  Original message 
>>> > > From: Robert Metzger 
>>> > > Date: 7/12/17 8:36 AM (GMT-08:00)
>>> > > To: d...@flink.apache.org
>>> > > Cc: user 
>>> > > Subject: Re: [POLL] Who still uses Java 7 with Flink ?
>>> > >
>>> > > +1 to drop Java 7 support
>>> > >
>>> > > I believe that we can move to Java 8 for the argument you've stated.
>>> > > ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will
>>> > > require it as well.
>>> > >
>>> > > On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko
>>> >> > >
>>> > > wrote:
>>> > >
>>> > >> Hi,
>>> > >>
>>> > >> I would be in favor of dropping Java 7 as we don't use it in our
>>> hadoop
>>> > >> infra (and we are a bank). Also, Spark 2.2 has been released today,
>>> > >> which doesn't
>>> > >> support Java 7 >> ses/spark-release-2-2-0.
>>> > html
>>> > >> >
>>> > >> anymore, and Flink should not lack behind :-)
>>> > >>
>>> > >> Cheers, Fokko
>>> > >>
>>> > >> 2017-07-12 15:56 GMT+02:00 Stephan Ewen :
>>> > >>
>>> > >> > Bumping this thread again.
>>> > >> >
>>> > >> > There are several strong points for dropping Java 7 support, apart
>>> > from
>>> > >> the
>>> > >> > fact that it is not maintained
>>> > >> >
>>> > >> >   - We could really use the Java 8 default methods feature in
>>> > >> interfaces to
>>> > >> > evolve the API without breaking backwards compatibility
>>> > >> >
>>> > >> >   - Easier build setup for Scala 2.12 (which requires Java 8), no
>>> need
>>> > >> to
>>> > >> > manage the tricky combinations of Java / Scala versions
>>> > >> >
>>> > >> >   - Ability to use vanilla Akka (rather than Flakka) which
>>> requires
>>> > >> Java 8.
>>> > >> > - Fewer problems for users that use Akka in the Flink
>>> applications
>>> > >> > - Flakka currently does not support Scala 2.12
>>> > >> > - Newer Akka versions shade protobuf, which is important
>>> > >> >
>>> > >> > I think these together make a pretty good case for bumping the
>>> > required
>>> > >> > Java version to Java 8.
>>> > >> >
>>> > >> > It would just help both Flink users (dependency management, Scala
>>> > >> versions)
>>> > >> > and developers (build simplification) a lot.
>>> > >> > Unless we see users stepping forward and making a case that it
>>> will be
>>> > >> > impossible for them to upgrade to Java 8, I suggest to go forward
>>> with
>>> > >> > this.
>>> > >> >
>>> > >> > Best,
>>> > >> > Stephan
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai 
>>> > wrote:
>>> > >> >
>>> > >> > > +1
>>> > >> > >
>>> > >> > > There are several high impacts security vulnerabilities in JDK
>>> 7 and
>>> > >> will
>>> > >> > > not be addressed.
>>> > >> > >
>>> > >> > > As a result we completely moved away from JDK 7.
>>> > >> > >
>>> > >> > > +1 on separating the tasks of supporting Scala 2.12 and JDK 8
>>> in two
>>> > >> > steps.
>>> > >> > >
>>> > >> > >
>>> > >> > > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan 
>>> > wrote:
>>> > >> > >
>>> > >> > > > Is this not two different issues?
>>> > >> > > > - adding builds for Scala 2.12
>>> > >> > > > - upgrading to Java version 1.8
>>> > >> > > >
>>> > >> > > > It may be time to switch, but I haven’t seen anything in
>>> > FLINK-5005
>>> > >> > which
>>> > >> > > > prevents simply adding Scala 2.12 to our supported build
>>> matrix
>>> > and
>>> > >> > > > continuing to build 2.10 / 2.11 against Java 1.7.
>>> > >> > > >
>>> > >> > > > Greg
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger <
>>> > rmetz...@apache.org>
>>> > >> > > wrote:
>>> > >> > > > >
>>> > >> > > > > Hi all,
>>> > >> > > > >
>>> > >> > > > > as promised in March, I want to revive this discussion!
>>> > >> > > > >
>>> > >> > > > > Our users are begging for Scala 2.12 support [1], migration
>>> to
>>> > >> Akka
>>> > >> > 2.4
>>> > >> > > > would solve a bunch of shading / dependency issues (Akka 2.4
>>> will
>>> > >> > remove
>>> > >> > > > Akka's protobuf dependency [2][3]) and generally Java 8's new
>>> > >> language
>>> > >> > > > features all speak for dropping Java 7.
>>> 

Re: [POLL] Who still uses Java 7 with Flink ?

[रविशंकर नायर]

+1 for dropping Java 1.7.

On Wed, Jul 12, 2017 at 9:10 PM, Kurt Young  wrote:

> +1 for droppint Java 7, we have been using Java 8 for more than one year
> in Alibaba and everything work fine.
>
> Best,
> Kurt
>
> On Thu, Jul 13, 2017 at 2:53 AM, Bowen Li  wrote:
>
>> +1 for dropping Java 7
>>
>> On Wed, Jul 12, 2017 at 9:04 AM, Gyula Fóra  wrote:
>>
>> > +1 for dropping 1.7 from me as well.
>> >
>> > Gyula
>> >
>> > On Wed, Jul 12, 2017, 17:53 Ted Yu  wrote:
>> >
>> > > +1 on dropping support for Java 1.7
>> > >
>> > >  Original message 
>> > > From: Robert Metzger 
>> > > Date: 7/12/17 8:36 AM (GMT-08:00)
>> > > To: d...@flink.apache.org
>> > > Cc: user 
>> > > Subject: Re: [POLL] Who still uses Java 7 with Flink ?
>> > >
>> > > +1 to drop Java 7 support
>> > >
>> > > I believe that we can move to Java 8 for the argument you've stated.
>> > > ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will
>> > > require it as well.
>> > >
>> > > On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko
>> > > >
>> > > wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> I would be in favor of dropping Java 7 as we don't use it in our
>> hadoop
>> > >> infra (and we are a bank). Also, Spark 2.2 has been released today,
>> > >> which doesn't
>> > >> support Java 7 > .
>> > html
>> > >> >
>> > >> anymore, and Flink should not lack behind :-)
>> > >>
>> > >> Cheers, Fokko
>> > >>
>> > >> 2017-07-12 15:56 GMT+02:00 Stephan Ewen :
>> > >>
>> > >> > Bumping this thread again.
>> > >> >
>> > >> > There are several strong points for dropping Java 7 support, apart
>> > from
>> > >> the
>> > >> > fact that it is not maintained
>> > >> >
>> > >> >   - We could really use the Java 8 default methods feature in
>> > >> interfaces to
>> > >> > evolve the API without breaking backwards compatibility
>> > >> >
>> > >> >   - Easier build setup for Scala 2.12 (which requires Java 8), no
>> need
>> > >> to
>> > >> > manage the tricky combinations of Java / Scala versions
>> > >> >
>> > >> >   - Ability to use vanilla Akka (rather than Flakka) which requires
>> > >> Java 8.
>> > >> > - Fewer problems for users that use Akka in the Flink
>> applications
>> > >> > - Flakka currently does not support Scala 2.12
>> > >> > - Newer Akka versions shade protobuf, which is important
>> > >> >
>> > >> > I think these together make a pretty good case for bumping the
>> > required
>> > >> > Java version to Java 8.
>> > >> >
>> > >> > It would just help both Flink users (dependency management, Scala
>> > >> versions)
>> > >> > and developers (build simplification) a lot.
>> > >> > Unless we see users stepping forward and making a case that it
>> will be
>> > >> > impossible for them to upgrade to Java 8, I suggest to go forward
>> with
>> > >> > this.
>> > >> >
>> > >> > Best,
>> > >> > Stephan
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai 
>> > wrote:
>> > >> >
>> > >> > > +1
>> > >> > >
>> > >> > > There are several high impacts security vulnerabilities in JDK 7
>> and
>> > >> will
>> > >> > > not be addressed.
>> > >> > >
>> > >> > > As a result we completely moved away from JDK 7.
>> > >> > >
>> > >> > > +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in
>> two
>> > >> > steps.
>> > >> > >
>> > >> > >
>> > >> > > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan 
>> > wrote:
>> > >> > >
>> > >> > > > Is this not two different issues?
>> > >> > > > - adding builds for Scala 2.12
>> > >> > > > - upgrading to Java version 1.8
>> > >> > > >
>> > >> > > > It may be time to switch, but I haven’t seen anything in
>> > FLINK-5005
>> > >> > which
>> > >> > > > prevents simply adding Scala 2.12 to our supported build matrix
>> > and
>> > >> > > > continuing to build 2.10 / 2.11 against Java 1.7.
>> > >> > > >
>> > >> > > > Greg
>> > >> > > >
>> > >> > > >
>> > >> > > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger <
>> > rmetz...@apache.org>
>> > >> > > wrote:
>> > >> > > > >
>> > >> > > > > Hi all,
>> > >> > > > >
>> > >> > > > > as promised in March, I want to revive this discussion!
>> > >> > > > >
>> > >> > > > > Our users are begging for Scala 2.12 support [1], migration
>> to
>> > >> Akka
>> > >> > 2.4
>> > >> > > > would solve a bunch of shading / dependency issues (Akka 2.4
>> will
>> > >> > remove
>> > >> > > > Akka's protobuf dependency [2][3]) and generally Java 8's new
>> > >> language
>> > >> > > > features all speak for dropping Java 7.
>> > >> > > > >
>> > >> > > > > Java 8 has been released in March, 2014. Java 7 is
>> unsupported
>> > >> since
>> > >> > > > June 2016.
>> > >> > > > >
>> > >> > > > > So what's the feeling in the community regarding the step?
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > 

Re: [POLL] Who still uses Java 7 with Flink ?

[Kurt Young]

+1 for droppint Java 7, we have been using Java 8 for more than one year in
Alibaba and everything work fine.

Best,
Kurt

On Thu, Jul 13, 2017 at 2:53 AM, Bowen Li  wrote:

> +1 for dropping Java 7
>
> On Wed, Jul 12, 2017 at 9:04 AM, Gyula Fóra  wrote:
>
> > +1 for dropping 1.7 from me as well.
> >
> > Gyula
> >
> > On Wed, Jul 12, 2017, 17:53 Ted Yu  wrote:
> >
> > > +1 on dropping support for Java 1.7
> > >
> > >  Original message 
> > > From: Robert Metzger 
> > > Date: 7/12/17 8:36 AM (GMT-08:00)
> > > To: d...@flink.apache.org
> > > Cc: user 
> > > Subject: Re: [POLL] Who still uses Java 7 with Flink ?
> > >
> > > +1 to drop Java 7 support
> > >
> > > I believe that we can move to Java 8 for the argument you've stated.
> > > ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will
> > > require it as well.
> > >
> > > On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko
>  > >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I would be in favor of dropping Java 7 as we don't use it in our
> hadoop
> > >> infra (and we are a bank). Also, Spark 2.2 has been released today,
> > >> which doesn't
> > >> support Java 7  > html
> > >> >
> > >> anymore, and Flink should not lack behind :-)
> > >>
> > >> Cheers, Fokko
> > >>
> > >> 2017-07-12 15:56 GMT+02:00 Stephan Ewen :
> > >>
> > >> > Bumping this thread again.
> > >> >
> > >> > There are several strong points for dropping Java 7 support, apart
> > from
> > >> the
> > >> > fact that it is not maintained
> > >> >
> > >> >   - We could really use the Java 8 default methods feature in
> > >> interfaces to
> > >> > evolve the API without breaking backwards compatibility
> > >> >
> > >> >   - Easier build setup for Scala 2.12 (which requires Java 8), no
> need
> > >> to
> > >> > manage the tricky combinations of Java / Scala versions
> > >> >
> > >> >   - Ability to use vanilla Akka (rather than Flakka) which requires
> > >> Java 8.
> > >> > - Fewer problems for users that use Akka in the Flink
> applications
> > >> > - Flakka currently does not support Scala 2.12
> > >> > - Newer Akka versions shade protobuf, which is important
> > >> >
> > >> > I think these together make a pretty good case for bumping the
> > required
> > >> > Java version to Java 8.
> > >> >
> > >> > It would just help both Flink users (dependency management, Scala
> > >> versions)
> > >> > and developers (build simplification) a lot.
> > >> > Unless we see users stepping forward and making a case that it will
> be
> > >> > impossible for them to upgrade to Java 8, I suggest to go forward
> with
> > >> > this.
> > >> >
> > >> > Best,
> > >> > Stephan
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai 
> > wrote:
> > >> >
> > >> > > +1
> > >> > >
> > >> > > There are several high impacts security vulnerabilities in JDK 7
> and
> > >> will
> > >> > > not be addressed.
> > >> > >
> > >> > > As a result we completely moved away from JDK 7.
> > >> > >
> > >> > > +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in
> two
> > >> > steps.
> > >> > >
> > >> > >
> > >> > > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan 
> > wrote:
> > >> > >
> > >> > > > Is this not two different issues?
> > >> > > > - adding builds for Scala 2.12
> > >> > > > - upgrading to Java version 1.8
> > >> > > >
> > >> > > > It may be time to switch, but I haven’t seen anything in
> > FLINK-5005
> > >> > which
> > >> > > > prevents simply adding Scala 2.12 to our supported build matrix
> > and
> > >> > > > continuing to build 2.10 / 2.11 against Java 1.7.
> > >> > > >
> > >> > > > Greg
> > >> > > >
> > >> > > >
> > >> > > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger <
> > rmetz...@apache.org>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > Hi all,
> > >> > > > >
> > >> > > > > as promised in March, I want to revive this discussion!
> > >> > > > >
> > >> > > > > Our users are begging for Scala 2.12 support [1], migration to
> > >> Akka
> > >> > 2.4
> > >> > > > would solve a bunch of shading / dependency issues (Akka 2.4
> will
> > >> > remove
> > >> > > > Akka's protobuf dependency [2][3]) and generally Java 8's new
> > >> language
> > >> > > > features all speak for dropping Java 7.
> > >> > > > >
> > >> > > > > Java 8 has been released in March, 2014. Java 7 is unsupported
> > >> since
> > >> > > > June 2016.
> > >> > > > >
> > >> > > > > So what's the feeling in the community regarding the step?
> > >> > > > >
> > >> > > > >
> > >> > > > > [1] https://issues.apache.org/jira/browse/FLINK-5005# <
> > >> > > > https://issues.apache.org/jira/browse/FLINK-5005#>
> > >> > > > > [2] https://issues.apache.org/jira/browse/FLINK-5989 <
> > >> > > > https://issues.apache.org/jira/browse/FLINK-5989>
> > >> > > > > 

Reading static data

[Mohit Anchlia]

What is the best way to read a map of lookup data? This lookup data is like
a small short lived data that is available in transformation to do things
like filtering, additional augmentation of data etc.

Re: sanity check in production

[Gyula Fóra]

Hi!

Assuming you have some spare compute resources on your cluster (which you
should have in a production setting to be safe). I think 2) would be the
best option, ideally started from a savepoint of the production job to
verify your state logic as well.

You could also run the test job on a smaller parallelism setting, and
verify that it actually works, then maybe run some live data through it as
well before killing of the test job and updating the prod job.

Even though this might have a fairly high temporary cost I think it is
ultimately worth it to test on live data before upgrading the production
job.

Cheers,
Gyula

burgesschen  ezt írta (időpont: 2017. júl. 12.,
Sze, 20:49):

> Hello everyone,
>
> Our team ran into an issue that testing new deployment of flink job is
> difficult as explained below
>
>
> Goal:
> When we are deploying new version of a flink job in production. we want to
> be able to have the job process some test messages and verify the output to
> make sure that the job is running correctly. (sanity check)
>
> Problem:
> The tests messages interfere with the watermark of the flink job,
> potentially causing it dropping real messages.
>
> Possible solutions:
> 1. have a separate watermark for the test messages
>   (looks not supported by the current framework)
>
> 2. run a separate Flink job (same code) in production for sanity check
> before actual deployment
>   (high operational costs)
>
> 3. cancel the running production job with a save point, run a new job with
> the save point, do sanity check and mess up the watermark of the new job,
> kill the new job, do actual deployment with the same save point.
>   (high operational costs)
>
> Any idea is appreciated, thanks!
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/sanity-check-in-production-tp14229.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

FlinkKafkaConsumer subscribes to partitions in restoredState only.

[ninad]

Hello,
We're noticing that FlinkKafkaConsumer subscribes to partitions in restored
state only. Thus, partitions which aren't in restored state aren't read. We
have to restart the job, for FlinkKafkaConsumer to read from all partitions. 

Here are the details:

Environment:
Flink-1.3.0, standalone cluster as well as hadoop-cloudera cluster
flink-connector-kafka-0.9_2.11:1.3.0

-Start a job which reads from kafka topic.
-Bring down all kafka brokers
-Bring up kafka brokers

At this point, we see this in the logs:

*2017-07-12 19:53:23,661 INFO 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase  -
Consumer subtask 0 will start reading 8 partitions with offsets in restored
state: {KafkaTopicPartition{topic='topic.event.filter', partition=0}=3,
KafkaTopicPartition{topic='topic.event.filter', partition=2}=18,
KafkaTopicPartition{topic='topic.event.filter', partition=8}=1,
KafkaTopicPartition{topic='topic.event.filter', partition=9}=-1,
KafkaTopicPartition{topic='topic.event.filter', partition=3}=17,
KafkaTopicPartition{topic='topic.event.filter', partition=4}=17,
KafkaTopicPartition{topic='topic.event.filter', partition=5}=17,
KafkaTopicPartition{topic='topic.event.filter', partition=6}=1}*

Flink subscribes to only 8 partitions, because they are in recovery.
Remaining partitions aren't subscribed to.

>From the code, I don't see a place where, the partitions in non-restored
state are being subscribed to.

Relevant code:

*if (restoredState != null) {
for (KafkaTopicPartition kafkaTopicPartition : 
kafkaTopicPartitions) {
if 
(restoredState.containsKey(kafkaTopicPartition)) {

subscribedPartitionsToStartOffsets.put(kafkaTopicPartition,
restoredState.get(kafkaTopicPartition));
}
}

LOG.info("Consumer subtask {} will start reading {} 
partitions with
offsets in restored state: {}",
getRuntimeContext().getIndexOfThisSubtask(),
subscribedPartitionsToStartOffsets.size(),
subscribedPartitionsToStartOffsets);
}*

We're not setting 'setStartFromEarliest' or 'setStartFromLatest', so it's
using the default: 'setStartFromGroupOffsets'.

Are we missing any setting? Doing something wrong? Please let us know.
Thanks !






--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkKafkaConsumer-subscribes-to-partitions-in-restoredState-only-tp14233.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: Why would a kafka source checkpoint take so long?

[vinay patil]

Hi Gyula,

I have observed similar issue with FlinkConsumer09 and 010 and posted it to
the mailing list as well . This issue is not consistent, however whenever
it happens it leads to checkpoints getting failed or taking a long time to
complete.

Regards,
Vinay Patil

On Wed, Jul 12, 2017 at 7:00 PM, Gyula Fóra [via Apache Flink User Mailing
List archive.]  wrote:

> I have added logging that will help determine this as well, next time this
> happens I will post the results. (Although there doesnt seem to be high
> backpressure)
>
> Thanks for the tips,
> Gyula
>
> Stephan Ewen <[hidden email]
> > ezt írta (időpont:
> 2017. júl. 12., Sze, 15:27):
>
>> Can it be that the checkpoint thread is waiting to grab the lock, which
>> is held by the chain under backpressure?
>>
>> On Wed, Jul 12, 2017 at 12:23 PM, Gyula Fóra <[hidden email]
>> > wrote:
>>
>>> Yes thats definitely what I am about to do next but just thought maybe
>>> someone has seen this before.
>>>
>>> Will post info next time it happens. (Not guaranteed to happen soon as
>>> it didn't happen for a long time before)
>>>
>>> Gyula
>>>
>>> On Wed, Jul 12, 2017, 12:13 Stefan Richter <[hidden email]
>>> > wrote:
>>>
 Hi,

 could you introduce some logging to figure out from which method call
 the delay is introduced?

 Best,
 Stefan

 Am 12.07.2017 um 11:37 schrieb Gyula Fóra <[hidden email]
 >:

 Hi,

 We are using the latest 1.3.1

 Gyula

 Urs Schoenenberger <[hidden email]
 > ezt írta
 (időpont: 2017. júl. 12., Sze, 10:44):

> Hi Gyula,
>
> I don't know the cause unfortunately, but we observed a similiar issue
> on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1.
> Which version are you running on?
>
> Urs
>
> On 12.07.2017 09:48, Gyula Fóra wrote:
> > Hi,
> >
> > I have noticed a strange behavior in one of our jobs: every once in
> a while
> > the Kafka source checkpointing time becomes extremely large compared
> to
> > what it usually is. (To be very specific it is a kafka source
> chained with
> > a stateless map operator)
> >
> > To be more specific checkpointing the offsets usually takes around
> 10ms
> > which sounds reasonable but in some checkpoints this goes into the
> 3-5
> > minutes range practically blocking the job for that period of time.
> > Yesterday I have observed even 10 minute delays. First I thought
> that some
> > sources might trigger checkpoints later than others, but adding some
> > logging and comparing it it seems that the triggerCheckpoint was
> received
> > at the same time.
> >
> > Interestingly only one of the 3 kafka sources in the job seems to be
> > affected (last time I checked at least). We are still using the 0.8
> > consumer with commit on checkpoints. Also I dont see this happen in
> other
> > jobs.
> >
> > Any clue on what might cause this?
> >
> > Thanks :)
> > Gyula
> >
> >
> >
> > Hi,
> >
> > I have noticed a strange behavior in one of our jobs: every once in a
> > while the Kafka source checkpointing time becomes extremely large
> > compared to what it usually is. (To be very specific it is a kafka
> > source chained with a stateless map operator)
> >
> > To be more specific checkpointing the offsets usually takes around
> 10ms
> > which sounds reasonable but in some checkpoints this goes into the
> 3-5
> > minutes range practically blocking the job for that period of time.
> > Yesterday I have observed even 10 minute delays. First I thought that
> > some sources might trigger checkpoints later than others, but adding
> > some logging and comparing it it seems that the triggerCheckpoint was
> > received at the same time.
> >
> > Interestingly only one of the 3 kafka sources in the job seems to be
> > affected (last time I checked at least). We are still using the 0.8
> > consumer with commit on checkpoints. Also I dont see this happen in
> > other jobs.
> >
> > Any clue on what might cause this?
> >
> > Thanks :)
> > Gyula
>
> --
> Urs Schönenberger - [hidden email]
> 
>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>


>>
>
> --
> If you reply to this email, your message will be added to the discussion
> 

Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

[Günter Hipler]

+1 to drop Java 7 support


On 12.07.2017 16:43, Stephan Ewen wrote:

Hi users!

Flink currently maintains backwards compatibility for savepoint 
formats, which means that savepoints taken with Flink version 1.1.x 
and 1.2.x can be resumed in Flink 1.3.x


We are discussing how many versions back to support. The proposition 
is the following:


*   Suggestion: Flink 1.4.0 will be able to resume savepoints taken 
with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x 
and 1.0.x*



The reason for that is that there is a lot of code mapping between the 
completely different legacy format (1.1.x, not re-scalable) and the 
key-group-oriented format (1.2.x onwards, re-scalable). It would 
greatly help the development of state and checkpointing features to 
drop that old code.


Please let us know if you have concerns about that.

Best,
Stephan



--
Universität Basel
Universitätsbibliothek
Günter Hipler
Projekt SwissBib
Schoenbeinstrasse 18-20
4056 Basel, Schweiz
Tel.: + 41 (0)61 267 31 12 Fax: ++41 61 267 3103
E-Mail guenter.hip...@unibas.ch
URL: www.swissbib.org  / http://www.ub.unibas.ch/

Re: [POLL] Who still uses Java 7 with Flink ?

[Bowen Li]

+1 for dropping Java 7

On Wed, Jul 12, 2017 at 9:04 AM, Gyula Fóra  wrote:

> +1 for dropping 1.7 from me as well.
>
> Gyula
>
> On Wed, Jul 12, 2017, 17:53 Ted Yu  wrote:
>
> > +1 on dropping support for Java 1.7
> >
> >  Original message 
> > From: Robert Metzger 
> > Date: 7/12/17 8:36 AM (GMT-08:00)
> > To: d...@flink.apache.org
> > Cc: user 
> > Subject: Re: [POLL] Who still uses Java 7 with Flink ?
> >
> > +1 to drop Java 7 support
> >
> > I believe that we can move to Java 8 for the argument you've stated.
> > ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will
> > require it as well.
> >
> > On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko  >
> > wrote:
> >
> >> Hi,
> >>
> >> I would be in favor of dropping Java 7 as we don't use it in our hadoop
> >> infra (and we are a bank). Also, Spark 2.2 has been released today,
> >> which doesn't
> >> support Java 7  html
> >> >
> >> anymore, and Flink should not lack behind :-)
> >>
> >> Cheers, Fokko
> >>
> >> 2017-07-12 15:56 GMT+02:00 Stephan Ewen :
> >>
> >> > Bumping this thread again.
> >> >
> >> > There are several strong points for dropping Java 7 support, apart
> from
> >> the
> >> > fact that it is not maintained
> >> >
> >> >   - We could really use the Java 8 default methods feature in
> >> interfaces to
> >> > evolve the API without breaking backwards compatibility
> >> >
> >> >   - Easier build setup for Scala 2.12 (which requires Java 8), no need
> >> to
> >> > manage the tricky combinations of Java / Scala versions
> >> >
> >> >   - Ability to use vanilla Akka (rather than Flakka) which requires
> >> Java 8.
> >> > - Fewer problems for users that use Akka in the Flink applications
> >> > - Flakka currently does not support Scala 2.12
> >> > - Newer Akka versions shade protobuf, which is important
> >> >
> >> > I think these together make a pretty good case for bumping the
> required
> >> > Java version to Java 8.
> >> >
> >> > It would just help both Flink users (dependency management, Scala
> >> versions)
> >> > and developers (build simplification) a lot.
> >> > Unless we see users stepping forward and making a case that it will be
> >> > impossible for them to upgrade to Java 8, I suggest to go forward with
> >> > this.
> >> >
> >> > Best,
> >> > Stephan
> >> >
> >> >
> >> >
> >> > On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai 
> wrote:
> >> >
> >> > > +1
> >> > >
> >> > > There are several high impacts security vulnerabilities in JDK 7 and
> >> will
> >> > > not be addressed.
> >> > >
> >> > > As a result we completely moved away from JDK 7.
> >> > >
> >> > > +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in two
> >> > steps.
> >> > >
> >> > >
> >> > > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan 
> wrote:
> >> > >
> >> > > > Is this not two different issues?
> >> > > > - adding builds for Scala 2.12
> >> > > > - upgrading to Java version 1.8
> >> > > >
> >> > > > It may be time to switch, but I haven’t seen anything in
> FLINK-5005
> >> > which
> >> > > > prevents simply adding Scala 2.12 to our supported build matrix
> and
> >> > > > continuing to build 2.10 / 2.11 against Java 1.7.
> >> > > >
> >> > > > Greg
> >> > > >
> >> > > >
> >> > > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger <
> rmetz...@apache.org>
> >> > > wrote:
> >> > > > >
> >> > > > > Hi all,
> >> > > > >
> >> > > > > as promised in March, I want to revive this discussion!
> >> > > > >
> >> > > > > Our users are begging for Scala 2.12 support [1], migration to
> >> Akka
> >> > 2.4
> >> > > > would solve a bunch of shading / dependency issues (Akka 2.4 will
> >> > remove
> >> > > > Akka's protobuf dependency [2][3]) and generally Java 8's new
> >> language
> >> > > > features all speak for dropping Java 7.
> >> > > > >
> >> > > > > Java 8 has been released in March, 2014. Java 7 is unsupported
> >> since
> >> > > > June 2016.
> >> > > > >
> >> > > > > So what's the feeling in the community regarding the step?
> >> > > > >
> >> > > > >
> >> > > > > [1] https://issues.apache.org/jira/browse/FLINK-5005# <
> >> > > > https://issues.apache.org/jira/browse/FLINK-5005#>
> >> > > > > [2] https://issues.apache.org/jira/browse/FLINK-5989 <
> >> > > > https://issues.apache.org/jira/browse/FLINK-5989>
> >> > > > > [3]
> >> > > > https://issues.apache.org/jira/browse/FLINK-3211?
> >> > > focusedCommentId=15274018=com.atlassian.jira.
> >> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> >> > > > <
> >> > > > https://issues.apache.org/jira/browse/FLINK-3211?
> >> > > focusedCommentId=15274018=com.atlassian.jira.
> >> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Mar 23, 2017 at 2:42 PM, Theodore Vasiloudis 

sanity check in production

[burgesschen]

Hello everyone, 

Our team ran into an issue that testing new deployment of flink job is
difficult as explained below 


Goal: 
When we are deploying new version of a flink job in production. we want to
be able to have the job process some test messages and verify the output to
make sure that the job is running correctly. (sanity check) 

Problem: 
The tests messages interfere with the watermark of the flink job,
potentially causing it dropping real messages. 

Possible solutions: 
1. have a separate watermark for the test messages 
  (looks not supported by the current framework) 

2. run a separate Flink job (same code) in production for sanity check
before actual deployment 
  (high operational costs) 

3. cancel the running production job with a save point, run a new job with
the save point, do sanity check and mess up the watermark of the new job,
kill the new job, do actual deployment with the same save point. 
  (high operational costs) 

Any idea is appreciated, thanks! 



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/sanity-check-in-production-tp14229.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: System properties when submitting flink job to YARN Session

[Jins George]

Hi Aljoscha,

I am still using Beam on Flink. I have one yarn session running multiple 
streaming jobs. The application jar contains some environment specific 
run time properties( like ip addresses, rest api end points etc). This 
adds overhead in my usecase as we have to deploy this in multiple 
environments.  I was trying to decouple these properties files from the 
uber jar and provide as as either a classpath resource or pass the path 
of the file as a system property to the jvm.


So far I noticed following options to achieve this.

 * put all properties in a file and use /--classpath/ file:// option  in /flink run /command . This needs the url to be
   accessible from all nodes, something like NFS
 * use -D in yarn-session to pass each properties. This will need to
   restart the yarn session if a new property gets added.

An ideal solution for me would to provide a local classpath to flink run 
command and that gets propagated to other workers automatically :)


Thanks,
Jins
On 07/12/2017 02:25 AM, Aljoscha Krettek wrote:

Hi,

Yes, setting the property using -D when creating the session should work to 
make it available on all workers. I think after that it cannot be changed since 
they JVMs are already running.

If I may ask, what’s your use case for this? Are you still using Beam on Flink 
or are you using vanilla Flink with this?

Best,
Aljoscha


On 11. Jul 2017, at 07:24, Jins George  wrote:

Thanks Nico. I am able to pass arguments to the  main program, that works, but 
not exactly that I was looking for.

I guess to have all worker jvms the same  system property,  I have to set it at 
yarn-session creation time using -D ( haven't tried it yet)

Thanks,
Jins George

On 07/10/2017 06:56 AM, Nico Kruber wrote:

Hi Jins,
I'm not sure whether you can define a system property, but you can include it
in the program arguments of "flink run [OPTIONS]  "

You may also be able to define system properties but these are probably only
valid in your main() function executed within the flink run script, not any
operators run on other JVM nodes. Have you tried that?


Nico

On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote:

Hello,

I want to set the path of a properties file as System property in my
application(something like -Dkey=value).
Is there a way to set it while submitting a flink job to running YARN
Session? I am using //bin/flink run/ to submit the job to a already
running YARN session.

Thanks,
Jins George

Flink Mesos Outstanding Offers - trouble launching task managers

[Prashant Nayak]

Hi

We’re running Flink 1.3.1 on Mesos.

>From time-to-time, the Flink app master seems to have trouble with Mesos
offers… At such time, it obviously ends up not launching the requested task
managers (mesos.initial-tasks) and we’ve noticed situations where it
launches zero tasks.  During such
times we see a long list of “Outstanding Offers” in the Mesos UI.  At the
same time, the app master logs have the following


2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
 org.apache.flink.mesos.scheduler.LaunchCoordinator  - Processing 12
task(s) against 0 new offer(s) plus outstanding offers.
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
 org.apache.flink.mesos.scheduler.LaunchCoordinator  - Resources
considered: (note: expired offers not deducted from below)
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
 org.apache.flink.mesos.scheduler.LaunchCoordinator  -   10.80.xx.6 has 0.0
MB, 0.0 cpus
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
 org.apache.flink.mesos.scheduler.LaunchCoordinator  -   10.80.xx.233 has
0.0 MB, 0.0 cpus
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
 org.apache.flink.mesos.scheduler.LaunchCoordinator  - Waiting for more
offers; 12 task(s) are not yet launched.

The two Mesos agents above (10.80.xx.6, 10.80.xx.233) are listed as having
offers outstanding to the Flink framework in the Mesos UI

Appreciate any input on how to go about resolving such an issue.

Thanks
Prashant

Re: Getting Errors when using keyby()

[Dawid Wysakowicz]

Hi Sridhar,

Your class is missing default constructor(without arguments) thus it is not a 
valid POJO in Flink.

You can check the requirements for POJO in link here: 
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/api_concepts.html#pojos


> On 12 Jul 2017, at 19:54, Sridhar Chellappa  wrote:
> 
> I have a DataStream on which I am applying a CEP pattern and grouping the 
> results using keyby(). The DataStream Object is a pojo :
> 
> public class DataStreamObject {
> private String field1;
> private String field2;
> 
> public DataStreamObject(String field1, String field2) {
> this.field1 = field1;
> this.field2 = field2;
> }
> 
> public void setField1(String field1) {
> this.field1 = field1;
> }
> 
> public String getField1() {
> return field1;
> }
> 
> 
> public void setField2(String field2) {
> this.field2 = field2;
> }
> 
> public String getField2() {
> return field2;
> }
> 
> @Override
> public boolean equals(Object o) {
> if (this == o) return true;
> if (!(o instanceof DataStreamObject)) return false;
> 
> DataStreamObject that = (DataStreamObject) o;
> 
> if (!getField1().equals(that.getField1())) return false;
> return getField2().equals(that.getField2());
> }
> 
> @Override
> public int hashCode() {
> int result = getField1().hashCode();
> result = 31 * result + getField2().hashCode();
> return result;
> }
> 
> @Override
> public String toString() {
> return "DriverSameAsCustomer{" +
> "field1='" + field1 + '\'' +
> ", field2='" + field2 + '\'' +
> '}';
> }
> }
> 
> When I submit my flinkjob, I get the following error :
> 
> 
> This type (GenericType) cannot be used as key.
>   
> org.apache.flink.api.common.operators.Keys$ExpressionKeys.(Keys.java:330)
>   
> org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:294)
>   com.foo.Main.main(Main.java:66)
> 
> 
> As I understand, I do not need to implement Key interface if the class is a 
> POJO (which it is).
> 
> Please help me understand where I am going wrong an suggest a fix.
> 
> 



signature.asc
Description: Message signed with OpenPGP

Re: Fink: KafkaProducer Data Loss

[ninad]

Hey guys, any update on this? If needed I can attach our code.



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Getting Errors when using keyby()

[Sridhar Chellappa]

I have a DataStream on which I am applying a CEP pattern and grouping the
results using keyby(). The DataStream Object is a pojo :

public class DataStreamObject {
private String field1;
private String field2;

public DataStreamObject(String field1, String field2) {
this.field1 = field1;
this.field2 = field2;
}

public void setField1(String field1) {
this.field1 = field1;
}

public String getField1() {
return field1;
}


public void setField2(String field2) {
this.field2 = field2;
}

public String getField2() {
return field2;
}

@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof DataStreamObject)) return false;

DataStreamObject that = (DataStreamObject) o;

if (!getField1().equals(that.getField1())) return false;
return getField2().equals(that.getField2());
}

@Override
public int hashCode() {
int result = getField1().hashCode();
result = 31 * result + getField2().hashCode();
return result;
}

@Override
public String toString() {
return "DriverSameAsCustomer{" +
"field1='" + field1 + '\'' +
", field2='" + field2 + '\'' +
'}';
}
}

When I submit my flinkjob, I get the following error :


This type (GenericType) cannot be used as key.

org.apache.flink.api.common.operators.Keys$ExpressionKeys.(Keys.java:330)

org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:294)
com.foo.Main.main(Main.java:66)


As I understand, I do not need to implement Key interface if the class
is a POJO (which it is).

Please help me understand where I am going wrong an suggest a fix.

Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

[Kanstantsin Kamkou]

Hi users!

I can't upgrade from 1.2.x to 1.3.x without code adaptations.
Upgrading from 1.(0|1).x to 1.2.x produces configuration mess.
Maybe you can discuss changing the release plan, speed it up a little
bit and use the major.minor.patch versions as advantage to organize
the release process more accurate. As the result, backwards
compatibility might be guarantied among a single major number only. It
also sounds like a new version will be 2.0.0. This way flink might get
more trust as I will see the complexity of changing the version
number.

> MINOR version when you add functionality in a backwards-compatible manner
That is exactly my expectation.

Best.


On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen  wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats,
> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
> resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is the
> following:
>
>Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x
>
>
> The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> help the development of state and checkpointing features to drop that old
> code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>

Re: [POLL] Who still uses Java 7 with Flink ?

[Gyula Fóra]

+1 for dropping 1.7 from me as well.

Gyula

On Wed, Jul 12, 2017, 17:53 Ted Yu  wrote:

> +1 on dropping support for Java 1.7
>
>  Original message 
> From: Robert Metzger 
> Date: 7/12/17 8:36 AM (GMT-08:00)
> To: d...@flink.apache.org
> Cc: user 
> Subject: Re: [POLL] Who still uses Java 7 with Flink ?
>
> +1 to drop Java 7 support
>
> I believe that we can move to Java 8 for the argument you've stated.
> ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will
> require it as well.
>
> On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko 
> wrote:
>
>> Hi,
>>
>> I would be in favor of dropping Java 7 as we don't use it in our hadoop
>> infra (and we are a bank). Also, Spark 2.2 has been released today,
>> which doesn't
>> support Java 7 > >
>> anymore, and Flink should not lack behind :-)
>>
>> Cheers, Fokko
>>
>> 2017-07-12 15:56 GMT+02:00 Stephan Ewen :
>>
>> > Bumping this thread again.
>> >
>> > There are several strong points for dropping Java 7 support, apart from
>> the
>> > fact that it is not maintained
>> >
>> >   - We could really use the Java 8 default methods feature in
>> interfaces to
>> > evolve the API without breaking backwards compatibility
>> >
>> >   - Easier build setup for Scala 2.12 (which requires Java 8), no need
>> to
>> > manage the tricky combinations of Java / Scala versions
>> >
>> >   - Ability to use vanilla Akka (rather than Flakka) which requires
>> Java 8.
>> > - Fewer problems for users that use Akka in the Flink applications
>> > - Flakka currently does not support Scala 2.12
>> > - Newer Akka versions shade protobuf, which is important
>> >
>> > I think these together make a pretty good case for bumping the required
>> > Java version to Java 8.
>> >
>> > It would just help both Flink users (dependency management, Scala
>> versions)
>> > and developers (build simplification) a lot.
>> > Unless we see users stepping forward and making a case that it will be
>> > impossible for them to upgrade to Java 8, I suggest to go forward with
>> > this.
>> >
>> > Best,
>> > Stephan
>> >
>> >
>> >
>> > On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai  wrote:
>> >
>> > > +1
>> > >
>> > > There are several high impacts security vulnerabilities in JDK 7 and
>> will
>> > > not be addressed.
>> > >
>> > > As a result we completely moved away from JDK 7.
>> > >
>> > > +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in two
>> > steps.
>> > >
>> > >
>> > > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan  wrote:
>> > >
>> > > > Is this not two different issues?
>> > > > - adding builds for Scala 2.12
>> > > > - upgrading to Java version 1.8
>> > > >
>> > > > It may be time to switch, but I haven’t seen anything in FLINK-5005
>> > which
>> > > > prevents simply adding Scala 2.12 to our supported build matrix and
>> > > > continuing to build 2.10 / 2.11 against Java 1.7.
>> > > >
>> > > > Greg
>> > > >
>> > > >
>> > > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger 
>> > > wrote:
>> > > > >
>> > > > > Hi all,
>> > > > >
>> > > > > as promised in March, I want to revive this discussion!
>> > > > >
>> > > > > Our users are begging for Scala 2.12 support [1], migration to
>> Akka
>> > 2.4
>> > > > would solve a bunch of shading / dependency issues (Akka 2.4 will
>> > remove
>> > > > Akka's protobuf dependency [2][3]) and generally Java 8's new
>> language
>> > > > features all speak for dropping Java 7.
>> > > > >
>> > > > > Java 8 has been released in March, 2014. Java 7 is unsupported
>> since
>> > > > June 2016.
>> > > > >
>> > > > > So what's the feeling in the community regarding the step?
>> > > > >
>> > > > >
>> > > > > [1] https://issues.apache.org/jira/browse/FLINK-5005# <
>> > > > https://issues.apache.org/jira/browse/FLINK-5005#>
>> > > > > [2] https://issues.apache.org/jira/browse/FLINK-5989 <
>> > > > https://issues.apache.org/jira/browse/FLINK-5989>
>> > > > > [3]
>> > > > https://issues.apache.org/jira/browse/FLINK-3211?
>> > > focusedCommentId=15274018=com.atlassian.jira.
>> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
>> > > > <
>> > > > https://issues.apache.org/jira/browse/FLINK-3211?
>> > > focusedCommentId=15274018=com.atlassian.jira.
>> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Mar 23, 2017 at 2:42 PM, Theodore Vasiloudis <
>> > > > theodoros.vasilou...@gmail.com > theodoros.vasilou...@gmail.com
>> > >>
>> > > > wrote:
>> > > > > Hello all,
>> > > > >
>> > > > > I'm sure you've considered this already, but what this data does
>> not
>> > > > include is all the potential future users,
>> > > > > i.e. slower moving organizations (banks etc.) which could be on
>> Java
>> > 7
>> > > > still.
>> > > > >
>> > > > > 

Re: [POLL] Who still uses Java 7 with Flink ?

[Ted Yu]

+1 on dropping support for Java 1.7
 Original message From: Robert Metzger  
Date: 7/12/17  8:36 AM  (GMT-08:00) To: d...@flink.apache.org Cc: user 
 Subject: Re: [POLL] Who still uses Java 7 with Flink ? 
+1 to drop Java 7 support
I believe that we can move to Java 8 for the argument you've 
stated.ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will 
require it as well.
On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko  wrote:
Hi,



I would be in favor of dropping Java 7 as we don't use it in our hadoop

infra (and we are a bank). Also, Spark 2.2 has been released today,

which doesn't

support Java 7 

anymore, and Flink should not lack behind :-)



Cheers, Fokko



2017-07-12 15:56 GMT+02:00 Stephan Ewen :



> Bumping this thread again.

>

> There are several strong points for dropping Java 7 support, apart from the

> fact that it is not maintained

>

>   - We could really use the Java 8 default methods feature in interfaces to

> evolve the API without breaking backwards compatibility

>

>   - Easier build setup for Scala 2.12 (which requires Java 8), no need to

> manage the tricky combinations of Java / Scala versions

>

>   - Ability to use vanilla Akka (rather than Flakka) which requires Java 8.

>     - Fewer problems for users that use Akka in the Flink applications

>     - Flakka currently does not support Scala 2.12

>     - Newer Akka versions shade protobuf, which is important

>

> I think these together make a pretty good case for bumping the required

> Java version to Java 8.

>

> It would just help both Flink users (dependency management, Scala versions)

> and developers (build simplification) a lot.

> Unless we see users stepping forward and making a case that it will be

> impossible for them to upgrade to Java 8, I suggest to go forward with

> this.

>

> Best,

> Stephan

>

>

>

> On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai  wrote:

>

> > +1

> >

> > There are several high impacts security vulnerabilities in JDK 7 and will

> > not be addressed.

> >

> > As a result we completely moved away from JDK 7.

> >

> > +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in two

> steps.

> >

> >

> > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan  wrote:

> >

> > > Is this not two different issues?

> > > - adding builds for Scala 2.12

> > > - upgrading to Java version 1.8

> > >

> > > It may be time to switch, but I haven’t seen anything in FLINK-5005

> which

> > > prevents simply adding Scala 2.12 to our supported build matrix and

> > > continuing to build 2.10 / 2.11 against Java 1.7.

> > >

> > > Greg

> > >

> > >

> > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger 

> > wrote:

> > > >

> > > > Hi all,

> > > >

> > > > as promised in March, I want to revive this discussion!

> > > >

> > > > Our users are begging for Scala 2.12 support [1], migration to Akka

> 2.4

> > > would solve a bunch of shading / dependency issues (Akka 2.4 will

> remove

> > > Akka's protobuf dependency [2][3]) and generally Java 8's new language

> > > features all speak for dropping Java 7.

> > > >

> > > > Java 8 has been released in March, 2014. Java 7 is unsupported since

> > > June 2016.

> > > >

> > > > So what's the feeling in the community regarding the step?

> > > >

> > > >

> > > > [1] https://issues.apache.org/jira/browse/FLINK-5005# <

> > > https://issues.apache.org/jira/browse/FLINK-5005#>

> > > > [2] https://issues.apache.org/jira/browse/FLINK-5989 <

> > > https://issues.apache.org/jira/browse/FLINK-5989>

> > > > [3]

> > > https://issues.apache.org/jira/browse/FLINK-3211?

> > focusedCommentId=15274018=com.atlassian.jira.

> > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018

> > > <

> > > https://issues.apache.org/jira/browse/FLINK-3211?

> > focusedCommentId=15274018=com.atlassian.jira.

> > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018

> > > >

> > > >

> > > >

> > > > On Thu, Mar 23, 2017 at 2:42 PM, Theodore Vasiloudis <

> > > theodoros.vasilou...@gmail.com  >>

> > > wrote:

> > > > Hello all,

> > > >

> > > > I'm sure you've considered this already, but what this data does not

> > > include is all the potential future users,

> > > > i.e. slower moving organizations (banks etc.) which could be on Java

> 7

> > > still.

> > > >

> > > > Whether those are relevant is up for debate.

> > > >

> > > > Cheers,

> > > > Theo

> > > >

> > > > On Thu, Mar 23, 2017 at 12:14 PM, Robert Metzger <

> rmetz...@apache.org

> > > > wrote:

> > > > Yeah, you are right :)

> > > > I'll put something in my calendar for end of May.

> > > >

> > > > On Thu, Mar 23, 2017 at 12:12 PM, Greg Hogan  > > 

Re: [POLL] Who still uses Java 7 with Flink ?

[Stefan Richter]

+1 for dropping Java 7.

> Am 12.07.2017 um 17:36 schrieb Robert Metzger :
> 
> +1 to drop Java 7 support
> 
> I believe that we can move to Java 8 for the argument you've stated.
> ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will require 
> it as well.
> 
> On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko  > wrote:
> Hi,
> 
> I would be in favor of dropping Java 7 as we don't use it in our hadoop
> infra (and we are a bank). Also, Spark 2.2 has been released today,
> which doesn't
> support Java 7  >
> anymore, and Flink should not lack behind :-)
> 
> Cheers, Fokko
> 
> 2017-07-12 15:56 GMT+02:00 Stephan Ewen  >:
> 
> > Bumping this thread again.
> >
> > There are several strong points for dropping Java 7 support, apart from the
> > fact that it is not maintained
> >
> >   - We could really use the Java 8 default methods feature in interfaces to
> > evolve the API without breaking backwards compatibility
> >
> >   - Easier build setup for Scala 2.12 (which requires Java 8), no need to
> > manage the tricky combinations of Java / Scala versions
> >
> >   - Ability to use vanilla Akka (rather than Flakka) which requires Java 8.
> > - Fewer problems for users that use Akka in the Flink applications
> > - Flakka currently does not support Scala 2.12
> > - Newer Akka versions shade protobuf, which is important
> >
> > I think these together make a pretty good case for bumping the required
> > Java version to Java 8.
> >
> > It would just help both Flink users (dependency management, Scala versions)
> > and developers (build simplification) a lot.
> > Unless we see users stepping forward and making a case that it will be
> > impossible for them to upgrade to Java 8, I suggest to go forward with
> > this.
> >
> > Best,
> > Stephan
> >
> >
> >
> > On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai  > > wrote:
> >
> > > +1
> > >
> > > There are several high impacts security vulnerabilities in JDK 7 and will
> > > not be addressed.
> > >
> > > As a result we completely moved away from JDK 7.
> > >
> > > +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in two
> > steps.
> > >
> > >
> > > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan  > > > wrote:
> > >
> > > > Is this not two different issues?
> > > > - adding builds for Scala 2.12
> > > > - upgrading to Java version 1.8
> > > >
> > > > It may be time to switch, but I haven’t seen anything in FLINK-5005
> > which
> > > > prevents simply adding Scala 2.12 to our supported build matrix and
> > > > continuing to build 2.10 / 2.11 against Java 1.7.
> > > >
> > > > Greg
> > > >
> > > >
> > > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger  > > > > >
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > as promised in March, I want to revive this discussion!
> > > > >
> > > > > Our users are begging for Scala 2.12 support [1], migration to Akka
> > 2.4
> > > > would solve a bunch of shading / dependency issues (Akka 2.4 will
> > remove
> > > > Akka's protobuf dependency [2][3]) and generally Java 8's new language
> > > > features all speak for dropping Java 7.
> > > > >
> > > > > Java 8 has been released in March, 2014. Java 7 is unsupported since
> > > > June 2016.
> > > > >
> > > > > So what's the feeling in the community regarding the step?
> > > > >
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-5005# 
> > > > >  <
> > > > https://issues.apache.org/jira/browse/FLINK-5005# 
> > > > >
> > > > > [2] https://issues.apache.org/jira/browse/FLINK-5989 
> > > > >  <
> > > > https://issues.apache.org/jira/browse/FLINK-5989 
> > > > >
> > > > > [3]
> > > > https://issues.apache.org/jira/browse/FLINK-3211 
> > > > ?
> > > focusedCommentId=15274018=com.atlassian.jira.
> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> > > > <
> > > > https://issues.apache.org/jira/browse/FLINK-3211 
> > > > ?
> > > focusedCommentId=15274018=com.atlassian.jira.
> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Mar 23, 2017 at 2:42 PM, Theodore Vasiloudis <
> > > > theodoros.vasilou...@gmail.com  
> > > >  > > > 
> > >>
> > > > wrote:
> > > > > Hello all,

Re: [POLL] Who still uses Java 7 with Flink ?

[Robert Metzger]

+1 to drop Java 7 support

I believe that we can move to Java 8 for the argument you've stated.
ElasticSearch 5, Spark 2.2  require Java 8 already, Hadoop 3.0.0 will
require it as well.

On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko 
wrote:

> Hi,
>
> I would be in favor of dropping Java 7 as we don't use it in our hadoop
> infra (and we are a bank). Also, Spark 2.2 has been released today,
> which doesn't
> support Java 7 
> anymore, and Flink should not lack behind :-)
>
> Cheers, Fokko
>
> 2017-07-12 15:56 GMT+02:00 Stephan Ewen :
>
> > Bumping this thread again.
> >
> > There are several strong points for dropping Java 7 support, apart from
> the
> > fact that it is not maintained
> >
> >   - We could really use the Java 8 default methods feature in interfaces
> to
> > evolve the API without breaking backwards compatibility
> >
> >   - Easier build setup for Scala 2.12 (which requires Java 8), no need to
> > manage the tricky combinations of Java / Scala versions
> >
> >   - Ability to use vanilla Akka (rather than Flakka) which requires Java
> 8.
> > - Fewer problems for users that use Akka in the Flink applications
> > - Flakka currently does not support Scala 2.12
> > - Newer Akka versions shade protobuf, which is important
> >
> > I think these together make a pretty good case for bumping the required
> > Java version to Java 8.
> >
> > It would just help both Flink users (dependency management, Scala
> versions)
> > and developers (build simplification) a lot.
> > Unless we see users stepping forward and making a case that it will be
> > impossible for them to upgrade to Java 8, I suggest to go forward with
> > this.
> >
> > Best,
> > Stephan
> >
> >
> >
> > On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai  wrote:
> >
> > > +1
> > >
> > > There are several high impacts security vulnerabilities in JDK 7 and
> will
> > > not be addressed.
> > >
> > > As a result we completely moved away from JDK 7.
> > >
> > > +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in two
> > steps.
> > >
> > >
> > > On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan  wrote:
> > >
> > > > Is this not two different issues?
> > > > - adding builds for Scala 2.12
> > > > - upgrading to Java version 1.8
> > > >
> > > > It may be time to switch, but I haven’t seen anything in FLINK-5005
> > which
> > > > prevents simply adding Scala 2.12 to our supported build matrix and
> > > > continuing to build 2.10 / 2.11 against Java 1.7.
> > > >
> > > > Greg
> > > >
> > > >
> > > > > On Jun 8, 2017, at 11:39 AM, Robert Metzger 
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > as promised in March, I want to revive this discussion!
> > > > >
> > > > > Our users are begging for Scala 2.12 support [1], migration to Akka
> > 2.4
> > > > would solve a bunch of shading / dependency issues (Akka 2.4 will
> > remove
> > > > Akka's protobuf dependency [2][3]) and generally Java 8's new
> language
> > > > features all speak for dropping Java 7.
> > > > >
> > > > > Java 8 has been released in March, 2014. Java 7 is unsupported
> since
> > > > June 2016.
> > > > >
> > > > > So what's the feeling in the community regarding the step?
> > > > >
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-5005# <
> > > > https://issues.apache.org/jira/browse/FLINK-5005#>
> > > > > [2] https://issues.apache.org/jira/browse/FLINK-5989 <
> > > > https://issues.apache.org/jira/browse/FLINK-5989>
> > > > > [3]
> > > > https://issues.apache.org/jira/browse/FLINK-3211?
> > > focusedCommentId=15274018=com.atlassian.jira.
> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> > > > <
> > > > https://issues.apache.org/jira/browse/FLINK-3211?
> > > focusedCommentId=15274018=com.atlassian.jira.
> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Mar 23, 2017 at 2:42 PM, Theodore Vasiloudis <
> > > > theodoros.vasilou...@gmail.com  gmail.com
> > >>
> > > > wrote:
> > > > > Hello all,
> > > > >
> > > > > I'm sure you've considered this already, but what this data does
> not
> > > > include is all the potential future users,
> > > > > i.e. slower moving organizations (banks etc.) which could be on
> Java
> > 7
> > > > still.
> > > > >
> > > > > Whether those are relevant is up for debate.
> > > > >
> > > > > Cheers,
> > > > > Theo
> > > > >
> > > > > On Thu, Mar 23, 2017 at 12:14 PM, Robert Metzger <
> > rmetz...@apache.org
> > > > > wrote:
> > > > > Yeah, you are right :)
> > > > > I'll put something in my calendar for end of May.
> > > > >
> > > > > On Thu, Mar 23, 2017 at 12:12 PM, Greg Hogan  > > > > wrote:
> > > > > Robert,
> > > > >
> > > > > Thanks for the report. Shouldn’t we be 

[POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

[Stephan Ewen]

Hi users!

Flink currently maintains backwards compatibility for savepoint formats,
which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
resumed in Flink 1.3.x

We are discussing how many versions back to support. The proposition is the
following:

*   Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*


The reason for that is that there is a lot of code mapping between the
completely different legacy format (1.1.x, not re-scalable) and the
key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
help the development of state and checkpointing features to drop that old
code.

Please let us know if you have concerns about that.

Best,
Stephan

delta iteration

[Alieh]

Hello all,

I need iteration number in delta iteration (or any kind of counter). Is 
there anyway to implement or extract it?


Cheers,

Alieh

Re: delta iteration

[Greg Hogan]

Hi Alieh,

From a rich function call getIterationRuntimeContext().getSuperstepNumber()

Greg


> On Jul 12, 2017, at 9:56 AM, Alieh  wrote:
> 
> Hello all,
> 
> I need iteration number in delta iteration (or any kind of counter). Is there 
> anyway to implement or extract it?
> 
> Cheers,
> 
> Alieh

Use processing time and time window flink acts like batch mode.

[yunfan123]

I using processing time and the data source comes from kafka.
My code is like follows:
streams.keyBy(XXX)
   .timeWindow(Time.seconds(30))
   .apply(myClassObject)

Log in myClassObject is like:
2017-07-12 20:00:00,
2017-07-12 20:00:00,

2017-07-12 20:00:30,
2017-07-12 20:00:30,
...
2017-07-12 20:01:00,
2017-07-12 20:01:00,

It means every 30 seconds, all of the window be triggered once.
And I'm sure the amounts of my key is very large.
So in the program based my time window length is 30, my expects is  :
key1: processing time 1
key2: processing time 1
key3: processing time 2
key4: processing time 3
The key1 and key2 should be processed in time 31.
The key3 should be processed in time 32.
The key4 should be processed in time 33.







--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Use-processing-time-and-time-window-flink-acts-like-batch-mode-tp14211.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

delta iteration

[Alieh]

Hello all,

I need iteration number in delta iteration (or any kind of counter). Is 
there anyway to implement or extract it?


Cheers,

Alieh

delta iteration

[Alieh]

Hello all,

I need iteration number in delta iteration (or any kind of counter). Is 
there anyway to implement or extract it?


Cheers,

Alieh

Re: [POLL] Who still uses Java 7 with Flink ?

[Stephan Ewen]

Bumping this thread again.

There are several strong points for dropping Java 7 support, apart from the
fact that it is not maintained

  - We could really use the Java 8 default methods feature in interfaces to
evolve the API without breaking backwards compatibility

  - Easier build setup for Scala 2.12 (which requires Java 8), no need to
manage the tricky combinations of Java / Scala versions

  - Ability to use vanilla Akka (rather than Flakka) which requires Java 8.
- Fewer problems for users that use Akka in the Flink applications
- Flakka currently does not support Scala 2.12
- Newer Akka versions shade protobuf, which is important

I think these together make a pretty good case for bumping the required
Java version to Java 8.

It would just help both Flink users (dependency management, Scala versions)
and developers (build simplification) a lot.
Unless we see users stepping forward and making a case that it will be
impossible for them to upgrade to Java 8, I suggest to go forward with this.

Best,
Stephan



On Thu, Jun 8, 2017 at 9:36 PM, Haohui Mai  wrote:

> +1
>
> There are several high impacts security vulnerabilities in JDK 7 and will
> not be addressed.
>
> As a result we completely moved away from JDK 7.
>
> +1 on separating the tasks of supporting Scala 2.12 and JDK 8 in two steps.
>
>
> On Thu, Jun 8, 2017 at 9:53 AM Greg Hogan  wrote:
>
> > Is this not two different issues?
> > - adding builds for Scala 2.12
> > - upgrading to Java version 1.8
> >
> > It may be time to switch, but I haven’t seen anything in FLINK-5005 which
> > prevents simply adding Scala 2.12 to our supported build matrix and
> > continuing to build 2.10 / 2.11 against Java 1.7.
> >
> > Greg
> >
> >
> > > On Jun 8, 2017, at 11:39 AM, Robert Metzger 
> wrote:
> > >
> > > Hi all,
> > >
> > > as promised in March, I want to revive this discussion!
> > >
> > > Our users are begging for Scala 2.12 support [1], migration to Akka 2.4
> > would solve a bunch of shading / dependency issues (Akka 2.4 will remove
> > Akka's protobuf dependency [2][3]) and generally Java 8's new language
> > features all speak for dropping Java 7.
> > >
> > > Java 8 has been released in March, 2014. Java 7 is unsupported since
> > June 2016.
> > >
> > > So what's the feeling in the community regarding the step?
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-5005# <
> > https://issues.apache.org/jira/browse/FLINK-5005#>
> > > [2] https://issues.apache.org/jira/browse/FLINK-5989 <
> > https://issues.apache.org/jira/browse/FLINK-5989>
> > > [3]
> > https://issues.apache.org/jira/browse/FLINK-3211?
> focusedCommentId=15274018=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> > <
> > https://issues.apache.org/jira/browse/FLINK-3211?
> focusedCommentId=15274018=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15274018
> > >
> > >
> > >
> > > On Thu, Mar 23, 2017 at 2:42 PM, Theodore Vasiloudis <
> > theodoros.vasilou...@gmail.com >
> > wrote:
> > > Hello all,
> > >
> > > I'm sure you've considered this already, but what this data does not
> > include is all the potential future users,
> > > i.e. slower moving organizations (banks etc.) which could be on Java 7
> > still.
> > >
> > > Whether those are relevant is up for debate.
> > >
> > > Cheers,
> > > Theo
> > >
> > > On Thu, Mar 23, 2017 at 12:14 PM, Robert Metzger  > > wrote:
> > > Yeah, you are right :)
> > > I'll put something in my calendar for end of May.
> > >
> > > On Thu, Mar 23, 2017 at 12:12 PM, Greg Hogan  > > wrote:
> > > Robert,
> > >
> > > Thanks for the report. Shouldn’t we be revisiting this decision at the
> > beginning of the new release cycle rather than near the end? There is
> > currently little cost to staying with Java 7 since no Flink code or pull
> > requests have been written for Java 8.
> > >
> > > Greg
> > >
> > >
> > >
> > >> On Mar 23, 2017, at 6:37 AM, Robert Metzger  > > wrote:
> > >>
> > >> Looks like 9% on twitter and 24% on the mailing list are still using
> > Java 7.
> > >>
> > >> I would vote to keep supporting Java 7 for Flink 1.3 and then revisit
> > once we are approaching 1.4 in September.
> > >>
> > >> On Thu, Mar 16, 2017 at 8:00 AM, Bowen Li  > > wrote:
> > >> There's always a tradeoff we need to make. I'm in favor of upgrading
> to
> > Java 8 to bring in all new Java features.
> > >>
> > >> The common way I've seen (and I agree) other software upgrading major
> > things like this is 1) upgrade for next big release without backward
> > compatibility and notify everyone 2) maintain and patch current, old-tech
> > compatible version at a reasonably limited scope. 

Re: Why would a kafka source checkpoint take so long?

[Gyula Fóra]

I have added logging that will help determine this as well, next time this
happens I will post the results. (Although there doesnt seem to be high
backpressure)

Thanks for the tips,
Gyula

Stephan Ewen  ezt írta (időpont: 2017. júl. 12., Sze,
15:27):

> Can it be that the checkpoint thread is waiting to grab the lock, which is
> held by the chain under backpressure?
>
> On Wed, Jul 12, 2017 at 12:23 PM, Gyula Fóra  wrote:
>
>> Yes thats definitely what I am about to do next but just thought maybe
>> someone has seen this before.
>>
>> Will post info next time it happens. (Not guaranteed to happen soon as it
>> didn't happen for a long time before)
>>
>> Gyula
>>
>> On Wed, Jul 12, 2017, 12:13 Stefan Richter 
>> wrote:
>>
>>> Hi,
>>>
>>> could you introduce some logging to figure out from which method call
>>> the delay is introduced?
>>>
>>> Best,
>>> Stefan
>>>
>>> Am 12.07.2017 um 11:37 schrieb Gyula Fóra :
>>>
>>> Hi,
>>>
>>> We are using the latest 1.3.1
>>>
>>> Gyula
>>>
>>> Urs Schoenenberger  ezt írta (időpont:
>>> 2017. júl. 12., Sze, 10:44):
>>>
 Hi Gyula,

 I don't know the cause unfortunately, but we observed a similiar issue
 on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1.
 Which version are you running on?

 Urs

 On 12.07.2017 09:48, Gyula Fóra wrote:
 > Hi,
 >
 > I have noticed a strange behavior in one of our jobs: every once in a
 while
 > the Kafka source checkpointing time becomes extremely large compared
 to
 > what it usually is. (To be very specific it is a kafka source chained
 with
 > a stateless map operator)
 >
 > To be more specific checkpointing the offsets usually takes around
 10ms
 > which sounds reasonable but in some checkpoints this goes into the 3-5
 > minutes range practically blocking the job for that period of time.
 > Yesterday I have observed even 10 minute delays. First I thought that
 some
 > sources might trigger checkpoints later than others, but adding some
 > logging and comparing it it seems that the triggerCheckpoint was
 received
 > at the same time.
 >
 > Interestingly only one of the 3 kafka sources in the job seems to be
 > affected (last time I checked at least). We are still using the 0.8
 > consumer with commit on checkpoints. Also I dont see this happen in
 other
 > jobs.
 >
 > Any clue on what might cause this?
 >
 > Thanks :)
 > Gyula
 >
 >
 >
 > Hi,
 >
 > I have noticed a strange behavior in one of our jobs: every once in a
 > while the Kafka source checkpointing time becomes extremely large
 > compared to what it usually is. (To be very specific it is a kafka
 > source chained with a stateless map operator)
 >
 > To be more specific checkpointing the offsets usually takes around
 10ms
 > which sounds reasonable but in some checkpoints this goes into the 3-5
 > minutes range practically blocking the job for that period of time.
 > Yesterday I have observed even 10 minute delays. First I thought that
 > some sources might trigger checkpoints later than others, but adding
 > some logging and comparing it it seems that the triggerCheckpoint was
 > received at the same time.
 >
 > Interestingly only one of the 3 kafka sources in the job seems to be
 > affected (last time I checked at least). We are still using the 0.8
 > consumer with commit on checkpoints. Also I dont see this happen in
 > other jobs.
 >
 > Any clue on what might cause this?
 >
 > Thanks :)
 > Gyula

 --
 Urs Schönenberger - urs.schoenenber...@tngtech.com

 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
 Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
 Sitz: Unterföhring * Amtsgericht München * HRB 135082

>>>
>>>
>

StreamTableSource

[nragon]

Hi,

I have two streams coming from kafka which I want to map into table
environment. Because they are not pojo or tuple I will have to map them
using, for instance, Types.ROW_NAMED. Can i use StreamTableSource and call
registerTableSource or should I use the same code inside getDataStream but
calling registerDataStream instead?

Thanks



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamTableSource-tp14209.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: Why would a kafka source checkpoint take so long?

[Stephan Ewen]

Can it be that the checkpoint thread is waiting to grab the lock, which is
held by the chain under backpressure?

On Wed, Jul 12, 2017 at 12:23 PM, Gyula Fóra  wrote:

> Yes thats definitely what I am about to do next but just thought maybe
> someone has seen this before.
>
> Will post info next time it happens. (Not guaranteed to happen soon as it
> didn't happen for a long time before)
>
> Gyula
>
> On Wed, Jul 12, 2017, 12:13 Stefan Richter 
> wrote:
>
>> Hi,
>>
>> could you introduce some logging to figure out from which method call the
>> delay is introduced?
>>
>> Best,
>> Stefan
>>
>> Am 12.07.2017 um 11:37 schrieb Gyula Fóra :
>>
>> Hi,
>>
>> We are using the latest 1.3.1
>>
>> Gyula
>>
>> Urs Schoenenberger  ezt írta (időpont:
>> 2017. júl. 12., Sze, 10:44):
>>
>>> Hi Gyula,
>>>
>>> I don't know the cause unfortunately, but we observed a similiar issue
>>> on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1.
>>> Which version are you running on?
>>>
>>> Urs
>>>
>>> On 12.07.2017 09:48, Gyula Fóra wrote:
>>> > Hi,
>>> >
>>> > I have noticed a strange behavior in one of our jobs: every once in a
>>> while
>>> > the Kafka source checkpointing time becomes extremely large compared to
>>> > what it usually is. (To be very specific it is a kafka source chained
>>> with
>>> > a stateless map operator)
>>> >
>>> > To be more specific checkpointing the offsets usually takes around 10ms
>>> > which sounds reasonable but in some checkpoints this goes into the 3-5
>>> > minutes range practically blocking the job for that period of time.
>>> > Yesterday I have observed even 10 minute delays. First I thought that
>>> some
>>> > sources might trigger checkpoints later than others, but adding some
>>> > logging and comparing it it seems that the triggerCheckpoint was
>>> received
>>> > at the same time.
>>> >
>>> > Interestingly only one of the 3 kafka sources in the job seems to be
>>> > affected (last time I checked at least). We are still using the 0.8
>>> > consumer with commit on checkpoints. Also I dont see this happen in
>>> other
>>> > jobs.
>>> >
>>> > Any clue on what might cause this?
>>> >
>>> > Thanks :)
>>> > Gyula
>>> >
>>> >
>>> >
>>> > Hi,
>>> >
>>> > I have noticed a strange behavior in one of our jobs: every once in a
>>> > while the Kafka source checkpointing time becomes extremely large
>>> > compared to what it usually is. (To be very specific it is a kafka
>>> > source chained with a stateless map operator)
>>> >
>>> > To be more specific checkpointing the offsets usually takes around 10ms
>>> > which sounds reasonable but in some checkpoints this goes into the 3-5
>>> > minutes range practically blocking the job for that period of time.
>>> > Yesterday I have observed even 10 minute delays. First I thought that
>>> > some sources might trigger checkpoints later than others, but adding
>>> > some logging and comparing it it seems that the triggerCheckpoint was
>>> > received at the same time.
>>> >
>>> > Interestingly only one of the 3 kafka sources in the job seems to be
>>> > affected (last time I checked at least). We are still using the 0.8
>>> > consumer with commit on checkpoints. Also I dont see this happen in
>>> > other jobs.
>>> >
>>> > Any clue on what might cause this?
>>> >
>>> > Thanks :)
>>> > Gyula
>>>
>>> --
>>> Urs Schönenberger - urs.schoenenber...@tngtech.com
>>>
>>> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>>> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>>> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>>>
>>
>>

Re: System properties when submitting flink job to YARN Session

[yunfan123]

Can I The specific the jars that I depend on when I submit my project?



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/System-properties-when-submitting-flink-job-to-YARN-Session-tp14158p14207.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

global window trigger

[jad mad]

for a global window with
a custom event time trigger that fires every 1 minute
and then apply a custom window function to it,

the trigger firing seems working but the element collection
i get inside of my custom WindowFunction is always
the whole inputs from start to end rather than
inputs subset from start to the every 1min window end(maxTimestamp).

is this because GlobalWindows is a processing time operator that
does not work with event time?

thanks a lot,

trigger testing

[jad mad]

I'm testing with ContinuousEventTimeTrigger with a TumblingWindow.

let's say in time frame A, B, C there are 1, 2 and 3 inputs
the count result I'd expected was something like

ContinuousEventTimeTrigger->  A:1,  B:3, C:6

but from the result I get, it seems the inputs haven't been accumulated.
and behaving just like an EventTimeTrigger.

Why is this? am I wrong somewhere?

Re: Associative operation + windowAll - possible parallelism

[Debski]

Thanks for the suggestions, I will take a look at transform function. 



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Associative-operation-windowAll-possible-parallelism-tp14187p14204.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: Why would a kafka source checkpoint take so long?

[Gyula Fóra]

Yes thats definitely what I am about to do next but just thought maybe
someone has seen this before.

Will post info next time it happens. (Not guaranteed to happen soon as it
didn't happen for a long time before)

Gyula

On Wed, Jul 12, 2017, 12:13 Stefan Richter 
wrote:

> Hi,
>
> could you introduce some logging to figure out from which method call the
> delay is introduced?
>
> Best,
> Stefan
>
> Am 12.07.2017 um 11:37 schrieb Gyula Fóra :
>
> Hi,
>
> We are using the latest 1.3.1
>
> Gyula
>
> Urs Schoenenberger  ezt írta (időpont:
> 2017. júl. 12., Sze, 10:44):
>
>> Hi Gyula,
>>
>> I don't know the cause unfortunately, but we observed a similiar issue
>> on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1.
>> Which version are you running on?
>>
>> Urs
>>
>> On 12.07.2017 09:48, Gyula Fóra wrote:
>> > Hi,
>> >
>> > I have noticed a strange behavior in one of our jobs: every once in a
>> while
>> > the Kafka source checkpointing time becomes extremely large compared to
>> > what it usually is. (To be very specific it is a kafka source chained
>> with
>> > a stateless map operator)
>> >
>> > To be more specific checkpointing the offsets usually takes around 10ms
>> > which sounds reasonable but in some checkpoints this goes into the 3-5
>> > minutes range practically blocking the job for that period of time.
>> > Yesterday I have observed even 10 minute delays. First I thought that
>> some
>> > sources might trigger checkpoints later than others, but adding some
>> > logging and comparing it it seems that the triggerCheckpoint was
>> received
>> > at the same time.
>> >
>> > Interestingly only one of the 3 kafka sources in the job seems to be
>> > affected (last time I checked at least). We are still using the 0.8
>> > consumer with commit on checkpoints. Also I dont see this happen in
>> other
>> > jobs.
>> >
>> > Any clue on what might cause this?
>> >
>> > Thanks :)
>> > Gyula
>> >
>> >
>> >
>> > Hi,
>> >
>> > I have noticed a strange behavior in one of our jobs: every once in a
>> > while the Kafka source checkpointing time becomes extremely large
>> > compared to what it usually is. (To be very specific it is a kafka
>> > source chained with a stateless map operator)
>> >
>> > To be more specific checkpointing the offsets usually takes around 10ms
>> > which sounds reasonable but in some checkpoints this goes into the 3-5
>> > minutes range practically blocking the job for that period of time.
>> > Yesterday I have observed even 10 minute delays. First I thought that
>> > some sources might trigger checkpoints later than others, but adding
>> > some logging and comparing it it seems that the triggerCheckpoint was
>> > received at the same time.
>> >
>> > Interestingly only one of the 3 kafka sources in the job seems to be
>> > affected (last time I checked at least). We are still using the 0.8
>> > consumer with commit on checkpoints. Also I dont see this happen in
>> > other jobs.
>> >
>> > Any clue on what might cause this?
>> >
>> > Thanks :)
>> > Gyula
>>
>> --
>> Urs Schönenberger - urs.schoenenber...@tngtech.com
>>
>> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>>
>
>

Re: Why would a kafka source checkpoint take so long?

[Stefan Richter]

Hi,

could you introduce some logging to figure out from which method call the delay 
is introduced?

Best,
Stefan

> Am 12.07.2017 um 11:37 schrieb Gyula Fóra :
> 
> Hi,
> 
> We are using the latest 1.3.1
> 
> Gyula
> 
> Urs Schoenenberger  > ezt írta (időpont: 2017. júl. 12., 
> Sze, 10:44):
> Hi Gyula,
> 
> I don't know the cause unfortunately, but we observed a similiar issue
> on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1.
> Which version are you running on?
> 
> Urs
> 
> On 12.07.2017 09:48, Gyula Fóra wrote:
> > Hi,
> >
> > I have noticed a strange behavior in one of our jobs: every once in a while
> > the Kafka source checkpointing time becomes extremely large compared to
> > what it usually is. (To be very specific it is a kafka source chained with
> > a stateless map operator)
> >
> > To be more specific checkpointing the offsets usually takes around 10ms
> > which sounds reasonable but in some checkpoints this goes into the 3-5
> > minutes range practically blocking the job for that period of time.
> > Yesterday I have observed even 10 minute delays. First I thought that some
> > sources might trigger checkpoints later than others, but adding some
> > logging and comparing it it seems that the triggerCheckpoint was received
> > at the same time.
> >
> > Interestingly only one of the 3 kafka sources in the job seems to be
> > affected (last time I checked at least). We are still using the 0.8
> > consumer with commit on checkpoints. Also I dont see this happen in other
> > jobs.
> >
> > Any clue on what might cause this?
> >
> > Thanks :)
> > Gyula
> >
> >
> >
> > Hi,
> >
> > I have noticed a strange behavior in one of our jobs: every once in a
> > while the Kafka source checkpointing time becomes extremely large
> > compared to what it usually is. (To be very specific it is a kafka
> > source chained with a stateless map operator)
> >
> > To be more specific checkpointing the offsets usually takes around 10ms
> > which sounds reasonable but in some checkpoints this goes into the 3-5
> > minutes range practically blocking the job for that period of time.
> > Yesterday I have observed even 10 minute delays. First I thought that
> > some sources might trigger checkpoints later than others, but adding
> > some logging and comparing it it seems that the triggerCheckpoint was
> > received at the same time.
> >
> > Interestingly only one of the 3 kafka sources in the job seems to be
> > affected (last time I checked at least). We are still using the 0.8
> > consumer with commit on checkpoints. Also I dont see this happen in
> > other jobs.
> >
> > Any clue on what might cause this?
> >
> > Thanks :)
> > Gyula
> 
> --
> Urs Schönenberger - urs.schoenenber...@tngtech.com 
> 
> 
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082

Re: Why would a kafka source checkpoint take so long?

[Gyula Fóra]

Hi,

We are using the latest 1.3.1

Gyula

Urs Schoenenberger  ezt írta (időpont:
2017. júl. 12., Sze, 10:44):

> Hi Gyula,
>
> I don't know the cause unfortunately, but we observed a similiar issue
> on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1.
> Which version are you running on?
>
> Urs
>
> On 12.07.2017 09:48, Gyula Fóra wrote:
> > Hi,
> >
> > I have noticed a strange behavior in one of our jobs: every once in a
> while
> > the Kafka source checkpointing time becomes extremely large compared to
> > what it usually is. (To be very specific it is a kafka source chained
> with
> > a stateless map operator)
> >
> > To be more specific checkpointing the offsets usually takes around 10ms
> > which sounds reasonable but in some checkpoints this goes into the 3-5
> > minutes range practically blocking the job for that period of time.
> > Yesterday I have observed even 10 minute delays. First I thought that
> some
> > sources might trigger checkpoints later than others, but adding some
> > logging and comparing it it seems that the triggerCheckpoint was received
> > at the same time.
> >
> > Interestingly only one of the 3 kafka sources in the job seems to be
> > affected (last time I checked at least). We are still using the 0.8
> > consumer with commit on checkpoints. Also I dont see this happen in other
> > jobs.
> >
> > Any clue on what might cause this?
> >
> > Thanks :)
> > Gyula
> >
> >
> >
> > Hi,
> >
> > I have noticed a strange behavior in one of our jobs: every once in a
> > while the Kafka source checkpointing time becomes extremely large
> > compared to what it usually is. (To be very specific it is a kafka
> > source chained with a stateless map operator)
> >
> > To be more specific checkpointing the offsets usually takes around 10ms
> > which sounds reasonable but in some checkpoints this goes into the 3-5
> > minutes range practically blocking the job for that period of time.
> > Yesterday I have observed even 10 minute delays. First I thought that
> > some sources might trigger checkpoints later than others, but adding
> > some logging and comparing it it seems that the triggerCheckpoint was
> > received at the same time.
> >
> > Interestingly only one of the 3 kafka sources in the job seems to be
> > affected (last time I checked at least). We are still using the 0.8
> > consumer with commit on checkpoints. Also I dont see this happen in
> > other jobs.
> >
> > Any clue on what might cause this?
> >
> > Thanks :)
> > Gyula
>
> --
> Urs Schönenberger - urs.schoenenber...@tngtech.com
>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>

Re: Can AsyncFunction be applied to connected streams

[Aljoscha Krettek]

I think this would not necessarily be a problem. If the async operation is 
directly after the enrichment operation the enriched operations will be 
directly forwarded to the async operation. (With a copy step, that can be 
disabled by enabling object reuse at the StreamExecutionEnvironment)

Best,
Aljoscha
> On 7. Jul 2017, at 15:59, PedroMrChaves  wrote:
> 
> Hello,
> 
> I wanted to keep the data locally, if I associate the fetched metadata with
> eachevent (in an enrichment phase) it would considerably increase their size
> since the metadata that I need to process the event in the async I/O is to
> large.
> 
> Regards,
> Pedro. 
> 
> 
> 
> -
> Best Regards,
> Pedro Chaves
> --
> View this message in context: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Can-AsyncFunction-be-applied-to-connected-streams-tp14137p14148.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at 
> Nabble.com.

Re: Associative operation + windowAll - possible parallelism

[Aljoscha Krettek]

Yes, your observations are correct! 

Currently, I see two possible solutions that you could implement as a user:

1. Use .window() with a dummy key followed by a .windowAll():

DataStream input = …;
input
  .map( (in) -> new Tuple2(, in)) 
  .keyBy(0)
  .window(…)
  .aggregate(...)
  .windowAll(…)
  .aggregate(…)

The problem here is that you incur a shuffle, which may or may not improve 
performance, depending on the aggregation operation.

To get around that shuffle you would have to use option 2.

2. Use a custom StreamOperator that does the pre-aggregation and emits results 
on event-time, followed by .windowAll();

DataStream input = …;
input
  .transform(, , new PreAggregationOperator()) 
  .windowAll(…)
  .aggregate(…)

Where PreAggregationOperator would pre-aggregate and checkpoint the 
pre-aggregated values and emit the pre-aggregate when the watermark for the end 
of a window arrives. The reason for why you have to use a custom operator is 
that a user function cannot “listen” on the watermark and therefore would not 
be able to emit the aggregate at the right time.

I hope this helps.

Best,
Aljoscha

> On 11. Jul 2017, at 22:05, Debski  wrote:
> 
> Let us assume that I want to perform some kind of aggregation in specified
> time windows (e.g. tumbling window of 1 minute) and my aggregation operation
> is associative. Wouldn't it be possible to represent windowAll in runtime as
> /parallelism + 1/ operator instances where /parallelism/ number of operators
> compute partial aggregates and then partial results are merged into one in
> the last instance of the operator by using merge function that is present in
> AggregateFunction function.
> 
> Basically I would like to compute single aggregated value for all events in
> given time window and aggregation operation itself can be parallelized. 
> 
> For example i could have mapped stream with .map operation that has
> parallelism 4, then each map operator instance would pass 1/4 of events to
> adjacent instance of windowAll operator that would compute desired aggregate
> over subset of events. When the window is closed all partial states would be
> transferred to single windowAll merging operator.
> 
> Are there any plans to support such situations/is it possible to somehow
> implement such operator in current version of Flink. 
> 
> Also there is a note in windowAll java-doc about possible parallelism but I
> don't know how relevant it is to my case:
> 
> Note: This operation can be inherently non-parallel since all elements have
> to pass through the same operator instance. (Only for special cases, such as
> aligned time windows is it possible to perform this operation in parallel).
> 
> 
> 
> --
> View this message in context: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Associative-operation-windowAll-possible-parallelism-tp14187.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at 
> Nabble.com.

Re: System properties when submitting flink job to YARN Session

[Aljoscha Krettek]

Hi,

Yes, setting the property using -D when creating the session should work to 
make it available on all workers. I think after that it cannot be changed since 
they JVMs are already running.

If I may ask, what’s your use case for this? Are you still using Beam on Flink 
or are you using vanilla Flink with this?

Best,
Aljoscha

> On 11. Jul 2017, at 07:24, Jins George  wrote:
> 
> Thanks Nico. I am able to pass arguments to the  main program, that works, 
> but not exactly that I was looking for.
> 
> I guess to have all worker jvms the same  system property,  I have to set it 
> at yarn-session creation time using -D ( haven't tried it yet)
> 
> Thanks,
> Jins George
> 
> On 07/10/2017 06:56 AM, Nico Kruber wrote:
>> Hi Jins,
>> I'm not sure whether you can define a system property, but you can include 
>> it 
>> in the program arguments of "flink run [OPTIONS]  "
>> 
>> You may also be able to define system properties but these are probably only 
>> valid in your main() function executed within the flink run script, not any 
>> operators run on other JVM nodes. Have you tried that?
>> 
>> 
>> Nico
>> 
>> On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote:
>>> Hello,
>>> 
>>> I want to set the path of a properties file as System property in my
>>> application(something like -Dkey=value).
>>> Is there a way to set it while submitting a flink job to running YARN
>>> Session? I am using //bin/flink run/ to submit the job to a already
>>> running YARN session.
>>> 
>>> Thanks,
>>> Jins George
> 

Re: Should customized Complex Events be Serializable?

[Dawid Wysakowicz]

What do you mean by ComplexEvents? Do you mean that the output of CEP library 
is DataStream? If so, then yes, they should be either 
Serializable or you should provide custom TypeSerializer.
> On 12 Jul 2017, at 06:58, Sridhar Chellappa  wrote:
> 
> Folks,
> 
> I am using the CEP library to create ComplexEvents. My question is, should 
> the ComplexEvents be serializable?



signature.asc
Description: Message signed with OpenPGP

Re: Why would a kafka source checkpoint take so long?

[Urs Schoenenberger]

Hi Gyula,

I don't know the cause unfortunately, but we observed a similiar issue
on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1.
Which version are you running on?

Urs

On 12.07.2017 09:48, Gyula Fóra wrote:
> Hi,
> 
> I have noticed a strange behavior in one of our jobs: every once in a while
> the Kafka source checkpointing time becomes extremely large compared to
> what it usually is. (To be very specific it is a kafka source chained with
> a stateless map operator)
> 
> To be more specific checkpointing the offsets usually takes around 10ms
> which sounds reasonable but in some checkpoints this goes into the 3-5
> minutes range practically blocking the job for that period of time.
> Yesterday I have observed even 10 minute delays. First I thought that some
> sources might trigger checkpoints later than others, but adding some
> logging and comparing it it seems that the triggerCheckpoint was received
> at the same time.
> 
> Interestingly only one of the 3 kafka sources in the job seems to be
> affected (last time I checked at least). We are still using the 0.8
> consumer with commit on checkpoints. Also I dont see this happen in other
> jobs.
> 
> Any clue on what might cause this?
> 
> Thanks :)
> Gyula
> 
> 
> 
> Hi,
> 
> I have noticed a strange behavior in one of our jobs: every once in a
> while the Kafka source checkpointing time becomes extremely large
> compared to what it usually is. (To be very specific it is a kafka
> source chained with a stateless map operator)
> 
> To be more specific checkpointing the offsets usually takes around 10ms
> which sounds reasonable but in some checkpoints this goes into the 3-5
> minutes range practically blocking the job for that period of time.
> Yesterday I have observed even 10 minute delays. First I thought that
> some sources might trigger checkpoints later than others, but adding
> some logging and comparing it it seems that the triggerCheckpoint was
> received at the same time.
> 
> Interestingly only one of the 3 kafka sources in the job seems to be
> affected (last time I checked at least). We are still using the 0.8
> consumer with commit on checkpoints. Also I dont see this happen in
> other jobs.
> 
> Any clue on what might cause this?
> 
> Thanks :)
> Gyula

-- 
Urs Schönenberger - urs.schoenenber...@tngtech.com

TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

Re: FlinkML ALS is taking too long to run

[Sebastian Schelter]

I don't think you need to employ a distributed system for working with this
dataset. An SGD implementation on a single machine should easily handle the
job.

Best,
Sebastian

2017-07-12 9:26 GMT+02:00 Andrea Spina :

> Dear Ziyad,
>
> Yep, I had encountered same very long runtimes with ALS as well at the time
> and I recorded improvements by increasing the number of blocks / decreasing
> #TSs/TM like you've stated out.
>
> Cheers,
>
> Andrea
>
>
>
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/FlinkML-ALS-is-
> taking-too-long-to-run-tp14154p14192.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Why would a kafka source checkpoint take so long?

[Gyula Fóra]

Hi,

I have noticed a strange behavior in one of our jobs: every once in a while
the Kafka source checkpointing time becomes extremely large compared to
what it usually is. (To be very specific it is a kafka source chained with
a stateless map operator)

To be more specific checkpointing the offsets usually takes around 10ms
which sounds reasonable but in some checkpoints this goes into the 3-5
minutes range practically blocking the job for that period of time.
Yesterday I have observed even 10 minute delays. First I thought that some
sources might trigger checkpoints later than others, but adding some
logging and comparing it it seems that the triggerCheckpoint was received
at the same time.

Interestingly only one of the 3 kafka sources in the job seems to be
affected (last time I checked at least). We are still using the 0.8
consumer with commit on checkpoints. Also I dont see this happen in other
jobs.

Any clue on what might cause this?

Thanks :)
Gyula

Re: FlinkML ALS is taking too long to run

[Andrea Spina]

Dear Ziyad, 

Yep, I had encountered same very long runtimes with ALS as well at the time
and I recorded improvements by increasing the number of blocks / decreasing
#TSs/TM like you've stated out.

Cheers,

Andrea






--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkML-ALS-is-taking-too-long-to-run-tp14154p14192.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.