Re: Backpressure rationale

2017-01-12 Thread Dean Wampler 
You can find longer explanations for backpressure on teh Interwebs, but
briefly,

1. Buffers are never infinite. Even if you're willing to write and maintain
a nontrivial scheme for flushing to disk, you might still have a major
failure scenario where the consumer is gone forever. Depending on volume
and disk size, you could run out of space quickly. Then what should you do?
Does this low-level producer know what makes sense from a strategic
perspective? The answers lead you to...
2. Backpressure composes. If you have a graph of backpressure-enabled
streams, when one slows down, the backpressure can propagate up stream.
Somewhere at the beginning of the graph, someone has to decide what to do
about a major slow down, like in the previous scenario, but now it's a
strategic concern that you can solve at the architecture level, rather than
forcing some low-level stream component make an arbitrary decision about
what to do.

dean

On Thu, Jan 12, 2017 at 9:36 AM, Gábor Gévay  wrote:

> Hello,
>
> I would like to ask about the rationale behind the backpressure
> mechanism in Flink.
>
> As I understand it, backpressure is for handling the problem of one
> operator (or source) producing records faster then the next operator
> can consume them. However, an alternative solution would be to have a
> potentially "infinite" buffer for the incoming records of an operator,
> by spilling the buffer to disk when it is getting too large. Could you
> tell me why is backpressure considered a better option?
>
> Also, I would be interested in whether going with backpressure was a
> streaming-specific decision, or do you think that having backpressure
> is also better in batch jobs?
>
> Thanks,
> Gábor
>



-- 
*Dean Wampler, Ph.D.*
Fast Data Product Architect, Office of the CTO

dean.wamp...@lightbend.com
Lightbend 
@deanwampler 
https://www.linkedin.com/in/deanwampler

Re: 答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

2017-01-12 Thread Till Rohrmann 
I'm wondering whether we should not depend the webserver encryption on the
global encryption activation and activating it instead per default.

On Thu, Jan 12, 2017 at 4:54 PM, Chesnay Schepler 
wrote:

> FLINK-5470 is a duplicate of FLINK-5298 for which there is also an open PR.
>
> FLINK-5472 is imo invalid since the webserver does support https, you just
> have to enable it as per the security documentation.
>
>
> On 12.01.2017 16:20, Till Rohrmann wrote:
>
> I also found an issue:
>
> https://issues.apache.org/jira/browse/FLINK-5470
>
> I also noticed that Flink's webserver does not support https requests. It
> might be worthwhile to add it, though.
>
> https://issues.apache.org/jira/browse/FLINK-5472
>
> On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger 
> wrote:
>
>> I also found a bunch of issues
>>
>> https://issues.apache.org/jira/browse/FLINK-5465
>> https://issues.apache.org/jira/browse/FLINK-5462
>> https://issues.apache.org/jira/browse/FLINK-5464
>> https://issues.apache.org/jira/browse/FLINK-5463
>>
>>
>> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske < 
>> fhue...@gmail.com> wrote:
>>
>> > I have another bugfix for 1.2.:
>> >
>> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR)
>> >
>> > 2017-01-10 15:16 GMT+01:00 Robert Metzger < 
>> rmetz...@apache.org>:
>> >
>> > > Hi,
>> > >
>> > > this depends a lot on the number of issues we find during the testing.
>> > >
>> > >
>> > > These are the issues I found so far:
>> > >
>> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved)
>> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved)
>> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved)
>> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved)
>> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR)
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui 
>> > wrote:
>> > >
>> > > > Do we have a probable time of 1.2 release? This month or Next month?
>> > > >
>> > > > -邮件原件-
>> > > > 发件人: Robert Metzger [mailto: 
>> rmetz...@apache.org]
>> > > > 发送时间: 2017年1月3日 20:44
>> > > > 收件人: dev@flink.apache.org
>> > > > 抄送: u...@flink.apache.org
>> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release
>> > > candidate)
>> > > >
>> > > > Hi,
>> > > >
>> > > > First of all, I wish everybody a happy new year 2017.
>> > > >
>> > > > I've set user@flink in CC so that users who are interested in
>> helping
>> > > > with the testing get notified. Please respond only to the dev@
>> list to
>> > > > keep the discussion there!
>> > > >
>> > > > According to the 1.2 release discussion thread, I've created a first
>> > > > release candidate for Flink 1.2.
>> > > > The release candidate will not be the final release, because I'm
>> > certain
>> > > > that we'll find at least one blocking issue in the candidate :)
>> > > >
>> > > > Therefore, the RC is meant as a testing only release candidate.
>> > > > Please report every issue we need to fix before the next RC in this
>> > > thread
>> > > > so that we have a good overview.
>> > > >
>> > > > The release artifacts are located here:
>> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/
>> > > >
>> > > > The maven staging repository is located here:
>> > > > https://repository.apache.org/content/repositories/orgapache
>> flink-
>> > > >
>> > > > The release commit (in branch "release-1.2.0-rc0"):
>> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced
>> > > >
>> > > >
>> > > > Happy testing!
>> > > >
>> > >
>> >
>>
>
>
>

Backpressure rationale

2017-01-12 Thread Gábor Gévay 
Hello,

I would like to ask about the rationale behind the backpressure
mechanism in Flink.

As I understand it, backpressure is for handling the problem of one
operator (or source) producing records faster then the next operator
can consume them. However, an alternative solution would be to have a
potentially "infinite" buffer for the incoming records of an operator,
by spilling the buffer to disk when it is getting too large. Could you
tell me why is backpressure considered a better option?

Also, I would be interested in whether going with backpressure was a
streaming-specific decision, or do you think that having backpressure
is also better in batch jobs?

Thanks,
Gábor

[jira] [Created] (FLINK-5476) Fail fast if trying to submit a job to a non-existing Flink cluster

2017-01-12 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-5476:


 Summary: Fail fast if trying to submit a job to a non-existing 
Flink cluster
 Key: FLINK-5476
 URL: https://issues.apache.org/jira/browse/FLINK-5476
 Project: Flink
  Issue Type: Improvement
  Components: Client
Affects Versions: 1.2.0, 1.3.0
Reporter: Till Rohrmann
Priority: Minor


In case of entering the wrong job manager address when submitting a job via 
{{flink run}}, the {{JobClientActor}} waits per default {{60 s}} until a 
{{JobClientActorConnectionException}}, indicating that the {{JobManager}} is no 
longer reachable, is thrown. In order to fail fast in case of wrong connection 
information, we could change it such that it uses initially a much lower 
timeout and only increases the timeout if it had at least once successfully 
connected to a {{JobManager}} before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5475) Extend DC/OS documentation

2017-01-12 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-5475:


 Summary: Extend DC/OS documentation
 Key: FLINK-5475
 URL: https://issues.apache.org/jira/browse/FLINK-5475
 Project: Flink
  Issue Type: Sub-task
  Components: Mesos
Reporter: Till Rohrmann
Priority: Minor


We could extend the DC/OS documentation a little bit to include information 
about how to submit a job (where to find the connection information) and that 
one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

2017-01-12 Thread Till Rohrmann 
I also found an issue:

https://issues.apache.org/jira/browse/FLINK-5470

I also noticed that Flink's webserver does not support https requests. It
might be worthwhile to add it, though.

https://issues.apache.org/jira/browse/FLINK-5472

On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger 
wrote:

> I also found a bunch of issues
>
> https://issues.apache.org/jira/browse/FLINK-5465
> https://issues.apache.org/jira/browse/FLINK-5462
> https://issues.apache.org/jira/browse/FLINK-5464
> https://issues.apache.org/jira/browse/FLINK-5463
>
>
> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske  wrote:
>
> > I have another bugfix for 1.2.:
> >
> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR)
> >
> > 2017-01-10 15:16 GMT+01:00 Robert Metzger :
> >
> > > Hi,
> > >
> > > this depends a lot on the number of issues we find during the testing.
> > >
> > >
> > > These are the issues I found so far:
> > >
> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved)
> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved)
> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved)
> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved)
> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR)
> > >
> > >
> > >
> > >
> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui 
> > wrote:
> > >
> > > > Do we have a probable time of 1.2 release? This month or Next month?
> > > >
> > > > -邮件原件-
> > > > 发件人: Robert Metzger [mailto:rmetz...@apache.org]
> > > > 发送时间: 2017年1月3日 20:44
> > > > 收件人: dev@flink.apache.org
> > > > 抄送: u...@flink.apache.org
> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release
> > > candidate)
> > > >
> > > > Hi,
> > > >
> > > > First of all, I wish everybody a happy new year 2017.
> > > >
> > > > I've set user@flink in CC so that users who are interested in
> helping
> > > > with the testing get notified. Please respond only to the dev@ list
> to
> > > > keep the discussion there!
> > > >
> > > > According to the 1.2 release discussion thread, I've created a first
> > > > release candidate for Flink 1.2.
> > > > The release candidate will not be the final release, because I'm
> > certain
> > > > that we'll find at least one blocking issue in the candidate :)
> > > >
> > > > Therefore, the RC is meant as a testing only release candidate.
> > > > Please report every issue we need to fix before the next RC in this
> > > thread
> > > > so that we have a good overview.
> > > >
> > > > The release artifacts are located here:
> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/
> > > >
> > > > The maven staging repository is located here:
> > > > https://repository.apache.org/content/repositories/
> orgapacheflink-
> > > >
> > > > The release commit (in branch "release-1.2.0-rc0"):
> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced
> > > >
> > > >
> > > > Happy testing!
> > > >
> > >
> >
>

[jira] [Created] (FLINK-5474) Extend DC/OS documentation

2017-01-12 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-5474:


 Summary: Extend DC/OS documentation
 Key: FLINK-5474
 URL: https://issues.apache.org/jira/browse/FLINK-5474
 Project: Flink
  Issue Type: Sub-task
  Components: Mesos
Reporter: Till Rohrmann
Priority: Minor


We could extend the DC/OS documentation a little bit to include information 
about how to submit a job (where to find the connection information) and that 
one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5473) setMaxParallelism() higher than 1 is possible on non-parallel operators

2017-01-12 Thread Robert Metzger (JIRA) 
Robert Metzger created FLINK-5473:
-

 Summary: setMaxParallelism() higher than 1 is possible on 
non-parallel operators
 Key: FLINK-5473
 URL: https://issues.apache.org/jira/browse/FLINK-5473
 Project: Flink
  Issue Type: Bug
  Components: DataStream API
Affects Versions: 1.2.0
Reporter: Robert Metzger


While trying out Flink 1.2, I found out that you can set a maxParallelism 
higher than 1 on a non-parallel operator.
I think we should have the same semantics as the setParallelism() method.

Also, when setting a global maxParallelism in the execution environment, it 
will be set as a default value for the non-parallel operator.
When restoring a savepoint from 1.1, you have to set the maxParallelism to the 
parallelism of the 1.1 job. Non-parallel operators will then also get the 
maxPar set to this value, leading to an error on restore.

So currently, users restoring from 1.1 to 1.2 have to manually set the 
maxParallelism to 1 for all non-parallel operators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5472) Flink's web server does not support https requests

2017-01-12 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-5472:


 Summary: Flink's web server does not support https requests
 Key: FLINK-5472
 URL: https://issues.apache.org/jira/browse/FLINK-5472
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Affects Versions: 1.2.0, 1.3.0
Reporter: Till Rohrmann
 Fix For: 1.3.0


Flink's webserver does not support HTTPS requests. When trying to access 
{{https://jobmanager:port}}, chrome says that the webserver answered with an 
invalid response {{ERR_SSL_PROTOCOL_ERROR}}.

This happens, for example, when one tries to access Flink's web UI from the 
DC/OS dashboard via the endpoint links.

I think we should add a ssl handler to Flink's web server pipeline (even though 
the certificates might not be trusted).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5471) Properly inform JobClientActor about terminated Mesos framework

2017-01-12 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-5471:


 Summary: Properly inform JobClientActor about terminated Mesos 
framework
 Key: FLINK-5471
 URL: https://issues.apache.org/jira/browse/FLINK-5471
 Project: Flink
  Issue Type: Improvement
  Components: Mesos
Affects Versions: 1.2.0, 1.3.0
Reporter: Till Rohrmann
Priority: Minor
 Fix For: 1.3.0


In case that the Mesos framework running Flink terminates (e.g. exceeded number 
of container restarts) the {{JobClientActor}} is not properly informed. As a 
consequence, the client only terminates after the {{JobClientActor}} detects 
that it lost the connection to the JobManager 
({{JobClientActorConnectionTimeoutException}}). The current default value for 
the timeout is 60s which is quite long to detect the connection loss in case of 
a termination.

I think it would be better to notify the {{JobClientActor}} which allows it to 
print a better message for the user and also allows it to react quicker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5470) Requesting non-existing log/stdout file from TM crashes the it

2017-01-12 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-5470:


 Summary: Requesting non-existing log/stdout file from TM crashes 
the it
 Key: FLINK-5470
 URL: https://issues.apache.org/jira/browse/FLINK-5470
 Project: Flink
  Issue Type: Bug
  Components: TaskManager
Affects Versions: 1.2.0, 1.3.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Critical
 Fix For: 1.2.0, 1.3.0


Requesting the TM log/stdout file via the web interface crashes the TM if the 
respective file does not exist. This is, for example, the case when running 
Flink via DC/OS.

{code}
java.io.FileNotFoundException: flink-taskmanager.out (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at 
org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleRequestTaskManagerLog(TaskManager.scala:833)
at 
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:340)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at 
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at 
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at 
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:122)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5469) build_docs.sh -p fails on Windows Subsystem for Linux

2017-01-12 Thread Chesnay Schepler (JIRA) 
Chesnay Schepler created FLINK-5469:
---

 Summary: build_docs.sh -p fails on Windows Subsystem for Linux
 Key: FLINK-5469
 URL: https://issues.apache.org/jira/browse/FLINK-5469
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.2.0, 1.3.0
Reporter: Chesnay Schepler
Priority: Trivial


As described [here|https://github.com/jekyll/jekyll/issues/5233] jekyll --watch 
(which is executed within build_docs.sh) fails when using it within Ubuntu on 
Windows. Adding --force_polling resolves this issue.

I was wondering whether we couldn't add --force_polling by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

FOSDEM 2017 Open Source Conference - Brussels

2017-01-12 Thread Sharan F 

Hello Everyone

This email is to tell you about ASF participation at FOSDEM. The event 
will be held in Brussels on 4^th & 5^th February 2017 and we are hoping 
that many people from our ASF projects will be there.


https://fosdem.org/2017/

Attending FOSDEM is completely free and the ASF will again be running a 
booth there. Our main focus will on talking to people about the ASF, our 
projects and communities.


*_Why Attend FOSDEM?_*
Some reasons for attending FOSDEM are:

1. Promoting your project: FOSDEM has up to 4-5000 attendees so is a
   great place to spread the word about your project
2. Learning, participating and meeting up: FOSDEM is a developers
   conference so includes presentations covering a range of
   technologies and includes lots of topic specific devrooms

_*FOSDEM Wiki *_
A page on the Community Development wiki has been created with the main 
details about our involvement at conference, so please take a look


https://cwiki.apache.org/confluence/display/COMDEV/FOSDEM+2017

If you would like to spend some time on the ASF booth promoting your 
project then please sign up on the FOSDEM wiki page. Initially we would 
like to split this into slots of 3-4 hours but this will depend on the 
number of projects that are represented.


We are also looking for volunteers to help out on the booth over the 2 
days of the conference, so if you are going to be there and are willing 
to help then please add your name to the volunteer list.


_*Project Stickers*_
If you are going to be at FOSDEM and do not have any project stickers to 
give away then we may (budget permitting) be able to help you get some 
printed. Please contact me with your requirements.


_*Social Event*_
Some people have asked about organising an ASF social event / meetup 
during the conference. This is possible but we will need know how many 
people are interested and which date works best. The FOSDEM wiki page 
also contains an 'Arrival / Departure' section so so please add your 
details if you would like to participate.


I hope this helps people see some of the advantages of attending FOSDEM 
and we are looking forward to seeing lots of people there from our ASF 
communities.


Thanks
Sharan

Apache Community Development
http://community.apache.org/

[jira] [Created] (FLINK-5468) Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with ClassNotFoundException

2017-01-12 Thread Robert Metzger (JIRA) 
Robert Metzger created FLINK-5468:
-

 Summary: Restoring from a semi async rocksdb statebackend (1.1) to 
1.2 fails with ClassNotFoundException
 Key: FLINK-5468
 URL: https://issues.apache.org/jira/browse/FLINK-5468
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.2.0
Reporter: Robert Metzger


I think we should catch this exception and explain what's going on and how 
users can resolve the issue.
{code}
org.apache.flink.client.program.ProgramInvocationException: The program 
execution failed: Job execution failed
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
at 
org.apache.flink.yarn.YarnClusterClient.submitJob(YarnClusterClient.java:210)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
at 
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
at com.dataartisans.eventwindow.Generator.main(Generator.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
at 
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
at 
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution 
failed
at 
org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:328)
at 
org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:382)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
... 22 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException: 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend$FinalSemiAsyncSnapshot
at 
org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:162)
at 
org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:70)
at 
org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.loadSavepoint(SavepointStore.java:138)
at 
org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:64)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1348)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.contrib.streamin

[jira] [Created] (FLINK-5467) Stateless chained tasks set legacy operator state

2017-01-12 Thread Ufuk Celebi (JIRA) 
Ufuk Celebi created FLINK-5467:
--

 Summary: Stateless chained tasks set legacy operator state
 Key: FLINK-5467
 URL: https://issues.apache.org/jira/browse/FLINK-5467
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Reporter: Ufuk Celebi


I discovered this while trying to rescale a job with a Kafka source with a 
chained stateless operator.

Looking into it, it turns out that this fails, because the checkpointed state 
contains legacy operator state for the chained operator although it is state 
less.

/cc [~aljoscha] You mentioned that this might be a possible duplicate?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

2017-01-12 Thread Robert Metzger 
I also found a bunch of issues

https://issues.apache.org/jira/browse/FLINK-5465
https://issues.apache.org/jira/browse/FLINK-5462
https://issues.apache.org/jira/browse/FLINK-5464
https://issues.apache.org/jira/browse/FLINK-5463


On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske  wrote:

> I have another bugfix for 1.2.:
>
> https://issues.apache.org/jira/browse/FLINK-2662 (pending PR)
>
> 2017-01-10 15:16 GMT+01:00 Robert Metzger :
>
> > Hi,
> >
> > this depends a lot on the number of issues we find during the testing.
> >
> >
> > These are the issues I found so far:
> >
> > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved)
> > https://issues.apache.org/jira/browse/FLINK-5383 (resolved)
> > https://issues.apache.org/jira/browse/FLINK-5382 (resolved)
> > https://issues.apache.org/jira/browse/FLINK-5381 (resolved)
> > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR)
> >
> >
> >
> >
> > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui 
> wrote:
> >
> > > Do we have a probable time of 1.2 release? This month or Next month?
> > >
> > > -邮件原件-
> > > 发件人: Robert Metzger [mailto:rmetz...@apache.org]
> > > 发送时间: 2017年1月3日 20:44
> > > 收件人: dev@flink.apache.org
> > > 抄送: u...@flink.apache.org
> > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release
> > candidate)
> > >
> > > Hi,
> > >
> > > First of all, I wish everybody a happy new year 2017.
> > >
> > > I've set user@flink in CC so that users who are interested in helping
> > > with the testing get notified. Please respond only to the dev@ list to
> > > keep the discussion there!
> > >
> > > According to the 1.2 release discussion thread, I've created a first
> > > release candidate for Flink 1.2.
> > > The release candidate will not be the final release, because I'm
> certain
> > > that we'll find at least one blocking issue in the candidate :)
> > >
> > > Therefore, the RC is meant as a testing only release candidate.
> > > Please report every issue we need to fix before the next RC in this
> > thread
> > > so that we have a good overview.
> > >
> > > The release artifacts are located here:
> > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/
> > >
> > > The maven staging repository is located here:
> > > https://repository.apache.org/content/repositories/orgapacheflink-
> > >
> > > The release commit (in branch "release-1.2.0-rc0"):
> > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced
> > >
> > >
> > > Happy testing!
> > >
> >
>

[jira] [Created] (FLINK-5466) Make production environment default in gulpfile

2017-01-12 Thread Ufuk Celebi (JIRA) 
Ufuk Celebi created FLINK-5466:
--

 Summary: Make production environment default in gulpfile
 Key: FLINK-5466
 URL: https://issues.apache.org/jira/browse/FLINK-5466
 Project: Flink
  Issue Type: Improvement
  Components: Webfrontend
Affects Versions: 1.1.4, 1.2.0
Reporter: Ufuk Celebi


Currently the default environment set in our gulpfile is development, which 
lead to very large created JS files. When building the web UI we apparently 
forgot to set the environment to production (build via gulp production).

Since this is likely to occur again, we should make the default environment 
production and make sure to use development manually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

2017-01-12 Thread Fabian Hueske 
I have another bugfix for 1.2.:

https://issues.apache.org/jira/browse/FLINK-2662 (pending PR)

2017-01-10 15:16 GMT+01:00 Robert Metzger :

> Hi,
>
> this depends a lot on the number of issues we find during the testing.
>
>
> These are the issues I found so far:
>
> https://issues.apache.org/jira/browse/FLINK-5379 (unresolved)
> https://issues.apache.org/jira/browse/FLINK-5383 (resolved)
> https://issues.apache.org/jira/browse/FLINK-5382 (resolved)
> https://issues.apache.org/jira/browse/FLINK-5381 (resolved)
> https://issues.apache.org/jira/browse/FLINK-5380 (pending PR)
>
>
>
>
> On Tue, Jan 10, 2017 at 11:58 AM, shijinkui  wrote:
>
> > Do we have a probable time of 1.2 release? This month or Next month?
> >
> > -邮件原件-
> > 发件人: Robert Metzger [mailto:rmetz...@apache.org]
> > 发送时间: 2017年1月3日 20:44
> > 收件人: dev@flink.apache.org
> > 抄送: u...@flink.apache.org
> > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release
> candidate)
> >
> > Hi,
> >
> > First of all, I wish everybody a happy new year 2017.
> >
> > I've set user@flink in CC so that users who are interested in helping
> > with the testing get notified. Please respond only to the dev@ list to
> > keep the discussion there!
> >
> > According to the 1.2 release discussion thread, I've created a first
> > release candidate for Flink 1.2.
> > The release candidate will not be the final release, because I'm certain
> > that we'll find at least one blocking issue in the candidate :)
> >
> > Therefore, the RC is meant as a testing only release candidate.
> > Please report every issue we need to fix before the next RC in this
> thread
> > so that we have a good overview.
> >
> > The release artifacts are located here:
> > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/
> >
> > The maven staging repository is located here:
> > https://repository.apache.org/content/repositories/orgapacheflink-
> >
> > The release commit (in branch "release-1.2.0-rc0"):
> > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced
> >
> >
> > Happy testing!
> >
>

Re: Flink CEP development is stalling

2017-01-12 Thread ruben.casado.tejedor 
+1

I have some clientes interested in CEP features

El 11/1/17 16:23, "Ivan Mushketyk"  escribió:

Hi Flink devs,

Roughly half a year ago I implemented several PRs for Flink-CEP[1][2][3]
but since then there were no progress with the reviews. What is frustrating
about this situation is that Flink customers are asking for features in
these PRs. For example customers commented on [1] and [2] asking for these
features. During a presentation about CEP during Flink Forward 2016
somebody asked[4] for a feature that is implemented in [1]. Another CEP
feature that was requested in this Stack Overflow post[5] was implemented
by PR [2].

I also started conversations regarding following JIRA issues:

https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D4641&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=ml7ZfW_GN8zahdrUAEQrRD3KTBbvr6RbPpqTg71Fhao&e=

https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D3414&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=uBNY1zlRRLXhvJQlguoBh4qXZVPRtHxmlavRCLD2UGE&e=

https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D3320&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=7lPpQ1mPiTRr7qz8mX6GK0Qq9CWWeXcg8Bf3h43uJ08&e=
  (wrote to Till about this
one)

and I would like to work on them, but it seems pointless if nobody is going
to review new PRs.

I wrote to Till(who is the only Flink CEP reviewer at the moment) but it
seems that he is very busy and cannot help with these PRs. On the other
hand Flink CEP has got some attention and customers are asking for new
features.

Is there any way for community to make progress with Flink CEP?
Are there other core committers that can review Flink CEP PRs?

Best regards,
Ivan.



[1] - 
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_flink_pull_2361&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=dNLjBUb6wz8125hkAlm3tEzxLnZNfzeRrkFPEFJR2qM&e=
[2] - 
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_flink_pull_2367&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=tpKgiz_VFKYGXCn26GJolxFfSvRwqaMSpOPYFqG3ZYI&e=
[3] - 
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_flink_pull_2396&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=28ZX8-B728xOrkxZ1DxhnOZfaZOB30fQQxWzJV5lrXY&e=
[4] - 
https://urldefense.proofpoint.com/v2/url?u=https-3A__youtu.be_vws5bv3XdD8-3Ft-3D35m26s&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=NQ0d1W3QqhUU5tMKy7vqbLWPtYbFQ-pOLdsEEsN6ugc&e=
[5] -

https://urldefense.proofpoint.com/v2/url?u=http-3A__stackoverflow.com_questions_38225286_ho-2Dcan-2Di-2Ddo-2Da-2Dlazy-2Dmatch-2Dwith-2Dflink-2Dcep&d=DgIBaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=brkRAgrW3LbdVDOiRLzI7SFUIWBL5aa2MIfENljA8xoe0lFg2u3-S6GnFTH7Pbmc&m=OvPHe08A8IQPc6PIPmNkgZWKDWmAMAIkfcqrek_iJbQ&s=Q4FXZ_t5rW5UZt-geVcKjq7QNf4TgEluZlGAXxtzXN0&e=





This message is for the designated recipient only and may contain privileged, 
proprietary, or otherwise confidential information. If you have received it in 
error, please notify the sender immediately and delete the original. Any other 
use of the e-mail by you is prohibited. Where allowed by local law, electronic 
communications with Accenture and its affiliates, including e-mail and instant 
messaging (including content), may be scanned by our systems for the purposes 
of information security and assessment of internal compliance with Accenture 
policy.
__

www.accenture.com