Re: java.lang.IllegalArgumentException: JDBC-Class not found. - org.postgresql.jdbc.Driver

2016-10-12 Thread Fabian Hueske 
Hi Sunny,

please avoid crossposting to all mailing lists.
The dev@f.a.o list is for issues related to the development of Flink not
the development of Flink applications.

The error message is actually quite descriptive. Flink does not find the
JDBC driver class.
You need to add it to the classpath for example by adding the corresponding
Maven dependency to your pom file.

Fabian


2016-10-12 23:18 GMT+02:00 sunny patel :

>
> Hi Guys,
>
> I am facing JDBC error, could you please some one advise me on this error?
>
> $ java -version
>
> java version "1.8.0_102"
>
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
>
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
>
> $ scala -version
>
> Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
>
>
> === Scala Code
>
> import org.apache.flink.api.common.typeinfo.TypeInformation
> import org.apache.flink.api.java.io.jdbc.JDBCInputFormat
> import org.apache.flink.api.scala._
> import org.apache.flink.api.table.typeutils.RowTypeInfo
>
> object WordCount {
>   def main(args: Array[String]) {
>
> val PATH = getClass.getResource("").getPath
>
> // set up the execution environment
> val env = ExecutionEnvironment.getExecutionEnvironment
>
> // Read data from JDBC (Kylin in our case)
> val stringColum: TypeInformation[Int] = createTypeInformation[Int]
> val DB_ROWTYPE = new RowTypeInfo(Seq(stringColum))
>
> val inputFormat = JDBCInputFormat.buildJDBCInputFormat()
>   .setDrivername("org.postgresql.jdbc.Driver")
>   .setDBUrl("jdbc:postgresql://localhost:5432/mydb")
>   .setUsername("MI")
>   .setPassword("MI")
>   .setQuery("select * FROM identity")
>   .setRowTypeInfo(DB_ROWTYPE)
>   .finish()
>
> val dataset =env.createInput(inputFormat)
> dataset.print()
>
> println(PATH)
>   }
> }
>
> ==
>
> ==POM.XML
>
>
> 
> http://maven.apache.org/POM/4.0.0; 
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>4.0.0
>
>   flink-parent
>   org.apache.flink
>   1.2-SNAPSHOT
>
>
>org.apache.flink.quickstart
>flink-scala-project
>0.1
>jar
>
>Flink Quickstart Job
>http://www.myorganization.org
>
>
>   
>  apache.snapshots
>  Apache Development Snapshot Repository
>  
> https://repository.apache.org/content/repositories/snapshots/
>  
> false
>  
>  
>  
>   
>
>
>
>   UTF-8
>   1.1.2
>
>
>
>
>
>   
>  org.apache.flink
>  flink-jdbc
>  ${flink.version}
>   
>   
>  org.apache.flink
>  flink-table_2.11
>  ${flink.version}
>   
>   
>  org.apache.flink
>  flink-scala_2.11
>  ${flink.version}
>   
>   
>  org.apache.flink
>  flink-streaming-scala_2.11
>  ${flink.version}
>   
>   
>  org.apache.flink
>  flink-clients_2.11
>  ${flink.version}
>   
>
>
>
>   
>  
>  build-jar
>  
>  
>  
> 
>org.apache.flink
>flink-scala_2.11
>${flink.version}
>provided
> 
> 
>org.apache.flink
>flink-streaming-scala_2.11
>${flink.version}
>provided
> 
> 
>org.apache.flink
>flink-clients_2.11
>${flink.version}
>provided
> 
>  
>
>  
> 
>
>
>   org.apache.maven.plugins
>   maven-shade-plugin
>   2.4.1
>   
>  
> package
> 
>shade
> 
> 
>
>   
>
> 
>  
>   
>
> 
>  
>   
>
>
>
>
>   
>  
>  
> org.apache.maven.plugins
> maven-shade-plugin
> 2.4.1
> 
>
>
>   package
>   
>  shade
>   
>   
>  
> 
>
>
> org.apache.flink:flink-annotations
>
> org.apache.flink:flink-shaded-hadoop1_2.11
>

Re: Exception from in-progress implementation of Python API bulk iterations

2016-10-12 Thread Geoffrey Mon 
Hello,

Has anyone had a chance to look into this? I am currently working on the
problem but I have minimal understanding of how the internal Flink Python
API works; any expertise would be greatly appreciated.

Thank you very much!

Geoffrey

On Sat, Oct 8, 2016 at 1:27 PM Geoffrey Mon  wrote:

> Hi Chesnay,
>
> Heh, I have discovered that if I do not restart Flink after running my
> original problematic script, then similar issues will manifest themselves
> in other otherwise working scripts. I haven't been able to completely
> narrow down the problem, but I promise this new script will have a
> ClassCastException that is completely reproducible. :)
> https://gist.github.com/GEOFBOT/abb7f81030aab160e6908093ebaa3b4a
>
> Thanks,
> Geoffrey
>
> On Wed, Sep 28, 2016 at 9:16 AM Chesnay Schepler 
> wrote:
>
> Hello Geoffrey,
>
> this one works for me as well :D
>
> Regards,
> Chesnay
>
> On 28.09.2016 05:38, Geoffrey Mon wrote:
> > Hello Chesnay,
> >
> > Thank you for your help. After receiving your message I recompiled my
> > version of Flink completely, and both the NullPointerException listed in
> > the TODO and the ClassCastException with the join operation went away.
> > Previously, I had been only recompiling the modules of Flink that had
> been
> > changed to save time using "mvn clean install -pl :module" and apparently
> > that may have been causing some of my issues.
> >
> > Now, the problem is more clear: when a specific group reduce function in
> my
> > research project plan file is used within an iteration, I get a
> > ClassCastException exception:
> > Caused by: java.lang.ClassCastException:
> > org.apache.flink.api.java.tuple.Tuple2 cannot be cast to [B
> > at
> >
> org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer.serialize(BytePrimitiveArraySerializer.java:31)
> > at
> > org.apache.flink.runtime.iterative.io
> .WorksetUpdateOutputCollector.collect(WorksetUpdateOutputCollector.java:58)
> > at
> >
> org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
> > at
> >
> org.apache.flink.python.api.streaming.data.PythonReceiver.collectBuffer(PythonReceiver.java:96)
> > at
> >
> org.apache.flink.python.api.streaming.data.PythonStreamer.streamBufferWithoutGroups(PythonStreamer.java:272)
> > at
> >
> org.apache.flink.python.api.functions.PythonMapPartition.mapPartition(PythonMapPartition.java:54)
> > at
> >
> org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
> > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:486)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:146)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailTask.run(IterationTailTask.java:107)
> > at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:351)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:591)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > I'm not sure why this is causing an exception, and I would greatly
> > appreciate any assistance. I've revised the barebones error-causing plan
> > file to focus on this new error source:
> > https://gist.github.com/GEOFBOT/abb7f81030aab160e6908093ebaa3b4a
> > The group reduce function in question seems to work just fine outside of
> > iterations. I have organized the commits and pushed to a new branch to
> make
> > it easier to test and hopefully review soon:
> > https://github.com/GEOFBOT/flink/tree/new-iterations
> >
> > Cheers,
> > Geoffrey
> >
> > On Mon, Sep 26, 2016 at 6:32 AM Chesnay Schepler 
> wrote:
> >
> >> Hello Geoffrey,
> >>
> >> i could not reproduce this issue with the commits and plan you provided.
> >>
> >> I tried out both the FLINK-4098 and bulk-iterations branches (and
> >> reverted back to the specified commits) and built Flink from scratch.
> >>
> >> Could you double check that the code you provided produces the error?
> >> Also, which OS/python version are you using?
> >>
> >> Regards,
> >> Chesnay
> >>
> >> On 20.09.2016 11:13, Chesnay Schepler wrote:
> >>> Hello,
> >>>
> >>> I'll try to take a look this week.
> >>>
> >>> Regards,
> >>> Chesnay
> >>>
> >>> On 20.09.2016 02:38, Geoffrey Mon wrote:
>  Hello all,
> 
>  I have recently been working on adding bulk iterations to the Python
>  API of
>  Flink in order to facilitate a research project I am working on. The
>  current changes can be seen in this GitHub diff:
> 
> >>
> https://github.com/apache/flink/compare/master...GEOFBOT:e8c9833b43675af66ce897da9880c4f8cd16aad0
> 
>  This implementation seems to work for, at least, simple examples,
>  such as
>  incrementing numbers in a data set. However, with the transformations
>  required for my project, I get an exception
>  "java.lang.ClassCastException:
>  [B cannot be cast to org.apache.flink.api.java.tuple.Tuple"

java.lang.IllegalArgumentException: JDBC-Class not found. - org.postgresql.jdbc.Driver

2016-10-12 Thread sunny patel 
Hi Guys,

I am facing JDBC error, could you please some one advise me on this error?

$ java -version

java version "1.8.0_102"

Java(TM) SE Runtime Environment (build 1.8.0_102-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

$ scala -version

Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL


=== Scala Code

import org.apache.flink.api.common.typeinfo.TypeInformation
import org.apache.flink.api.java.io.jdbc.JDBCInputFormat
import org.apache.flink.api.scala._
import org.apache.flink.api.table.typeutils.RowTypeInfo

object WordCount {
  def main(args: Array[String]) {

val PATH = getClass.getResource("").getPath

// set up the execution environment
val env = ExecutionEnvironment.getExecutionEnvironment

// Read data from JDBC (Kylin in our case)
val stringColum: TypeInformation[Int] = createTypeInformation[Int]
val DB_ROWTYPE = new RowTypeInfo(Seq(stringColum))

val inputFormat = JDBCInputFormat.buildJDBCInputFormat()
  .setDrivername("org.postgresql.jdbc.Driver")
  .setDBUrl("jdbc:postgresql://localhost:5432/mydb")
  .setUsername("MI")
  .setPassword("MI")
  .setQuery("select * FROM identity")
  .setRowTypeInfo(DB_ROWTYPE)
  .finish()

val dataset =env.createInput(inputFormat)
dataset.print()

println(PATH)
  }
}

==

==POM.XML



http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
   4.0.0
   
  flink-parent
  org.apache.flink
  1.2-SNAPSHOT
   

   org.apache.flink.quickstart
   flink-scala-project
   0.1
   jar

   Flink Quickstart Job
   http://www.myorganization.org

   
  
 apache.snapshots
 Apache Development Snapshot Repository
 
https://repository.apache.org/content/repositories/snapshots/
 
false
 
 
 
  
   

   
  UTF-8
  1.1.2
   

   

   
  
 org.apache.flink
 flink-jdbc
 ${flink.version}
  
  
 org.apache.flink
 flink-table_2.11
 ${flink.version}
  
  
 org.apache.flink
 flink-scala_2.11
 ${flink.version}
  
  
 org.apache.flink
 flink-streaming-scala_2.11
 ${flink.version}
  
  
 org.apache.flink
 flink-clients_2.11
 ${flink.version}
  
   

   
  
 
 build-jar
 
 
 

   org.apache.flink
   flink-scala_2.11
   ${flink.version}
   provided


   org.apache.flink
   flink-streaming-scala_2.11
   ${flink.version}
   provided


   org.apache.flink
   flink-clients_2.11
   ${flink.version}
   provided

 

 

   
   
  org.apache.maven.plugins
  maven-shade-plugin
  2.4.1
  
 
package

   shade


   
  
   

 
  
   

 
  
   

   
   
  
 
 
org.apache.maven.plugins
maven-shade-plugin
2.4.1

   
   
  package
  
 shade
  
  
 

   
   org.apache.flink:flink-annotations

org.apache.flink:flink-shaded-hadoop1_2.11

org.apache.flink:flink-shaded-hadoop2

org.apache.flink:flink-shaded-curator-recipes
   org.apache.flink:flink-core
   org.apache.flink:flink-java
   org.apache.flink:flink-scala_2.11

org.apache.flink:flink-runtime_2.11

org.apache.flink:flink-optimizer_2.11

org.apache.flink:flink-clients_2.11
   org.apache.flink:flink-avro_2.11

org.apache.flink:flink-examples-batch_2.11

org.apache.flink:flink-examples-streaming_2.11

org.apache.flink:flink-streaming-java_2.11

   

   org.scala-lang:scala-library
   org.scala-lang:scala-compiler
   org.scala-lang:scala-reflect
   com.typesafe.akka:akka-actor_*

[jira] [Created] (FLINK-4820) Slf4j / log4j version upgrade to support dynamic change of log levels.

2016-10-12 Thread Zhenzhong Xu (JIRA) 
Zhenzhong Xu created FLINK-4820:
---

 Summary: Slf4j / log4j version upgrade to support dynamic change 
of log levels.
 Key: FLINK-4820
 URL: https://issues.apache.org/jira/browse/FLINK-4820
 Project: Flink
  Issue Type: Task
Reporter: Zhenzhong Xu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4819) Checkpoint metadata+data inspection tool (view / update)

2016-10-12 Thread Zhenzhong Xu (JIRA) 
Zhenzhong Xu created FLINK-4819:
---

 Summary: Checkpoint metadata+data inspection tool (view / update)
 Key: FLINK-4819
 URL: https://issues.apache.org/jira/browse/FLINK-4819
 Project: Flink
  Issue Type: Task
  Components: State Backends, Checkpointing
Reporter: Zhenzhong Xu


Checkpoint inspection tool for operationalization, troubleshooting, 
diagnostics, etc, or performing brain surgery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4818) RestartStrategy should track how many failed restore attempts the same checkpoint has and fall back to earlier checkpoints

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4818:
---

 Summary: RestartStrategy should track how many failed restore 
attempts the same checkpoint has and fall back to earlier checkpoints
 Key: FLINK-4818
 URL: https://issues.apache.org/jira/browse/FLINK-4818
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
Reporter: Stephan Ewen


The restart strategies can use the exception information from FLINK-4816 to 
keep track of how often a checkpoint restore has failed. After a certain number 
of consecutive failures, they should take earlier completed checkpoints as 
recovery points.

It is up to discussion whether the restart strategies are the right place to 
implement that, or whether this is an orthogonal feature that should go into 
the checkpoint coordinator (which knows how many checkpoints are available) or 
a separate class altogether.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4817) Checkpoint Coordinator should be called to restore state with a specific checkpoint ID

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4817:
---

 Summary: Checkpoint Coordinator should be called to restore state 
with a specific checkpoint ID
 Key: FLINK-4817
 URL: https://issues.apache.org/jira/browse/FLINK-4817
 Project: Flink
  Issue Type: Sub-task
  Components: State Backends, Checkpointing
Reporter: Stephan Ewen


Rather than being called to restore the "latest" checkpoint, the Checkpoint 
Coordinator should be called to restore a specific checkpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4816) Executions from "DEPLOYING" should retain restored checkpoint information

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4816:
---

 Summary: Executions from "DEPLOYING" should retain restored 
checkpoint information
 Key: FLINK-4816
 URL: https://issues.apache.org/jira/browse/FLINK-4816
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
Reporter: Stephan Ewen


When an execution fails from state {{DEPLOYING}}, it should wrap the failure to 
better report the failure cause:
  - If no checkpoint was restored, it should wrap the exception in a 
{{DeployTaskException}}
  - If a checkpoint was restored, it should wrap the exception in a 
{{RestoreTaskException}} and record the id of the checkpoint that was attempted 
to be restored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4815) Automatic fallback to earlier checkpoints when checkpoint restore fails

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4815:
---

 Summary: Automatic fallback to earlier checkpoints when checkpoint 
restore fails
 Key: FLINK-4815
 URL: https://issues.apache.org/jira/browse/FLINK-4815
 Project: Flink
  Issue Type: New Feature
  Components: State Backends, Checkpointing
Reporter: Stephan Ewen


Flink should keep multiple completed checkpoints.
When the restore of one completed checkpoint fails for a certain number of 
times, the CheckpointCoordinator should fall back to an earlier checkpoint to 
restore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE：[DISCUSS] FLIP-11: Table API Stream Aggregations

2016-10-12 Thread 王绍翾(大沙) 
Hi Fabian, Timo, and Jark.Thanks for kicking off this FLIP. This is a really 
great and promising proposal. I have a few comments to the "window" operator 
proposed in this FLIP (I am hoping it is not too late to bring up this). First, 
window is not always needed for the stream aggregation. There are cases where 
we want do an aggreation on a stream, while the query/emit strategy decides 
when to emit a streaming output. Second, window is needed when we want do an 
aggregation for a certain rage, but window is not an operator. We basically use 
window to define the range for aggregation. In tableAPI, a window should be 
defined together with "groupby" and "select" operators, either inside a 
"groupby" operator or after an "over" clause in "select" operator. This will 
make the TableAPI in the similar manner as SQL.
For instance,[A groupby without window]

val res = tab
.groupBy(‘a)
.select(‘a, ‘b.sum)

SELECT a, SUM(b)
FROM tab
GROUP BY a
[A tumble window inside groupby]
val res = tab
.groupBy(‘a, tumble(10.minutes, ‘rowtime))
.select(‘a, ‘b.sum)
SELECT a, SUM(b)FROM tab GROUP BY a, TUMBLE(10.minutes , ‘rowtime)
[A row tumble window after OVER]
.groupby('a) //optional
.select(‘a, ‘b.count over rowTumble(10.minutes, ‘rowtime))SELECT a, 
COUNT(b) OVER ROWTUMBLE(10.minutes, ‘rowtime)FROM tab GROUP BY a
Please let me know what you think.
Regards,Shaoxuan
--发件人：Fabian 
Hueske 发送时间：2016年9月26日(星期一) 21:13收件人：dev@flink.apache.org 
主　题：Re: [DISCUSS] FLIP-11: Table API Stream Aggregations
Hi everybody,

Timo proposed our FLIP-11 a bit more than three weeks ago.
I will update the status of the FLIP to accepted.

Thanks,
Fabian

2016-09-19 9:16 GMT+02:00 Timo Walther :

> Hi Jark,
>
> yes I think enough time has passed. We can start implementing the changes.
> What do you think Fabian?
>
> If there are no objections, I will create the subtasks in Jira today. For
> FLIP-11/1 I already have implemented a prototype, I just have to do some
> refactoring/documentation before opening a PR.
>
> Timo
>
>
> Am 18/09/16 um 04:46 schrieb Jark Wu:
>
> Hi all,
>>
>> It seems that there’s no objections to the window design. So could we
>> open subtasks to start working on it now ?
>>
>> - Jark Wu
>>
>> 在 2016年9月7日，下午4:29，Jark Wu  写道：
>>>
>>> Hi Fabian,
>>>
>>> Thanks for sharing your ideas.
>>>
>>> They all make sense to me. Regarding to reassigning timestamp, I do not
>>> have an use case. I come up with this because DataStream has a
>>> TimestampAssigner :)
>>>
>>> +1 for this FLIP.
>>>
>>> - Jark Wu
>>>
>>> 在 2016年9月7日，下午2:59，Fabian Hueske >> fhue...@gmail.com>> 写道：

 Hi,

 thanks for your comments and questions!
 Actually, you are bringing up the points that Timo and I discussed the
 most
 when designing the FLIP ;-)

 - We also thought about the syntactic shortcut for running aggregates
 like
 you proposed (table.groupBy(‘a).select(…)). Our motivation to not allow
 this shortcut is to prevent users from accidentally performing a
 "dangerous" operation. The problem with unbounded sliding row-windows is
 that their state does never expire. If you have an evolving key space,
 you
 will likely run into problems at some point because the operator state
 grows too large. IMO, a row-window session is a better approach,
 because it
 defines a timeout after which state can be discarded. groupBy.select is
 a
 very common operation in batch but its semantics in streaming are very
 different. In my opinion it makes sense to make users aware of these
 differences through the API.

 - Reassigning timestamps and watermarks is a very delicate issue. You
 are
 right, that Calcite exposes this field which is necessary due to the
 semantics of SQL. However, also in Calcite you cannot freely choose the
 timestamp attribute for streaming queries (it must be a monotone or
 quasi-monotone attribute) which is hard to reason about (and guarantee)
 after a few operators have been applied. Streaming tables in Flink will
 likely have a time attribute which is identical to the initial rowtime.
 However, Flink does modify timestamps internally, e.g., for records that
 are emitted from time windows, in order to ensure that consecutive
 windows
 perform as expected. Modify or reassign timestamps in the middle of a
 job
 can result in unexpected results which are very hard to reason about. Do
 you have a concrete use case in mind for reassigning timestamps?

 - The idea to represent rowtime and systime as object is good. Our
 motivation to go for reserved Scala symbols was to have a uniform syntax
 with windows over streaming and batch tables. On batch tables you can
 compute time windows basically over 

[RESULT] [VOTE] Release Apache Flink 1.1.3 (RC2)

2016-10-12 Thread Ufuk Celebi 
This vote has passed with 5 binding +1 and 1 non-binding +1 vote.
Thanks to everyone who was involved. I'll go ahead and finalize and
package this release.

+1s:
Robert Metzger  (binding)
Maximilian Michels  (binding)
Fabian Hueske  (binding)
Stephan Ewen  (binding)
Neelesh Salian (non-binding)
Till Rohrmann (binding)

There are no 0s or -1s.


On Wed, Oct 12, 2016 at 6:53 PM, Till Rohrmann  wrote:
> +1 (binding)
>
> - mvn clean verify for (Scala 2.10, Hadoop 2.3.0), (Scala 2.11, Hadoop
> 2.4.1), (Scala 2.10, Hadoop 2.6.3), (Scala 2.11, Hadoop 2.7.2)
> - Run examples using the FliRRT tool on multi node cluster.
>
> Cheers,
> Till
>
> On Wed, Oct 12, 2016 at 5:32 PM, Neelesh Salian 
> wrote:
>
>> + 1 (non-binding)
>>
>> -  For Hadoop 2.x: mvn clean install -DskipTests -Dhadoop.version=2.6.3
>> - For Hadoop 1.x: mvn clean install -DskipTests -Dhadoop.profile=1
>> - For Scala 2.11: tools/change-scala-version.sh 2.11, mvn clean install
>> -DskipTests
>>
>> Ran Examples for Batch and Streaming after initiating the local and cluster
>> setup.
>>
>> Thank you for working on this.
>>
>>
>> On Wed, Oct 12, 2016 at 6:53 AM, Stephan Ewen  wrote:
>>
>> > +1 (binding)
>> >
>> >  - Verified that no changed to LICENSE or NOTICE necessary since the last
>> > release
>> >  - mvn clean verily for Scala 2.11, Hadoop 2.6.3 including YARN tests
>> >
>> >
>> > On Wed, Oct 12, 2016 at 3:22 PM, Fabian Hueske 
>> wrote:
>> >
>> > > +1 to release (binding)
>> > >
>> > > - checked hashes and signatures
>> > > - checked diffs against 1.1.2: no dependencies added or modified
>> > > - successfully built Flink from source archive
>> > >   - mvn clean install (Scala 2.10)
>> > >
>> > > Cheers, Fabian
>> > >
>> > > 2016-10-12 14:05 GMT+02:00 Maximilian Michels :
>> > >
>> > > > +1 (binding)
>> > > >
>> > > > - scanned commit history for changes
>> > > > - ran "mvn clean install -Dhadoop.version=2.6.0 -Pinclude-yarn-tests"
>> > > > successfully
>> > > > - started cluster via "./bin/start-cluster.sh"
>> > > > - run batch and streaming examples via web interface and CLI
>> > > > - used web interface for monitoring
>> > > > - ran example job with quickstart project and staging repository
>> > > > (cleaned .m2 cache for 1.1.3 before doing that)
>> > > > - various tests with ContinuousFileMonitoringFunction and verified
>> the
>> > > > fix from the previous RC1
>> > > >
>> > > >
>> > > > -Max
>> > > >
>> > > >
>> > > > On Tue, Oct 11, 2016 at 4:40 PM, Robert Metzger > >
>> > > > wrote:
>> > > > > +1 for releasing this as Flink 1.1.3
>> > > > >
>> > > > > - Checked the staging repository for hadoop2 / hadoop1 mixup;
>> > > quickstart
>> > > > > version; build a test project against repository
>> > > > > - Checked the artifacts:
>> > > > >- src doesn't contain any binaries
>> > > > >- started Flink locally & executed example & checked web
>> interface
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Oct 10, 2016 at 6:52 PM, Ufuk Celebi 
>> wrote:
>> > > > >
>> > > > >> Dear Flink community,
>> > > > >>
>> > > > >> Please vote on releasing the following candidate as Apache Flink
>> > > version
>> > > > >> 1.1.3.
>> > > > >>
>> > > > >> The commit to be voted on:
>> > > > >> 8e8d454 (http://git-wip-us.apache.org/
>> > repos/asf/flink/commit/8e8d454)
>> > > > >>
>> > > > >> Branch:
>> > > > >> release-1.1.3-rc2
>> > > > >> (https://git1-us-west.apache.org/repos/asf/flink/repo?p=flin
>> > > > >> k.git;a=shortlog;h=refs/heads/release-1.1.3-rc2)
>> > > > >>
>> > > > >> The release artifacts to be voted on can be found at:
>> > > > >> http://people.apache.org/~uce/flink-1.1.3-rc2/
>> > > > >>
>> > > > >> The release artifacts are signed with the key with fingerprint
>> > > 9D403309:
>> > > > >> http://www.apache.org/dist/flink/KEYS
>> > > > >>
>> > > > >> The staging repository for this release can be found at:
>> > > > >> https://repository.apache.org/content/repositories/orgapache
>> > > flink-1106
>> > > > >>
>> > > > >> -
>> > > > >>
>> > > > >> RC2 adds two new commits since RC1. If there are no objections, I
>> > > > >> would like to reduce the voting time to (at least) 2 days. The
>> vote
>> > > > >> passes if a majority of at least three +1 PMC votes are cast.
>> > > > >>
>> > > > >> The vote ends on Wed, October 12th, 2016.
>> > > > >>
>> > > > >> [ ] +1 Release this package as Apache Flink 1.1.3
>> > > > >> [ ] -1 Do not release this package, because ...
>> > > > >>
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Neelesh Srinivas Salian
>> Customer Operations Engineer
>>

Re: [VOTE] Release Apache Flink 1.1.3 (RC2)

2016-10-12 Thread Till Rohrmann 
+1 (binding)

- mvn clean verify for (Scala 2.10, Hadoop 2.3.0), (Scala 2.11, Hadoop
2.4.1), (Scala 2.10, Hadoop 2.6.3), (Scala 2.11, Hadoop 2.7.2)
- Run examples using the FliRRT tool on multi node cluster.

Cheers,
Till

On Wed, Oct 12, 2016 at 5:32 PM, Neelesh Salian 
wrote:

> + 1 (non-binding)
>
> -  For Hadoop 2.x: mvn clean install -DskipTests -Dhadoop.version=2.6.3
> - For Hadoop 1.x: mvn clean install -DskipTests -Dhadoop.profile=1
> - For Scala 2.11: tools/change-scala-version.sh 2.11, mvn clean install
> -DskipTests
>
> Ran Examples for Batch and Streaming after initiating the local and cluster
> setup.
>
> Thank you for working on this.
>
>
> On Wed, Oct 12, 2016 at 6:53 AM, Stephan Ewen  wrote:
>
> > +1 (binding)
> >
> >  - Verified that no changed to LICENSE or NOTICE necessary since the last
> > release
> >  - mvn clean verily for Scala 2.11, Hadoop 2.6.3 including YARN tests
> >
> >
> > On Wed, Oct 12, 2016 at 3:22 PM, Fabian Hueske 
> wrote:
> >
> > > +1 to release (binding)
> > >
> > > - checked hashes and signatures
> > > - checked diffs against 1.1.2: no dependencies added or modified
> > > - successfully built Flink from source archive
> > >   - mvn clean install (Scala 2.10)
> > >
> > > Cheers, Fabian
> > >
> > > 2016-10-12 14:05 GMT+02:00 Maximilian Michels :
> > >
> > > > +1 (binding)
> > > >
> > > > - scanned commit history for changes
> > > > - ran "mvn clean install -Dhadoop.version=2.6.0 -Pinclude-yarn-tests"
> > > > successfully
> > > > - started cluster via "./bin/start-cluster.sh"
> > > > - run batch and streaming examples via web interface and CLI
> > > > - used web interface for monitoring
> > > > - ran example job with quickstart project and staging repository
> > > > (cleaned .m2 cache for 1.1.3 before doing that)
> > > > - various tests with ContinuousFileMonitoringFunction and verified
> the
> > > > fix from the previous RC1
> > > >
> > > >
> > > > -Max
> > > >
> > > >
> > > > On Tue, Oct 11, 2016 at 4:40 PM, Robert Metzger  >
> > > > wrote:
> > > > > +1 for releasing this as Flink 1.1.3
> > > > >
> > > > > - Checked the staging repository for hadoop2 / hadoop1 mixup;
> > > quickstart
> > > > > version; build a test project against repository
> > > > > - Checked the artifacts:
> > > > >- src doesn't contain any binaries
> > > > >- started Flink locally & executed example & checked web
> interface
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 10, 2016 at 6:52 PM, Ufuk Celebi 
> wrote:
> > > > >
> > > > >> Dear Flink community,
> > > > >>
> > > > >> Please vote on releasing the following candidate as Apache Flink
> > > version
> > > > >> 1.1.3.
> > > > >>
> > > > >> The commit to be voted on:
> > > > >> 8e8d454 (http://git-wip-us.apache.org/
> > repos/asf/flink/commit/8e8d454)
> > > > >>
> > > > >> Branch:
> > > > >> release-1.1.3-rc2
> > > > >> (https://git1-us-west.apache.org/repos/asf/flink/repo?p=flin
> > > > >> k.git;a=shortlog;h=refs/heads/release-1.1.3-rc2)
> > > > >>
> > > > >> The release artifacts to be voted on can be found at:
> > > > >> http://people.apache.org/~uce/flink-1.1.3-rc2/
> > > > >>
> > > > >> The release artifacts are signed with the key with fingerprint
> > > 9D403309:
> > > > >> http://www.apache.org/dist/flink/KEYS
> > > > >>
> > > > >> The staging repository for this release can be found at:
> > > > >> https://repository.apache.org/content/repositories/orgapache
> > > flink-1106
> > > > >>
> > > > >> -
> > > > >>
> > > > >> RC2 adds two new commits since RC1. If there are no objections, I
> > > > >> would like to reduce the voting time to (at least) 2 days. The
> vote
> > > > >> passes if a majority of at least three +1 PMC votes are cast.
> > > > >>
> > > > >> The vote ends on Wed, October 12th, 2016.
> > > > >>
> > > > >> [ ] +1 Release this package as Apache Flink 1.1.3
> > > > >> [ ] -1 Do not release this package, because ...
> > > > >>
> > > >
> > >
> >
>
>
>
> --
> Neelesh Srinivas Salian
> Customer Operations Engineer
>

[jira] [Created] (FLINK-4814) Remove extra storage location for externalized checkpoint metadata

2016-10-12 Thread Ufuk Celebi (JIRA) 
Ufuk Celebi created FLINK-4814:
--

 Summary: Remove extra storage location for externalized checkpoint 
metadata
 Key: FLINK-4814
 URL: https://issues.apache.org/jira/browse/FLINK-4814
 Project: Flink
  Issue Type: Improvement
Reporter: Ufuk Celebi


Follow up for FLINK-4512.

Store checkpoint meta data in checkpoint directory.  That makes it simpler for 
users to track and clean up checkpoints manually, if they want to retain 
externalized checkpoints across cancellations and terminal failures.

Every state backend needs to be able to provide a storage location for the 
checkpoint metadata. The memory state backend would hence not work with 
externalized checkpoints, unless one sets explicitly a parameter 
`setExternalizedCheckpointsLocation(uri)`.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Removing delete*Timer from the WindowOperator.Context

2016-10-12 Thread Kostas Kloudas 
Thanks for the feedback Konstantin!
Good to hear that.

As far as the Trigger DSL is concerned, 
it is not currently in the master but it will come soon.

Kostas

> On Oct 12, 2016, at 6:05 PM, Konstantin Knauf  
> wrote:
> 
> Hi all,
> 
> thank you for looping me in. Because of the memory leak we first
> experienced we have built a work-around, which did not need to delete
> timers and are still using it. So for us, I think, this would currently
> not be a problem. Nevertheless, I think, it is a strong limitation if
> custom triggers can not delete timers. I am not familiar with the new
> Trigger DSL though.
> 
> Cheers,
> 
> Konstantin
> 
> On 12.10.2016 15:38, Kostas Kloudas wrote:
>> Hi all,
>> 
>> This thread has been dormant for some time now.
>> 
>> Given that this change may affect user code, I am sending this as a 
>> reminder that the discussion is still open and to re-invite anyone who
>> may be affected to participate.
>> 
>> I would suggest to leave it open till the end of next week and then, 
>> if nobody objects, we can proceed to the change.
>> 
>> What do you think?
>> 
>> Kostas
>> 
>>> On Sep 28, 2016, at 3:21 PM, Maximilian Michels  wrote:
>>> 
>>> What are the use cases where you actually need to delete a timer? How
>>> about we only let users delete timers which they created themselves?
>>> 
>>> I guessing most of these use cases will be obsolete with the new
>>> Trigger DSL because the trigger logic can be expressed more easily. So
>>> +1 for removing the delete methods from the context.
>>> 
>>> On Tue, Sep 27, 2016 at 3:43 PM, Kostas Kloudas
>>>  wrote:
 Hi all,
 
 As the title of this email suggests, I am proposing to remove the  methods
 deleteProcessingTimeTimer(long time) and deleteEventTimeTimer(long time)
 from the WindowOperator.Context. With this change, registered timers that
 have nothing to do (e.g. because their state has already been cleaned up)
 will be simply ignored by the windowOperator, when their time comes.
 
 The reason for the change is that by allowing custom user code, e.g. a 
 custom Trigger,
 to delete timers we may have unpredictable behavior.
 
 As an example, one can imagine the case where we have allowed_lateness = 0 
 and the cleanup
 timer for a window collides with the end_of_window one. In this case, by 
 deleting the end_of_window
 timer from the trigger (possibly a custom one), we end up also deleting 
 the cleanup one,
 which in turn can lead to the window state never being garbage collected.
 
 To see what can be the consequences apart from memory leaks, this can 
 easily lead
 to wrong session windows, as a session that should have been garbage 
 collected, will
 still be around and ready to accept new data.
 
 With this change, timers that should correctly be deleted will now remain 
 in the queue of
 pending timers, but they will do nothing, while cleanup timers will 
 cleanup the state of their
 corresponding window.
 
 Other possible solutions like keeping a separate list for cleanup timers 
 would complicate
 the codebase and also introduce memory overheads which can be avoided 
 using the
 solution above (i.e. just ignoring timers the have nothing to do anymore).
 
 What do you think?
 
 Kostas
 
>> 
>> 
> 
> -- 
> Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>

Re: [DISCUSS] Removing delete*Timer from the WindowOperator.Context

2016-10-12 Thread Konstantin Knauf 
Hi all,

thank you for looping me in. Because of the memory leak we first
experienced we have built a work-around, which did not need to delete
timers and are still using it. So for us, I think, this would currently
not be a problem. Nevertheless, I think, it is a strong limitation if
custom triggers can not delete timers. I am not familiar with the new
Trigger DSL though.

Cheers,

Konstantin

On 12.10.2016 15:38, Kostas Kloudas wrote:
> Hi all,
> 
> This thread has been dormant for some time now.
> 
> Given that this change may affect user code, I am sending this as a 
> reminder that the discussion is still open and to re-invite anyone who
> may be affected to participate.
> 
> I would suggest to leave it open till the end of next week and then, 
> if nobody objects, we can proceed to the change.
> 
> What do you think?
> 
> Kostas
> 
>> On Sep 28, 2016, at 3:21 PM, Maximilian Michels  wrote:
>>
>> What are the use cases where you actually need to delete a timer? How
>> about we only let users delete timers which they created themselves?
>>
>> I guessing most of these use cases will be obsolete with the new
>> Trigger DSL because the trigger logic can be expressed more easily. So
>> +1 for removing the delete methods from the context.
>>
>> On Tue, Sep 27, 2016 at 3:43 PM, Kostas Kloudas
>>  wrote:
>>> Hi all,
>>>
>>> As the title of this email suggests, I am proposing to remove the  methods
>>> deleteProcessingTimeTimer(long time) and deleteEventTimeTimer(long time)
>>> from the WindowOperator.Context. With this change, registered timers that
>>> have nothing to do (e.g. because their state has already been cleaned up)
>>> will be simply ignored by the windowOperator, when their time comes.
>>>
>>> The reason for the change is that by allowing custom user code, e.g. a 
>>> custom Trigger,
>>> to delete timers we may have unpredictable behavior.
>>>
>>> As an example, one can imagine the case where we have allowed_lateness = 0 
>>> and the cleanup
>>> timer for a window collides with the end_of_window one. In this case, by 
>>> deleting the end_of_window
>>> timer from the trigger (possibly a custom one), we end up also deleting the 
>>> cleanup one,
>>> which in turn can lead to the window state never being garbage collected.
>>>
>>> To see what can be the consequences apart from memory leaks, this can 
>>> easily lead
>>> to wrong session windows, as a session that should have been garbage 
>>> collected, will
>>> still be around and ready to accept new data.
>>>
>>> With this change, timers that should correctly be deleted will now remain 
>>> in the queue of
>>> pending timers, but they will do nothing, while cleanup timers will cleanup 
>>> the state of their
>>> corresponding window.
>>>
>>> Other possible solutions like keeping a separate list for cleanup timers 
>>> would complicate
>>> the codebase and also introduce memory overheads which can be avoided using 
>>> the
>>> solution above (i.e. just ignoring timers the have nothing to do anymore).
>>>
>>> What do you think?
>>>
>>> Kostas
>>>
> 
> 

-- 
Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082



signature.asc
Description: OpenPGP digital signature

Re: S3/S3A support

2016-10-12 Thread Cliff Resnick 
Regarding S3 and the Rolling/BucketingSink, we've seen data loss when
resuming from checkpoints, as S3 FileSystem implementations flush to
temporary files while the RollingSink expects a direct flush to in-progress
files. Because there is no such think as "flush and resume writing" to S3,
I don't know if RollingSink can be workable in a pure S3 environment. We
worked around it by using HDFS in a transient way.

On Tue, Oct 11, 2016 at 12:01 PM, Stephan Ewen  wrote:

> Hi!
>
> The "truncate()" functionality is only needed for the rolling/bucketing
> sink. The core checkpoint functionality does not need any truncate()
> behavior...
>
> Best,
> Stephan
>
>
> On Tue, Oct 11, 2016 at 5:22 PM, Vijay Srinivasaraghavan <
> vijikar...@yahoo.com.invalid> wrote:
>
> > Thanks Stephan. My understanding is checkpoint uses truncate API but S3A
> > does not support it. Will this have any impact?
> > Some of the known S3A client limitations are captured in Hortonworks site
> > https://hortonworks.github.io/hdp-aws/s3-s3aclient/index.html and
> > wondering if that has any impact on Flink deployment using S3?
> > RegardsVijay
> >
> >
> >
> > On Tuesday, October 11, 2016 1:46 AM, Stephan Ewen  >
> > wrote:
> >
> >
> >  Hi!
> > In 1.2-SNAPSHOT, we recently fixed issues due to the "eventual
> > consistency" nature of S3. The fix is not in v1.1 - that is the only
> known
> > issue I can think of.
> > It results in occasional (seldom) periods of heavy restart retries, until
> > all files are visible to all participants.
> > If you run into that issue, may be worthwhile to look at Flink
> > 1.2-SNAPSHOT.
> > Best,
> > Stephan
> >
> > On Tue, Oct 11, 2016 at 12:13 AM, Vijay Srinivasaraghavan
> >  wrote:
> >
> > Hello,
> > Per documentation (https://ci.apache.org/ projects/flink/flink-docs-
> > master/setup/aws.html), it looks like S3/S3A FS implementation is
> supported
> > using standard Hadoop S3 FS client APIs.
> > In the absence of using standard HCFS and going with S3/S3A,
> > 1) Are there any known limitations/issues?
> > 2) Does checkpoint/savepoint works properly?
> > Regards
> > Vijay
> >
> >
> >
> >
> >
>

Re: currentLowWatermark metric not reported for all tasks?

2016-10-12 Thread dan bress 
Thank you Robert!

On Wed, Oct 12, 2016 at 2:55 AM Robert Metzger  wrote:

> I added a JIRA for this feature request:
> https://issues.apache.org/jira/browse/FLINK-4812
>
> On Fri, Sep 30, 2016 at 6:13 PM, dan bress  wrote:
>
> > Awesome!  It would definitely help me troubleshoot lagging watermarks if
> i
> > can see what watermark all my sources have seen.  Thanks for looking into
> > this!
> >
> > Dan
> >
> > On Fri, Sep 30, 2016 at 5:48 AM Stephan Ewen  wrote:
> >
> > > I think what you describe, Chesnay, is exactly what we should do...
> > >
> > > On Fri, Sep 30, 2016 at 1:15 PM, Chesnay Schepler 
> > > wrote:
> > >
> > > > Hello Dan,
> > > >
> > > > the technical reason is that this metric is only collected in the
> > > > *InputProcessor classes, which aren't used for source tasks.
> > > >
> > > > I do recall that there were discussions about source watermarks, but
> > > > frankly i don't remember why we didn't add them.
> > > >
> > > > In order to add them one would only have to modify the SourceContext
> > > > classes to a) store the last emitted watermark and b) expose it
> > through a
> > > > metric.
> > > >
> > > > Regards,
> > > > Chesnay
> > > >
> > > >
> > > > On 29.09.2016 23:03, dan bress wrote:
> > > >
> > > >> I am trying to measure the currentLowWatermark throughout my
> dataflow,
> > > but
> > > >> I am not seeing it for tasks with sources.  For those tasks I see
> > these
> > > >> metrics:
> > > >>
> > > >> lastCheckpointSize
> > > >> numBytesInLocal
> > > >> numBytesInRemote
> > > >> numBytesOut
> > > >>
> > > >> why am I not seeing currentLowWatermark on these tasks?
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Dan
> > > >>
> > > >>
> > > >
> > >
> >
>

Re: 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support

2016-10-12 Thread Stephan Ewen 
+1 to add the source to Bahir

That is an easier way to iterate fast on this and release quickly

On Wed, Oct 12, 2016 at 10:38 AM, Robert Metzger 
wrote:

> Just a quick update on this one: The bahir community started already
> discussing the first bahir-flink release. I expect it to happen soon.
>
> I would really like to see the netty source in Bahir.
>
> On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen  wrote:
>
> > The Bahir-Flink stuff is fairly new - the first release has not yet
> > happened.
> > Robert is pushing that, will probably happen once he is back.
> >
> > On Tue, Sep 27, 2016 at 3:31 PM, shijinkui  wrote:
> >
> > > It's nice. Will present flink source connector be pushed to
> bahir-flink?
> > > I can add netty-source to bahir-flink.
> > >
> > > Maven repository have no bahir-flink's.
> > > https://mvnrepository.com/artifact/org.apache.bahir
> > >
> > > -邮件原件-
> > > 发件人: Greg Hogan [mailto:c...@greghogan.com]
> > > 发送时间: 2016年9月27日 20:58
> > > 收件人: dev@flink.apache.org
> > > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
> > >
> > > Apache Bahir's website only suggests support for additional frameworks,
> > > but there is a Flink repository at
> > >   https://github.com/apache/bahir-flink
> > >
> > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui 
> wrote:
> > >
> > > > Hey, Stephan Ewen
> > > >
> > > > 1.  bahir's target is spark. The contributer are rxin, srowen,
> > tdas,
> > > > mateiz and so on.
> > > > If we want bahir used by flink, we can suggest bahir provide
> > > > streaming connecter interface, such as store(), start(), stop(),
> > > > restart(), receiving(Any)...
> > > > Then same streaming connector can be implemented by spark and
> > > > flink. But I think this is impossible, as bahir depend
> spark-streaming
> > > > and spark sql.
> > > > 2.  About connector fault tolerance. Bahir's mqtt and akka
> > connector
> > > > are themselves' storage. But netty have no persist data feature.
> > > > I think we can append data to a ringbuffer. When
> SourceContext
> > > > collect() throw error, then write message to inform client to stop
> > > > send message. When flink SourceContext is normal, then write the
> > > > ringbuffer data to flink, inform client to go on.
> > > > Because pushing mode is hard to control the flow throughput,
> > > > the upstream client can.
> > > > This netty connector's purpose is end-to-end streaming,
> > > > minimum time delay.
> > > > 3.  Later on, We can provide http protocol ability, put, post,
> get,
> > > > even websocket or jersey restful based on netty.
> > > >
> > > > -邮件原件-
> > > > 发件人: Stephan Ewen [mailto:se...@apache.org]
> > > > 发送时间: 2016年9月27日 16:54
> > > > 收件人: dev@flink.apache.org
> > > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
> > > >
> > > > I think that could be an interesting source. Two quick questions to
> > > > move forward
> > > >
> > > >   - To keep the Flink code base from becoming too big (hard to
> > > > maintain and
> > > > test) we started working with Apache Bahir as a project dedicated to
> > > > streaming connectors. Would that be a good target for the connector?
> > > >
> > > >   - What are your thoughts on fault tolerance for that connector?
> > > >
> > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui 
> > wrote:
> > > >
> > > > > Hi, all
> > > > >
> > > > > 1.In order to support end-to-end pushed source, I create
> > > FLINK-4630<
> > > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > > > > whether is this idea worth?
> > > > >
> > > > > ---
> > > > > When source stream get start, listen a provided tcp port, receive
> > > > > stream data from user data source.
> > > > > This netty tcp source is keepping alive and end-to-end, that is
> from
> > > > > business system to flink worker directly.
> > > > >
> > > > > user app push ->  netty server source of Flink
> > > > >
> > > > > describe the source in detail below:
> > > > >
> > > > > 1.source run as a netty tcp server
> > > > > 2.user provide a tcp port, if the port is in used, increace the
> > > port
> > > > > number between 1024 to 65535. Source can parallel.
> > > > > 3.callback the provided url to report the real port to listen
> > > > > 4.user push streaming data to netty server, then collect the
> data
> > > to
> > > > > flink
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Jinkui Shi
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Use specific worker for heavy computations

2016-10-12 Thread Stephan Ewen 
So far, we have not introduced location constraints.
The reason is that this goes a bit against the paradigm of location
transparency, which is necessary for failover, dynamically adjusting
parallelism (which is a feature being worked on), etc.

On Wed, Oct 12, 2016 at 10:35 AM, Robert Metzger 
wrote:

> Hi Mariano,
>
> currently, there is nothing available in Flink to execute an operation on a
> specific machine.
>
> Regards,
> Robert
>
>
> On Wed, Sep 28, 2016 at 9:40 PM, Mariano Gonzalez <
> mariano.gonza...@uptake.com> wrote:
>
> > I need to load a PFA (portable format for analytics) that can be around
> 30
> > GB and later process it with hadrian which is the java implementation for
> > PFA's (https://github.com/opendatagroup/hadrian).
> >
> > I would like to execute this transformation step inside a specific worker
> > of the cluster (since I don't want to load 30 GB on every single worker
> > node). Unfortunately, hadrian cannot be executed in a distributed way.
> >
> > So my question would be if there is a way to do some routing with Flink
> and
> > execute this particular transformation step using always the same worker
> > node?
> >
> > Perhaps my approach is completely wrong, so if anybody has any
> suggestions
> > I would be more than happy to hear them:)
> >
> > Thanks
> >
>

Re: [VOTE] Release Apache Flink 1.1.3 (RC2)

2016-10-12 Thread Stephan Ewen 
+1 (binding)

 - Verified that no changed to LICENSE or NOTICE necessary since the last
release
 - mvn clean verily for Scala 2.11, Hadoop 2.6.3 including YARN tests


On Wed, Oct 12, 2016 at 3:22 PM, Fabian Hueske  wrote:

> +1 to release (binding)
>
> - checked hashes and signatures
> - checked diffs against 1.1.2: no dependencies added or modified
> - successfully built Flink from source archive
>   - mvn clean install (Scala 2.10)
>
> Cheers, Fabian
>
> 2016-10-12 14:05 GMT+02:00 Maximilian Michels :
>
> > +1 (binding)
> >
> > - scanned commit history for changes
> > - ran "mvn clean install -Dhadoop.version=2.6.0 -Pinclude-yarn-tests"
> > successfully
> > - started cluster via "./bin/start-cluster.sh"
> > - run batch and streaming examples via web interface and CLI
> > - used web interface for monitoring
> > - ran example job with quickstart project and staging repository
> > (cleaned .m2 cache for 1.1.3 before doing that)
> > - various tests with ContinuousFileMonitoringFunction and verified the
> > fix from the previous RC1
> >
> >
> > -Max
> >
> >
> > On Tue, Oct 11, 2016 at 4:40 PM, Robert Metzger 
> > wrote:
> > > +1 for releasing this as Flink 1.1.3
> > >
> > > - Checked the staging repository for hadoop2 / hadoop1 mixup;
> quickstart
> > > version; build a test project against repository
> > > - Checked the artifacts:
> > >- src doesn't contain any binaries
> > >- started Flink locally & executed example & checked web interface
> > >
> > >
> > >
> > > On Mon, Oct 10, 2016 at 6:52 PM, Ufuk Celebi  wrote:
> > >
> > >> Dear Flink community,
> > >>
> > >> Please vote on releasing the following candidate as Apache Flink
> version
> > >> 1.1.3.
> > >>
> > >> The commit to be voted on:
> > >> 8e8d454 (http://git-wip-us.apache.org/repos/asf/flink/commit/8e8d454)
> > >>
> > >> Branch:
> > >> release-1.1.3-rc2
> > >> (https://git1-us-west.apache.org/repos/asf/flink/repo?p=flin
> > >> k.git;a=shortlog;h=refs/heads/release-1.1.3-rc2)
> > >>
> > >> The release artifacts to be voted on can be found at:
> > >> http://people.apache.org/~uce/flink-1.1.3-rc2/
> > >>
> > >> The release artifacts are signed with the key with fingerprint
> 9D403309:
> > >> http://www.apache.org/dist/flink/KEYS
> > >>
> > >> The staging repository for this release can be found at:
> > >> https://repository.apache.org/content/repositories/orgapache
> flink-1106
> > >>
> > >> -
> > >>
> > >> RC2 adds two new commits since RC1. If there are no objections, I
> > >> would like to reduce the voting time to (at least) 2 days. The vote
> > >> passes if a majority of at least three +1 PMC votes are cast.
> > >>
> > >> The vote ends on Wed, October 12th, 2016.
> > >>
> > >> [ ] +1 Release this package as Apache Flink 1.1.3
> > >> [ ] -1 Do not release this package, because ...
> > >>
> >
>

Re: [DISCUSS] Removing delete*Timer from the WindowOperator.Context

2016-10-12 Thread Kostas Kloudas 
Hi all,

This thread has been dormant for some time now.

Given that this change may affect user code, I am sending this as a 
reminder that the discussion is still open and to re-invite anyone who
may be affected to participate.

I would suggest to leave it open till the end of next week and then, 
if nobody objects, we can proceed to the change.

What do you think?

Kostas

> On Sep 28, 2016, at 3:21 PM, Maximilian Michels  wrote:
> 
> What are the use cases where you actually need to delete a timer? How
> about we only let users delete timers which they created themselves?
> 
> I guessing most of these use cases will be obsolete with the new
> Trigger DSL because the trigger logic can be expressed more easily. So
> +1 for removing the delete methods from the context.
> 
> On Tue, Sep 27, 2016 at 3:43 PM, Kostas Kloudas
>  wrote:
>> Hi all,
>> 
>> As the title of this email suggests, I am proposing to remove the  methods
>> deleteProcessingTimeTimer(long time) and deleteEventTimeTimer(long time)
>> from the WindowOperator.Context. With this change, registered timers that
>> have nothing to do (e.g. because their state has already been cleaned up)
>> will be simply ignored by the windowOperator, when their time comes.
>> 
>> The reason for the change is that by allowing custom user code, e.g. a 
>> custom Trigger,
>> to delete timers we may have unpredictable behavior.
>> 
>> As an example, one can imagine the case where we have allowed_lateness = 0 
>> and the cleanup
>> timer for a window collides with the end_of_window one. In this case, by 
>> deleting the end_of_window
>> timer from the trigger (possibly a custom one), we end up also deleting the 
>> cleanup one,
>> which in turn can lead to the window state never being garbage collected.
>> 
>> To see what can be the consequences apart from memory leaks, this can easily 
>> lead
>> to wrong session windows, as a session that should have been garbage 
>> collected, will
>> still be around and ready to accept new data.
>> 
>> With this change, timers that should correctly be deleted will now remain in 
>> the queue of
>> pending timers, but they will do nothing, while cleanup timers will cleanup 
>> the state of their
>> corresponding window.
>> 
>> Other possible solutions like keeping a separate list for cleanup timers 
>> would complicate
>> the codebase and also introduce memory overheads which can be avoided using 
>> the
>> solution above (i.e. just ignoring timers the have nothing to do anymore).
>> 
>> What do you think?
>> 
>> Kostas
>>

Re: [VOTE] Release Apache Flink 1.1.3 (RC2)

2016-10-12 Thread Fabian Hueske 
+1 to release (binding)

- checked hashes and signatures
- checked diffs against 1.1.2: no dependencies added or modified
- successfully built Flink from source archive
  - mvn clean install (Scala 2.10)

Cheers, Fabian

2016-10-12 14:05 GMT+02:00 Maximilian Michels :

> +1 (binding)
>
> - scanned commit history for changes
> - ran "mvn clean install -Dhadoop.version=2.6.0 -Pinclude-yarn-tests"
> successfully
> - started cluster via "./bin/start-cluster.sh"
> - run batch and streaming examples via web interface and CLI
> - used web interface for monitoring
> - ran example job with quickstart project and staging repository
> (cleaned .m2 cache for 1.1.3 before doing that)
> - various tests with ContinuousFileMonitoringFunction and verified the
> fix from the previous RC1
>
>
> -Max
>
>
> On Tue, Oct 11, 2016 at 4:40 PM, Robert Metzger 
> wrote:
> > +1 for releasing this as Flink 1.1.3
> >
> > - Checked the staging repository for hadoop2 / hadoop1 mixup; quickstart
> > version; build a test project against repository
> > - Checked the artifacts:
> >- src doesn't contain any binaries
> >- started Flink locally & executed example & checked web interface
> >
> >
> >
> > On Mon, Oct 10, 2016 at 6:52 PM, Ufuk Celebi  wrote:
> >
> >> Dear Flink community,
> >>
> >> Please vote on releasing the following candidate as Apache Flink version
> >> 1.1.3.
> >>
> >> The commit to be voted on:
> >> 8e8d454 (http://git-wip-us.apache.org/repos/asf/flink/commit/8e8d454)
> >>
> >> Branch:
> >> release-1.1.3-rc2
> >> (https://git1-us-west.apache.org/repos/asf/flink/repo?p=flin
> >> k.git;a=shortlog;h=refs/heads/release-1.1.3-rc2)
> >>
> >> The release artifacts to be voted on can be found at:
> >> http://people.apache.org/~uce/flink-1.1.3-rc2/
> >>
> >> The release artifacts are signed with the key with fingerprint 9D403309:
> >> http://www.apache.org/dist/flink/KEYS
> >>
> >> The staging repository for this release can be found at:
> >> https://repository.apache.org/content/repositories/orgapacheflink-1106
> >>
> >> -
> >>
> >> RC2 adds two new commits since RC1. If there are no objections, I
> >> would like to reduce the voting time to (at least) 2 days. The vote
> >> passes if a majority of at least three +1 PMC votes are cast.
> >>
> >> The vote ends on Wed, October 12th, 2016.
> >>
> >> [ ] +1 Release this package as Apache Flink 1.1.3
> >> [ ] -1 Do not release this package, because ...
> >>
>

[jira] [Created] (FLINK-4813) Having flink-test-utils as a dependency outside Flink fails the build

2016-10-12 Thread Robert Metzger (JIRA) 
Robert Metzger created FLINK-4813:
-

 Summary: Having flink-test-utils as a dependency outside Flink 
fails the build
 Key: FLINK-4813
 URL: https://issues.apache.org/jira/browse/FLINK-4813
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.2.0
Reporter: Robert Metzger


The {{flink-test-utils}} depend on {{hadoop-minikdc}}, which has a dependency, 
which is only resolvable, if the {{maven-bundle-plugin}} is loaded.

{{flink-parent}} loads that plugin, so all "internal" dependencies to the test 
utils can resolve the plugin.
Right now, users have to use the maven bundle plugin to use our test utils 
externally.

By making the hadoop minikdc dependency optional, we can probably resolve the 
issues. Then, only users who want to use the security-related tools in the test 
utils need to manually add the hadoop minikdc dependency + the plugin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Support Suspending and Resuming of Flink Jobs

2016-10-12 Thread Till Rohrmann 
Hi Greg,

at the moment the serialization of savepoints costs the same as the
serialization of checkpoints, because they use the same serialization
logic. In fact, with Ufuk's changes [1], a savepoint is a checkpoint with
special properties. However, in the future we will probably have different
serialization formats. The savepoint would use a generalized format which
allows to restore the savepoint with a different state backend. When
drawing a checkpoint, we don't have to do this, because we know that the
state backend won't change. Thus, we could store the checkpoint in a more
compressed format which exploits the characteristics of the respective
state backend. But this is not implemented yet.

[1] https://github.com/apache/flink/pull/2608

Cheers,
Till

On Wed, Oct 12, 2016 at 2:35 PM, Greg Hogan  wrote:

> Sorry, I haven't followed this development, but roughly how much more
> costly is the new serialization for savepoints?
>
> On Wed, Oct 12, 2016 at 5:51 AM, SHI Xiaogang 
> wrote:
>
> > Hi all,
> >
> > Currently, savepoints are exactly the completed checkpoints, and Flink
> > provides commands (save/run) to allow saving and restoring jobs. But in
> the
> > near future, savepoints will be very different from checkpoints because
> > they will have common serialization formats and allow recover from major
> > updates. The saving and restoring based on savepoints will be more
> costly.
> >
> > To provide efficient saving and restoring of jobs, we propose to add two
> > more commands in Flink: SUSPEND and RESUME which are based on
> checkpoints.
> >
> > As the implementation of checkpoints depends on the backends (and many
> > other components in Flink), suspending and resuming may not work if there
> > exist major changes in the job or Flink (e.g., different backends). But
> as
> > the implementation is based on checkpoints instead of savepoints, they
> are
> > supposed to be more efficient.
> >
> > The details of the design can be viewed in the Google Doc: Support
> Resuming
> > and Suspending of Flink Jobs
> >  > NgQWZpDTkSxVjg/edit?usp=sharing>
> > .
> >
> > Look forward to your comments. Any feedback is appreciated. :)
> >
> > Thanks,
> > Xiaogang
> >
>

Re: [DISCUSS] Support Suspending and Resuming of Flink Jobs

2016-10-12 Thread Greg Hogan 
Sorry, I haven't followed this development, but roughly how much more
costly is the new serialization for savepoints?

On Wed, Oct 12, 2016 at 5:51 AM, SHI Xiaogang 
wrote:

> Hi all,
>
> Currently, savepoints are exactly the completed checkpoints, and Flink
> provides commands (save/run) to allow saving and restoring jobs. But in the
> near future, savepoints will be very different from checkpoints because
> they will have common serialization formats and allow recover from major
> updates. The saving and restoring based on savepoints will be more costly.
>
> To provide efficient saving and restoring of jobs, we propose to add two
> more commands in Flink: SUSPEND and RESUME which are based on checkpoints.
>
> As the implementation of checkpoints depends on the backends (and many
> other components in Flink), suspending and resuming may not work if there
> exist major changes in the job or Flink (e.g., different backends). But as
> the implementation is based on checkpoints instead of savepoints, they are
> supposed to be more efficient.
>
> The details of the design can be viewed in the Google Doc: Support Resuming
> and Suspending of Flink Jobs
>  NgQWZpDTkSxVjg/edit?usp=sharing>
> .
>
> Look forward to your comments. Any feedback is appreciated. :)
>
> Thanks,
> Xiaogang
>

Re: Assign issue

2016-10-12 Thread Fabian Hueske 
Hi Alexander,

Welcome to the Flink community.

I gave you contributor permissions for JIRA (you can assign issues to
yourself) and assigned the issue to you.
Looking forward to your contribution.

Best, Fabian

2016-10-12 13:14 GMT+02:00 Alexander Shoshin :

> Hi.
>
> I want to do something for flink. I would like to try to resolve this
> issue https://issues.apache.org/jira/browse/FLINK-4283, cause I
> reproduced it in my local repo. Could you assign it to me in JIRA.
>
> Regards, Alexander Shoshin
>
>

Assign issue

2016-10-12 Thread Alexander Shoshin 
Hi.

I want to do something for flink. I would like to try to resolve this issue 
https://issues.apache.org/jira/browse/FLINK-4283, cause I reproduced it in my 
local repo. Could you assign it to me in JIRA.

Regards, Alexander Shoshin

[jira] [Created] (FLINK-4812) Export currentLowWatermark metric also for sources

2016-10-12 Thread Robert Metzger (JIRA) 
Robert Metzger created FLINK-4812:
-

 Summary: Export currentLowWatermark metric also for sources
 Key: FLINK-4812
 URL: https://issues.apache.org/jira/browse/FLINK-4812
 Project: Flink
  Issue Type: Improvement
  Components: Metrics
Reporter: Robert Metzger


As reported by a user, Flink does currently not export the current low 
watermark for sources 
(http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html).

This JIRA is for adding such a metric for the sources as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] Support Suspending and Resuming of Flink Jobs

2016-10-12 Thread SHI Xiaogang 
Hi all,

Currently, savepoints are exactly the completed checkpoints, and Flink
provides commands (save/run) to allow saving and restoring jobs. But in the
near future, savepoints will be very different from checkpoints because
they will have common serialization formats and allow recover from major
updates. The saving and restoring based on savepoints will be more costly.

To provide efficient saving and restoring of jobs, we propose to add two
more commands in Flink: SUSPEND and RESUME which are based on checkpoints.

As the implementation of checkpoints depends on the backends (and many
other components in Flink), suspending and resuming may not work if there
exist major changes in the job or Flink (e.g., different backends). But as
the implementation is based on checkpoints instead of savepoints, they are
supposed to be more efficient.

The details of the design can be viewed in the Google Doc: Support Resuming
and Suspending of Flink Jobs

.

Look forward to your comments. Any feedback is appreciated. :)

Thanks,
Xiaogang

[jira] [Created] (FLINK-4811) Checkpoint Overview should list failed checkpoints

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4811:
---

 Summary: Checkpoint Overview should list failed checkpoints
 Key: FLINK-4811
 URL: https://issues.apache.org/jira/browse/FLINK-4811
 Project: Flink
  Issue Type: Sub-task
  Components: Web Client
Reporter: Stephan Ewen


To let users understand what is happening with failed/skipped checkpoints, the 
web UI should display failed checkpoints as well, with their error cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4810) Checkpoint Coordinator should fail ExecutionGraph after "n" unsuccessful checkpoints

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4810:
---

 Summary: Checkpoint Coordinator should fail ExecutionGraph after 
"n" unsuccessful checkpoints
 Key: FLINK-4810
 URL: https://issues.apache.org/jira/browse/FLINK-4810
 Project: Flink
  Issue Type: Sub-task
  Components: State Backends, Checkpointing
Reporter: Stephan Ewen
 Fix For: 1.2.0


The Checkpoint coordinator should track the number of consecutive unsuccessful 
checkpoints.

If more than {{n}} (configured value) checkpoints fail in a row, it should call 
{{fail()}} on the execution graph to trigger a recovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4809) Operators should tolerate checkpoint failures

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4809:
---

 Summary: Operators should tolerate checkpoint failures
 Key: FLINK-4809
 URL: https://issues.apache.org/jira/browse/FLINK-4809
 Project: Flink
  Issue Type: Sub-task
  Components: State Backends, Checkpointing
Reporter: Stephan Ewen
 Fix For: 1.2.0


Operators should try/catch exceptions in the synchronous and asynchronous part 
of the checkpoint and send a {{DeclineCheckpoint}} message as a result.

The decline message should have the failure cause attached to it.

The checkpoint barrier should be sent anyways as a first step before attempting 
to make a state checkpoint, to make sure that downstream operators do not block 
in alignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4808) Allow skipping failed checkpoints

2016-10-12 Thread Stephan Ewen (JIRA) 
Stephan Ewen created FLINK-4808:
---

 Summary: Allow skipping failed checkpoints
 Key: FLINK-4808
 URL: https://issues.apache.org/jira/browse/FLINK-4808
 Project: Flink
  Issue Type: New Feature
Affects Versions: 1.1.2, 1.1.3
Reporter: Stephan Ewen
 Fix For: 1.2.0


Currently, if Flink cannot complete a checkpoint, it results in a failure and 
recovery.

To make the impact of less stable storage infrastructure on the performance of 
Flink less severe, Flink should be able to tolerate a certain number of failed 
checkpoints and simply keep executing.

This should be controllable via a parameter, for example:
{code}
env.getCheckpointConfig().setAllowedFailedCheckpoints(3);
{code}

A value of {{-1}} could indicate an infinite number of checkpoint failures 
tolerated by Flink.
The default value should still be {{0}}, to keep compatibility with the 
existing behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support

2016-10-12 Thread Robert Metzger 
Just a quick update on this one: The bahir community started already
discussing the first bahir-flink release. I expect it to happen soon.

I would really like to see the netty source in Bahir.

On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen  wrote:

> The Bahir-Flink stuff is fairly new - the first release has not yet
> happened.
> Robert is pushing that, will probably happen once he is back.
>
> On Tue, Sep 27, 2016 at 3:31 PM, shijinkui  wrote:
>
> > It's nice. Will present flink source connector be pushed to bahir-flink?
> > I can add netty-source to bahir-flink.
> >
> > Maven repository have no bahir-flink's.
> > https://mvnrepository.com/artifact/org.apache.bahir
> >
> > -邮件原件-
> > 发件人: Greg Hogan [mailto:c...@greghogan.com]
> > 发送时间: 2016年9月27日 20:58
> > 收件人: dev@flink.apache.org
> > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
> >
> > Apache Bahir's website only suggests support for additional frameworks,
> > but there is a Flink repository at
> >   https://github.com/apache/bahir-flink
> >
> > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui  wrote:
> >
> > > Hey, Stephan Ewen
> > >
> > > 1.  bahir's target is spark. The contributer are rxin, srowen,
> tdas,
> > > mateiz and so on.
> > > If we want bahir used by flink, we can suggest bahir provide
> > > streaming connecter interface, such as store(), start(), stop(),
> > > restart(), receiving(Any)...
> > > Then same streaming connector can be implemented by spark and
> > > flink. But I think this is impossible, as bahir depend spark-streaming
> > > and spark sql.
> > > 2.  About connector fault tolerance. Bahir's mqtt and akka
> connector
> > > are themselves' storage. But netty have no persist data feature.
> > > I think we can append data to a ringbuffer. When SourceContext
> > > collect() throw error, then write message to inform client to stop
> > > send message. When flink SourceContext is normal, then write the
> > > ringbuffer data to flink, inform client to go on.
> > > Because pushing mode is hard to control the flow throughput,
> > > the upstream client can.
> > > This netty connector's purpose is end-to-end streaming,
> > > minimum time delay.
> > > 3.  Later on, We can provide http protocol ability, put, post, get,
> > > even websocket or jersey restful based on netty.
> > >
> > > -邮件原件-
> > > 发件人: Stephan Ewen [mailto:se...@apache.org]
> > > 发送时间: 2016年9月27日 16:54
> > > 收件人: dev@flink.apache.org
> > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
> > >
> > > I think that could be an interesting source. Two quick questions to
> > > move forward
> > >
> > >   - To keep the Flink code base from becoming too big (hard to
> > > maintain and
> > > test) we started working with Apache Bahir as a project dedicated to
> > > streaming connectors. Would that be a good target for the connector?
> > >
> > >   - What are your thoughts on fault tolerance for that connector?
> > >
> > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui 
> wrote:
> > >
> > > > Hi, all
> > > >
> > > > 1.In order to support end-to-end pushed source, I create
> > FLINK-4630<
> > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > > > whether is this idea worth?
> > > >
> > > > ---
> > > > When source stream get start, listen a provided tcp port, receive
> > > > stream data from user data source.
> > > > This netty tcp source is keepping alive and end-to-end, that is from
> > > > business system to flink worker directly.
> > > >
> > > > user app push ->  netty server source of Flink
> > > >
> > > > describe the source in detail below:
> > > >
> > > > 1.source run as a netty tcp server
> > > > 2.user provide a tcp port, if the port is in used, increace the
> > port
> > > > number between 1024 to 65535. Source can parallel.
> > > > 3.callback the provided url to report the real port to listen
> > > > 4.user push streaming data to netty server, then collect the data
> > to
> > > > flink
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Jinkui Shi
> > > >
> > > >
> > >
> >
>

Re: Use specific worker for heavy computations

2016-10-12 Thread Robert Metzger 
Hi Mariano,

currently, there is nothing available in Flink to execute an operation on a
specific machine.

Regards,
Robert


On Wed, Sep 28, 2016 at 9:40 PM, Mariano Gonzalez <
mariano.gonza...@uptake.com> wrote:

> I need to load a PFA (portable format for analytics) that can be around 30
> GB and later process it with hadrian which is the java implementation for
> PFA's (https://github.com/opendatagroup/hadrian).
>
> I would like to execute this transformation step inside a specific worker
> of the cluster (since I don't want to load 30 GB on every single worker
> node). Unfortunately, hadrian cannot be executed in a distributed way.
>
> So my question would be if there is a way to do some routing with Flink and
> execute this particular transformation step using always the same worker
> node?
>
> Perhaps my approach is completely wrong, so if anybody has any suggestions
> I would be more than happy to hear them:)
>
> Thanks
>