[jira] [Commented] (FLINK-4008) Hello,I am not able to create jar of flink-streaming-connectors ...I am able to create jar of others like twitter,kafka,flume but I am not able to create jar of flink-s

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320019#comment-15320019
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-4008:


Don't build under the kinesis connector only.
Either go to the root of the flink project, or under flink-streaming-connectors 
if you've already built the whole project already before.
Also, currently you'll need to include the `include-kinesis` profile to be able 
to build Kinesis, i.e. `mvn clean package -Pinclude-kinesis`.

> Hello,I am not able to create jar of flink-streaming-connectors ...I am able 
> to create jar of others like twitter,kafka,flume but I am not able to create 
> jar of flink-streaming connectors ?? How can I create this jar ??
> ---
>
> Key: FLINK-4008
> URL: https://issues.apache.org/jira/browse/FLINK-4008
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Akshay Shingote
>
> I filed an issue here https://github.com/apache/flink/pull/2058 ... I want to 
> know how can we create jar of Flink-Streaming-Connectors ??



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4008) Hello,I am not able to create jar of flink-streaming-connectors ...I am able to create jar of others like twitter,kafka,flume but I am not able to create jar of flink-s

[Akshay Shingote (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319984#comment-15319984
 ] 

Akshay Shingote commented on FLINK-4008:


How can I build the snapshot ?? Can you please help it out ?? Thank You

> Hello,I am not able to create jar of flink-streaming-connectors ...I am able 
> to create jar of others like twitter,kafka,flume but I am not able to create 
> jar of flink-streaming connectors ?? How can I create this jar ??
> ---
>
> Key: FLINK-4008
> URL: https://issues.apache.org/jira/browse/FLINK-4008
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Akshay Shingote
>
> I filed an issue here https://github.com/apache/flink/pull/2058 ... I want to 
> know how can we create jar of Flink-Streaming-Connectors ??



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4016) FoldApplyWindowFunction is not properly initialized

[RWenden (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319181#comment-15319181
 ] 

RWenden commented on FLINK-4016:


I already made a commit with message 
  [FLINK-3977] initialize FoldApplyWindowFunction 
properly 
but
 it stays visible in TravisCI under [FLINK-4016] initialize 
FoldApplyWindowFunction properly 

Please consider this as a fix under [FLINK-3977].

Regards, RvdWenden




> FoldApplyWindowFunction is not properly initialized
> ---
>
> Key: FLINK-4016
> URL: https://issues.apache.org/jira/browse/FLINK-4016
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.0.3
>Reporter: RWenden
>Priority: Blocker
>  Labels: easyfix
> Fix For: 1.1.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> FoldApplyWindowFunction's outputtype is not set.
> We're using constructions like (excerpt):
>   .keyBy(0)
>   .countWindow(10, 5)
>   .fold(...)
> Running this stream gives an runtime exception in FoldApplyWindowFunction:
> "No initial value was serialized for the fold window function. Probably the 
> setOutputType method was not called."
> This can be easily fixed in WindowedStream.java by (around line# 449):
> FoldApplyWindowFunction foldApplyWindowFunction = new 
> FoldApplyWindowFunction<>(initialValue, foldFunction, function);
> foldApplyWindowFunction.setOutputType(resultType, 
> input.getExecutionConfig());
> operator = new EvictingWindowOperator<>(windowAssigner,
> 
> windowAssigner.getWindowSerializer(getExecutionEnvironment().getConfig()),
> keySel,
> 
> input.getKeyType().createSerializer(getExecutionEnvironment().getConfig()),
> stateDesc,
> new 
> InternalIterableWindowFunction<>(foldApplyWindowFunction),
> trigger,
> evictor);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4003) Use intrinsics for MathUtils logarithms

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-4003.
-
Resolution: Implemented

Implemented in 1bbc1fb4d7b8701c38e6f6a78f3709448181d559

> Use intrinsics for MathUtils logarithms
> ---
>
> Key: FLINK-4003
> URL: https://issues.apache.org/jira/browse/FLINK-4003
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.1.0
>
>
> {{MathUtils.log2floor}} and {{MathUtils.log2strict}} use naive loops which 
> have efficient implementations in {{Integer}} that are commonly implemented 
> as intrinsics [0].
> [0]: 
> http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/tip/src/share/vm/classfile/vmSymbols.hpp#l680



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4003) Use intrinsics for MathUtils logarithms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319093#comment-15319093
 ] 

ASF GitHub Bot commented on FLINK-4003:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2059


> Use intrinsics for MathUtils logarithms
> ---
>
> Key: FLINK-4003
> URL: https://issues.apache.org/jira/browse/FLINK-4003
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.1.0
>
>
> {{MathUtils.log2floor}} and {{MathUtils.log2strict}} use naive loops which 
> have efficient implementations in {{Integer}} that are commonly implemented 
> as intrinsics [0].
> [0]: 
> http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/tip/src/share/vm/classfile/vmSymbols.hpp#l680



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2059: [FLINK-4003] Use intrinsics for MathUtils logarith...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2059


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2832) Failing test: RandomSamplerTest.testReservoirSamplerWithReplacement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319071#comment-15319071
 ] 

ASF GitHub Bot commented on FLINK-2832:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2076
  
Merging this, addressing @chiwanpark 's comments...


> Failing test: RandomSamplerTest.testReservoirSamplerWithReplacement
> ---
>
> Key: FLINK-2832
> URL: https://issues.apache.org/jira/browse/FLINK-2832
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.10.0
>Reporter: Vasia Kalavri
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.0.0
>
>
> Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 19.133 sec 
> <<< FAILURE! - in org.apache.flink.api.java.sampling.RandomSamplerTest
> testReservoirSamplerWithReplacement(org.apache.flink.api.java.sampling.RandomSamplerTest)
>   Time elapsed: 2.534 sec  <<< FAILURE!
> java.lang.AssertionError: KS test result with p value(0.11), d 
> value(0.103090)
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> org.apache.flink.api.java.sampling.RandomSamplerTest.verifyKSTest(RandomSamplerTest.java:342)
>   at 
> org.apache.flink.api.java.sampling.RandomSamplerTest.verifyRandomSamplerWithSampleSize(RandomSamplerTest.java:330)
>   at 
> org.apache.flink.api.java.sampling.RandomSamplerTest.verifyReservoirSamplerWithReplacement(RandomSamplerTest.java:289)
>   at 
> org.apache.flink.api.java.sampling.RandomSamplerTest.testReservoirSamplerWithReplacement(RandomSamplerTest.java:192)
> Results :
> Failed tests: 
>   
> RandomSamplerTest.testReservoirSamplerWithReplacement:192->verifyReservoirSamplerWithReplacement:289->verifyRandomSamplerWithSampleSize:330->verifyKSTest:342
>  KS test result with p value(0.11), d value(0.103090)
> Full log [here|https://travis-ci.org/apache/flink/jobs/84120131].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2076: [FLINK-2832] [tests] Hardens RandomSamplerTest

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2076
  
Merging this, addressing @chiwanpark 's comments...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2080: [FLINK-3530] Fix Kafka08 instability: Avoid restarts from...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2080
  
Do you know where the canceling looses the time?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4002) [py] Improve testing infraestructure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319051#comment-15319051
 ] 

ASF GitHub Bot commented on FLINK-4002:
---

Github user omaralvarez commented on the issue:

https://github.com/apache/flink/pull/2063
  
I think everything should be ready now. I was not able to pinpoint why HDFS 
worked, I assume `distributeFiles()` copied the complete directory and that is 
why it worked, but I'm not 100% sure.


> [py] Improve testing infraestructure
> 
>
> Key: FLINK-4002
> URL: https://issues.apache.org/jira/browse/FLINK-4002
> Project: Flink
>  Issue Type: Bug
>  Components: Python API
>Affects Versions: 1.0.3
>Reporter: Omar Alvarez
>Priority: Minor
>  Labels: Python, Testing
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The Verify() test function errors out when array elements are missing:
> {code}
> env.generate_sequence(1, 5)\
>  .map(Id()).map_partition(Verify([1,2,3,4], "Sequence")).output()
> {code}
> {quote}
> IndexError: list index out of range
> {quote}
> There should also be more documentation in test functions.
> I am already working on a pull request to fix this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2063: [FLINK-4002] [py] Improve testing infraestructure

[omaralvarez]

Github user omaralvarez commented on the issue:

https://github.com/apache/flink/pull/2063
  
I think everything should be ready now. I was not able to pinpoint why HDFS 
worked, I assume `distributeFiles()` copied the complete directory and that is 
why it worked, but I'm not 100% sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-3906) Global Clustering Coefficient

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-3906.
-
Resolution: Implemented

Implemented in 8da1a75ceb30ef1bce27a2426dab3a0f66b94b64

> Global Clustering Coefficient
> -
>
> Key: FLINK-3906
> URL: https://issues.apache.org/jira/browse/FLINK-3906
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.1.0
>
>
> The global clustering coefficient measures the ratio of triplets in a graph 
> which are closed (form a triangle). Scores range from 0.0 (no triangles) to 
> 1.0 (complete graph).
> As part of this ticket the inefficient `GSATriangleCount` will be removed and 
> a new analytic will be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1962: [FLINK-3857][Streaming Connectors]Add reconnect attempt t...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/1962
  
I took a quick look at this. I am wondering if this actually needs an extra 
timer service for retries.

Can this be solved without a timer? The failures could be detected in the 
`invoke(...)` method, and the retry done directly there (with some minimal 
backoff or so). 

Triggering asynchronous timers is very complex and easily creates leaks, 
races, or leftover work / tasks at shutdown.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3857) Add reconnect attempt to Elasticsearch host

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318993#comment-15318993
 ] 

ASF GitHub Bot commented on FLINK-3857:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/1962
  
I took a quick look at this. I am wondering if this actually needs an extra 
timer service for retries.

Can this be solved without a timer? The failures could be detected in the 
`invoke(...)` method, and the retry done directly there (with some minimal 
backoff or so). 

Triggering asynchronous timers is very complex and easily creates leaks, 
races, or leftover work / tasks at shutdown.


> Add reconnect attempt to Elasticsearch host
> ---
>
> Key: FLINK-3857
> URL: https://issues.apache.org/jira/browse/FLINK-3857
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Affects Versions: 1.1.0, 1.0.2
>Reporter: Fabian Hueske
>Assignee: Subhobrata Dey
>
> Currently, the connection to the Elasticsearch host is opened in 
> {{ElasticsearchSink.open()}}. In case the connection is lost (maybe due to a 
> changed DNS entry), the sink fails.
> I propose to catch the Exception for lost connections in the {{invoke()}} 
> method and try to re-open the connection for a configurable number of times 
> with a certain delay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3906) Global Clustering Coefficient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318992#comment-15318992
 ] 

ASF GitHub Bot commented on FLINK-3906:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1997


> Global Clustering Coefficient
> -
>
> Key: FLINK-3906
> URL: https://issues.apache.org/jira/browse/FLINK-3906
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.1.0
>
>
> The global clustering coefficient measures the ratio of triplets in a graph 
> which are closed (form a triangle). Scores range from 0.0 (no triangles) to 
> 1.0 (complete graph).
> As part of this ticket the inefficient `GSATriangleCount` will be removed and 
> a new analytic will be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #1997: [FLINK-3906] [gelly] Global Clustering Coefficient

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1997


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4025) Add possiblity for the RMQ Streaming Source to customize the queue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318977#comment-15318977
 ] 

ASF GitHub Bot commented on FLINK-4025:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2073
  
The change looks good, I'll merge it.


> Add possiblity for the RMQ Streaming Source to customize the queue
> --
>
> Key: FLINK-4025
> URL: https://issues.apache.org/jira/browse/FLINK-4025
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Affects Versions: 1.0.2
>Reporter: Dominik Bruhn
>
> This patch adds the possibilty for the user of the RabbitMQ
> Streaming Connector to customize the queue which is used. There
> are use-cases in which you want to set custom parameters for the
> queue (i.e. TTL of the messages if Flink reboots) or the
> possibility to bind the queue to an exchange afterwards.
> The commit doesn't change the actual behaviour but makes it
> possible for users to override the newly create `setupQueue`
> method and cutomize their implementation. This was not possible
> before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4025) Add possiblity for the RMQ Streaming Source to customize the queue

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318978#comment-15318978
 ] 

Robert Metzger commented on FLINK-4025:
---

I understand. I'll merge your change first and then the other one.

> Add possiblity for the RMQ Streaming Source to customize the queue
> --
>
> Key: FLINK-4025
> URL: https://issues.apache.org/jira/browse/FLINK-4025
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Affects Versions: 1.0.2
>Reporter: Dominik Bruhn
>
> This patch adds the possibilty for the user of the RabbitMQ
> Streaming Connector to customize the queue which is used. There
> are use-cases in which you want to set custom parameters for the
> queue (i.e. TTL of the messages if Flink reboots) or the
> possibility to bind the queue to an exchange afterwards.
> The commit doesn't change the actual behaviour but makes it
> possible for users to override the newly create `setupQueue`
> method and cutomize their implementation. This was not possible
> before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3869) WindowedStream.apply with FoldFunction is too restrictive

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318972#comment-15318972
 ] 

ASF GitHub Bot commented on FLINK-3869:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/1973
  
I think the API breaking changes preventer will reject this.
It is all public methods, and the signature changes (implicit parameters 
and context bounds are parameters in the bytecode)


> WindowedStream.apply with FoldFunction is too restrictive
> -
>
> Key: FLINK-3869
> URL: https://issues.apache.org/jira/browse/FLINK-3869
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> Right now we have this signature:
> {code}
> public  SingleOutputStreamOperator apply(R initialValue, 
> FoldFunction foldFunction, WindowFunction function) {
> {code}
> but we should have this signature to allow users to return a type other than 
> the fold accumulator type from their window function:
> {code}
> public  SingleOutputStreamOperator apply(ACC initialValue, 
> FoldFunction foldFunction, WindowFunction function) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1973: [FLINK-3869] Relax window fold generic parameters

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/1973
  
I think the API breaking changes preventer will reject this.
It is all public methods, and the signature changes (implicit parameters 
and context bounds are parameters in the bytecode)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3805) BloomFilter initialized with no memory available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318963#comment-15318963
 ] 

ASF GitHub Bot commented on FLINK-3805:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/1999
  
@rekhajoshm Will you be adding a test case for this pull request? If not, I 
think it should be closed, because we cannot merge it without a test.


> BloomFilter initialized with no memory available
> 
>
> Key: FLINK-3805
> URL: https://issues.apache.org/jira/browse/FLINK-3805
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.1.0, 1.0.1
>Reporter: Greg Hogan
>Assignee: Rekha Joshi
>Priority: Critical
>
> I flagged this as 1.1.0 and 1.0.1 without checking the latter.
> Link to build, command, and stacktrace follow.
> {{MutableHashTable.initTable}} is calling {{initBloomFilter}} when 
> {{this.availableMemory.size()==0}}.
> https://s3.amazonaws.com/apache-flink/flink_bloomfilter_crash.tar.bz2
> ./bin/flink run -class org.apache.flink.graph.examples.TriangleListing 
> ~/flink-gelly-examples_2.10-1.1-SNAPSHOT.jar --clip_and_flip false --output 
> print --output print --scale 14 --count
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed.
>   at org.apache.flink.client.program.Client.runBlocking(Client.java:381)
>   at org.apache.flink.client.program.Client.runBlocking(Client.java:355)
>   at org.apache.flink.client.program.Client.runBlocking(Client.java:315)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:61)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:898)
>   at org.apache.flink.api.java.DataSet.collect(DataSet.java:410)
>   at org.apache.flink.api.java.DataSet.print(DataSet.java:1605)
>   at 
> org.apache.flink.graph.examples.TriangleListing.main(TriangleListing.java:106)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
>   at 
> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:860)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:327)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1187)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1238)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:805)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:751)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:751)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.IllegalArgumentException: expectedEntries should be > 0
>   at 
> org.apache.flink.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>   at 
> org.apache.flink.runtime.operators.util.BloomFilter.(BloomFilter.java:53)
>   at 
> org.apache.flink.runtime.operators.hash.MutableHashTable.initBloomFilter(MutableHashTable.java:823)
>   at 
> 

[GitHub] flink issue #1999: [FLINK-3805] BloomFilter initialized with no memory avail...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/1999
  
@rekhajoshm Will you be adding a test case for this pull request? If not, I 
think it should be closed, because we cannot merge it without a test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4008) Hello,I am not able to create jar of flink-streaming-connectors ...I am able to create jar of others like twitter,kafka,flume but I am not able to create jar of flink-s

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318955#comment-15318955
 ] 

Stephan Ewen commented on FLINK-4008:
-

There is no "flink-streaming-connectors" JAR file and you do not need one.
The error message says you need a POM, and that exists. You need to 
build/install the entire snapshot - that will work.

> Hello,I am not able to create jar of flink-streaming-connectors ...I am able 
> to create jar of others like twitter,kafka,flume but I am not able to create 
> jar of flink-streaming connectors ?? How can I create this jar ??
> ---
>
> Key: FLINK-4008
> URL: https://issues.apache.org/jira/browse/FLINK-4008
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Akshay Shingote
>
> I filed an issue here https://github.com/apache/flink/pull/2058 ... I want to 
> know how can we create jar of Flink-Streaming-Connectors ??



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4015) FlinkKafkaProducer08 fails when partition leader changes

[Sebastian Klemke (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318949#comment-15318949
 ] 

Sebastian Klemke commented on FLINK-4015:
-

For our application, it should never drop records. Instead, it should be 
determined which record failed and this record and the following records 
produced to that partition should be repeated in order. This would require 
keeping a list of sent but not acknowledged records per partition in the 
producer. But for other applications, dropping might be suitable.

> FlinkKafkaProducer08 fails when partition leader changes
> 
>
> Key: FLINK-4015
> URL: https://issues.apache.org/jira/browse/FLINK-4015
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.2
>Reporter: Sebastian Klemke
>
> When leader for a partition changes, producer fails with the following 
> exception:
> {code}
> 06:34:50,813 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job b323f5de3d32504651e861d5ecb27e7c (JOB_NAME) changed to 
> FAILING.
> java.lang.RuntimeException: Could not forward element to next operator
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
>   at 
> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>   at OPERATOR.flatMap2(OPERATOR.java:82)
>   at OPERATOR.flatMap2(OPERATOR.java:16)
>   at 
> org.apache.flink.streaming.api.operators.co.CoStreamFlatMap.processElement2(CoStreamFlatMap.java:63)
>   at 
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:207)
>   at 
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.run(TwoInputStreamTask.java:89)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:225)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not forward element to next 
> operator
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
>   at 
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:39)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:351)
>   ... 10 more
> Caused by: java.lang.RuntimeException: Could not forward element to next 
> operator
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
>   at 
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:39)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:351)
>   ... 13 more
> Caused by: java.lang.Exception: Failed to send data to Kafka: This server is 
> not the leader for that topic-partition.
>   at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.checkErroneous(FlinkKafkaProducerBase.java:282)
>   at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.invoke(FlinkKafkaProducerBase.java:249)
>   at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:351)
>   ... 16 more
> Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: 
> This server is not the leader for that topic-partition.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3922) Infinite recursion on TypeExtractor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318947#comment-15318947
 ] 

ASF GitHub Bot commented on FLINK-3922:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2011
  
Will merge this...


> Infinite recursion on TypeExtractor
> ---
>
> Key: FLINK-3922
> URL: https://issues.apache.org/jira/browse/FLINK-3922
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 1.0.2
>Reporter: Flavio Pompermaier
>Assignee: Timo Walther
>Priority: Critical
>
> This program cause a StackOverflow (infinite recursion) in the TypeExtractor:
> {code:title=TypeSerializerStackOverflowOnRecursivePojo.java|borderStyle=solid}
> public class TypeSerializerStackOverflowOnRecursivePojo {
>   public static class RecursivePojo implements Serializable {
>   private static final long serialVersionUID = 1L;
>   
>   private RecursivePojo parent;
>   public RecursivePojo(){}
>   public RecursivePojo(K k, V v) {
>   }
>   public RecursivePojo getParent() {
>   return parent;
>   }
>   public void setParent(RecursivePojo parent) {
>   this.parent = parent;
>   }
>   
>   }
>   public static class TypedTuple extends Tuple3 RecursivePojo>>{
>   private static final long serialVersionUID = 1L;
>   }
>   
>   public static void main(String[] args) throws Exception {
>   ExecutionEnvironment env = 
> ExecutionEnvironment.getExecutionEnvironment();
>   env.fromCollection(Arrays.asList(new RecursivePojo Map>("test",new HashMap(
>   .map(t-> {TypedTuple ret = new TypedTuple();ret.setFields("1", 
> "1", t);return ret;}).returns(TypedTuple.class)
>   .print();
>   }
>   
> }
> {code}
> The thrown Exception is the following:
> {code:title=Exception thrown}
> Exception in thread "main" java.lang.StackOverflowError
>   at 
> sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:328)
>   at 
> sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
>   at 
> sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
>   at 
> sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
>   at 
> sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
>   at 
> sun.reflect.generics.parser.SignatureParser.parseReturnType(SignatureParser.java:627)
>   at 
> sun.reflect.generics.parser.SignatureParser.parseMethodTypeSignature(SignatureParser.java:577)
>   at 
> sun.reflect.generics.parser.SignatureParser.parseMethodSig(SignatureParser.java:171)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.parse(ConstructorRepository.java:55)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.parse(ConstructorRepository.java:43)
>   at 
> sun.reflect.generics.repository.AbstractRepository.(AbstractRepository.java:74)
>   at 
> sun.reflect.generics.repository.GenericDeclRepository.(GenericDeclRepository.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.(ConstructorRepository.java:51)
>   at 
> sun.reflect.generics.repository.MethodRepository.(MethodRepository.java:46)
>   at 
> sun.reflect.generics.repository.MethodRepository.make(MethodRepository.java:59)
>   at java.lang.reflect.Method.getGenericInfo(Method.java:102)
>   at java.lang.reflect.Method.getGenericReturnType(Method.java:255)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.isValidPojoField(TypeExtractor.java:1610)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1671)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1559)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:732)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1678)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1559)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:732)
>   at 
> org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1678)
>   at 
> 

[GitHub] flink issue #2011: [FLINK-3922] [types] Infinite recursion on TypeExtractor

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2011
  
Will merge this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4011) Unable to access completed job in web frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318945#comment-15318945
 ] 

ASF GitHub Bot commented on FLINK-4011:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2065
  
Thank you for the review. I'll merge it


> Unable to access completed job in web frontend
> --
>
> Key: FLINK-4011
> URL: https://issues.apache.org/jira/browse/FLINK-4011
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
>
> In the current master, I'm not able to access a finished job's detail page.
> The JobManager logs shows the following exception:
> {code}
> 2016-06-02 15:23:08,581 WARN  
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandler - Error while 
> handling request
> java.lang.RuntimeException: Couldn't deserialize ExecutionConfig.
> at 
> org.apache.flink.runtime.webmonitor.handlers.JobConfigHandler.handleRequest(JobConfigHandler.java:52)
> at 
> org.apache.flink.runtime.webmonitor.handlers.AbstractExecutionGraphRequestHandler.handleRequest(AbstractExecutionGraphRequestHandler.java:61)
> at 
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandler.respondAsLeader(RuntimeMonitorHandler.java:88)
> at 
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:84)
> at 
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:44)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.handler.codec.http.router.Handler.routed(Handler.java:62)
> at 
> io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:57)
> at 
> io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:20)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at 
> org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:105)
> at 
> org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:65)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at 
> io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at 
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:55)
> at 
> org.apache.flink.runtime.webmonitor.handlers.JobConfigHandler.handleRequest(JobConfigHandler.java:50)
> ... 31 more
> {code}



--

[GitHub] flink issue #2065: [FLINK-4011] Keep UserCodeClassLoader in archived Executi...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2065
  
Thank you for the review. I'll merge it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4015) FlinkKafkaProducer08 fails when partition leader changes

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318920#comment-15318920
 ] 

Stephan Ewen commented on FLINK-4015:
-

What kind of behavior should the sink have?
If a retry fails, should it simply drop the record?

> FlinkKafkaProducer08 fails when partition leader changes
> 
>
> Key: FLINK-4015
> URL: https://issues.apache.org/jira/browse/FLINK-4015
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.2
>Reporter: Sebastian Klemke
>
> When leader for a partition changes, producer fails with the following 
> exception:
> {code}
> 06:34:50,813 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job b323f5de3d32504651e861d5ecb27e7c (JOB_NAME) changed to 
> FAILING.
> java.lang.RuntimeException: Could not forward element to next operator
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
>   at 
> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>   at OPERATOR.flatMap2(OPERATOR.java:82)
>   at OPERATOR.flatMap2(OPERATOR.java:16)
>   at 
> org.apache.flink.streaming.api.operators.co.CoStreamFlatMap.processElement2(CoStreamFlatMap.java:63)
>   at 
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:207)
>   at 
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.run(TwoInputStreamTask.java:89)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:225)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not forward element to next 
> operator
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
>   at 
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:39)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:351)
>   ... 10 more
> Caused by: java.lang.RuntimeException: Could not forward element to next 
> operator
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
>   at 
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:39)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:351)
>   ... 13 more
> Caused by: java.lang.Exception: Failed to send data to Kafka: This server is 
> not the leader for that topic-partition.
>   at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.checkErroneous(FlinkKafkaProducerBase.java:282)
>   at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.invoke(FlinkKafkaProducerBase.java:249)
>   at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:351)
>   ... 16 more
> Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: 
> This server is not the leader for that topic-partition.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3405) Extend NiFiSource with interface StoppableFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318914#comment-15318914
 ] 

ASF GitHub Bot commented on FLINK-3405:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2047
  
All right, merging this, thanks!


> Extend NiFiSource with interface StoppableFunction
> --
>
> Key: FLINK-3405
> URL: https://issues.apache.org/jira/browse/FLINK-3405
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Matthias J. Sax
>Assignee: Suneel Marthi
> Fix For: 1.1.0, 1.0.4
>
>
> Nifi source is not stoppable right now. To make it stoppable, is must 
> implement {{StoppableFunction}}. Implementing method {{stop()}} must ensure, 
> that the source stops receiving new messages from Nifi and issues a final 
> checkpoint. Afterwards, {{run()}} must return.
> When implementing this, keep in mind, that the gathered checkpoint might 
> later be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2047: FLINK-3405: Extend NiFiSource with interface StoppableFun...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2047
  
All right, merging this, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2058: Not able to create flink-streaming-connectors jar

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2058
  
Can you close the pull request and continue the discussion on JIRA?
This here is not the right place for such a discussion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4003) Use intrinsics for MathUtils logarithms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318904#comment-15318904
 ] 

ASF GitHub Bot commented on FLINK-4003:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2059
  
Looks good, +1 to merge this


> Use intrinsics for MathUtils logarithms
> ---
>
> Key: FLINK-4003
> URL: https://issues.apache.org/jira/browse/FLINK-4003
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.1.0
>
>
> {{MathUtils.log2floor}} and {{MathUtils.log2strict}} use naive loops which 
> have efficient implementations in {{Integer}} that are commonly implemented 
> as intrinsics [0].
> [0]: 
> http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/tip/src/share/vm/classfile/vmSymbols.hpp#l680



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2059: [FLINK-4003] Use intrinsics for MathUtils logarithms

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2059
  
Looks good, +1 to merge this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3921) StringParser not specifying encoding to use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318898#comment-15318898
 ] 

ASF GitHub Bot commented on FLINK-3921:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2060
  
Let's make this configurable on `GenericCsvInputFormat`, with the default 
of "UTF-8" (this is what we use in other places as the default, too).

The charset affects the
  - delimiter
  - comments
  - parsers

I think the `FieldParser` should have a `setCharset()` method. That way, we 
need not pass the charset to every method call.

This would also need a test.


> StringParser not specifying encoding to use
> ---
>
> Key: FLINK-3921
> URL: https://issues.apache.org/jira/browse/FLINK-3921
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.3
>Reporter: Tatu Saloranta
>Assignee: Rekha Joshi
>Priority: Trivial
>
> Class `flink.types.parser.StringParser` has javadocs indicating that contents 
> are expected to be Ascii, similar to `StringValueParser`. That makes sense, 
> but when constructing actual instance, no encoding is specified; on line 66 
> f.ex:
>this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents 
> really are always Ascii (would not count on that as parser is used from CSV 
> reader), not a big deal, but it can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be 
> used: javadocs claim ascii, so could be "us-ascii", but could well be UTF-8 
> or even ISO-8859-1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2060: [FLINK-3921] StringParser encoding

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2060
  
Let's make this configurable on `GenericCsvInputFormat`, with the default 
of "UTF-8" (this is what we use in other places as the default, too).

The charset affects the
  - delimiter
  - comments
  - parsers

I think the `FieldParser` should have a `setCharset()` method. That way, we 
need not pass the charset to every method call.

This would also need a test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3967) Provide RethinkDB Sink for Flink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318883#comment-15318883
 ] 

ASF GitHub Bot commented on FLINK-3967:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2031
  
Hi @mans2singh !

This looks like a good piece of work, and ReThinkDB is an interesting 
technology!
An Apache-compatible licence is also good.

We are facing a bit of a maintenance issue right now, though, with a large 
number of new projects. There are so many possible DBs and connectors, and it 
is tough for the community to maintain them all. For that reason, I would 
suggest to stage such contributions of new and interesting connectors.

Could you put this into an standalone GitHub project for now? We would link 
this repository from the Apache Flink website and that way make it visible to 
the Flink community.
When we see that we get frequent requests for a ReThinkDB, we would like to 
move this to the core Flink code base.

Following this idea, we would end up with frequently used connectors in the 
core Flink repository, and a larger number of connectors in third party 
community repositories. I think that would help the stability of the project 
and all connectors.

Stephan


> Provide RethinkDB Sink for Flink
> 
>
> Key: FLINK-3967
> URL: https://issues.apache.org/jira/browse/FLINK-3967
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming, Streaming Connectors
>Affects Versions: 1.0.3
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: features
> Fix For: 1.1.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Provide Sink to stream data from flink to rethink db.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2031: FLINK-3967 - Flink Sink for Rethink Db

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2031
  
Hi @mans2singh !

This looks like a good piece of work, and ReThinkDB is an interesting 
technology!
An Apache-compatible licence is also good.

We are facing a bit of a maintenance issue right now, though, with a large 
number of new projects. There are so many possible DBs and connectors, and it 
is tough for the community to maintain them all. For that reason, I would 
suggest to stage such contributions of new and interesting connectors.

Could you put this into an standalone GitHub project for now? We would link 
this repository from the Apache Flink website and that way make it visible to 
the Flink community.
When we see that we get frequent requests for a ReThinkDB, we would like to 
move this to the core Flink code base.

Following this idea, we would end up with frequently used connectors in the 
core Flink repository, and a larger number of connectors in third party 
community repositories. I think that would help the stability of the project 
and all connectors.

Stephan


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2032
  
I see no problem with merging this, from a code perspective.

I do not completely get all the mechanics and reasoning behind this, so 
would be great if @vasia could comment on this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3965) Delegating GraphAlgorithm

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318821#comment-15318821
 ] 

ASF GitHub Bot commented on FLINK-3965:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2032
  
I see no problem with merging this, from a code perspective.

I do not completely get all the mechanics and reasoning behind this, so 
would be great if @vasia could comment on this.


> Delegating GraphAlgorithm
> -
>
> Key: FLINK-3965
> URL: https://issues.apache.org/jira/browse/FLINK-3965
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.1.0
>
>
> Complex and related algorithms often overlap in computation of data. Two such 
> examples are:
> 1) the local and global clustering coefficients each use a listing of 
> triangles
> 2) the local clustering coefficient joins on vertex degree, and the 
> underlying triangle listing annotates edge degree which uses vertex degree
> We can reuse and rewrite algorithm output by creating a {{ProxyObject}} as a 
> delegate for method calls to the {{DataSet}} returned by the algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3868) Specialized CopyableValue serializers and comparators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318814#comment-15318814
 ] 

ASF GitHub Bot commented on FLINK-3868:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/1983
  
I think the code looks good.

There are no tests so far, though. I am wondering if some would help to 
catch copy/paste errors.
It is very easy to write comparator tests simply extending the 
`ComparatorTestBase`.

For the future: I am wondering if we can "generate" these serializers and 
comparators somehow (and their test classes), like we do for the tuple classes. 
Writing them seems like very repetitive work.


> Specialized CopyableValue serializers and comparators
> -
>
> Key: FLINK-3868
> URL: https://issues.apache.org/jira/browse/FLINK-3868
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.1.0
>
>
> This need was discussed on the mailing list [1] and will be obviated by code 
> generation for POJO serializers and comparators [2] (as I understand, i.e., 
> {{LongValue}} will now simply be treated as a POJO which happens to contain a 
> {{long}} and a specialized serializer and comparator will be generated).
> In the meantime, and since {{ValueTypeInfo}} will need to be reworked to use 
> the new generators, I think it is worthwhile to add specialized serializers 
> and comparators for the {{CopyableValue}} types.
> This will also provide another point of comparison for the performance of the 
> generated serializers and comparators.
> [1] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Tuple-performance-and-the-curious-JIT-compiler-td10666.html
> [2] https://issues.apache.org/jira/browse/FLINK-3599



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3530) Kafka09ITCase.testBigRecordJob fails on Travis

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318798#comment-15318798
 ] 

ASF GitHub Bot commented on FLINK-3530:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/2080

[FLINK-3530] Fix Kafka08 instability: Avoid restarts from SuccessExce…

This pull request is improving the Kafka tests stability.

These tests have one artificial test failure, the code should recover from, 
then, they throw a SuccessException once a stopping condition has been met.
With the number of restarts set to 3, the job was restarting two times due 
to the SuccessException's. Sometimes, the task cancellation takes a lot of 
time, letting the test exceed the timeout of 60 seconds.

This pull request sets the number of restarts to one: There will be one 
artificial test failure, then a success exception, finishing the tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3530

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2080


commit f7033a9e4426a8351a2039bd91db377d8bdc76b0
Author: Robert Metzger 
Date:   2016-06-07T16:20:17Z

[FLINK-3530] Fix Kafka08 instability: Avoid restarts from SuccessException




> Kafka09ITCase.testBigRecordJob fails on Travis
> --
>
> Key: FLINK-3530
> URL: https://issues.apache.org/jira/browse/FLINK-3530
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Robert Metzger
>  Labels: test-stability
>
> The test case {{Kafka09ITCase.testBigRecordJob}} failed on Travis.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/112049279/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2080: [FLINK-3530] Fix Kafka08 instability: Avoid restar...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/2080

[FLINK-3530] Fix Kafka08 instability: Avoid restarts from SuccessExce…

This pull request is improving the Kafka tests stability.

These tests have one artificial test failure, the code should recover from, 
then, they throw a SuccessException once a stopping condition has been met.
With the number of restarts set to 3, the job was restarting two times due 
to the SuccessException's. Sometimes, the task cancellation takes a lot of 
time, letting the test exceed the timeout of 60 seconds.

This pull request sets the number of restarts to one: There will be one 
artificial test failure, then a success exception, finishing the tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3530

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2080


commit f7033a9e4426a8351a2039bd91db377d8bdc76b0
Author: Robert Metzger 
Date:   2016-06-07T16:20:17Z

[FLINK-3530] Fix Kafka08 instability: Avoid restarts from SuccessException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-3907) Directed Clustering Coefficient

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan updated FLINK-3907:
--
Fix Version/s: 1.1.0

> Directed Clustering Coefficient
> ---
>
> Key: FLINK-3907
> URL: https://issues.apache.org/jira/browse/FLINK-3907
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.1.0
>
>
> A directed clustering coefficient algorithm can be implemented using an 
> efficient triangle listing implementation which emits not only the three 
> vertex IDs forming the triangle but also a bitmask indicating which edges 
> form the triangle. A triangle can be formed with a minimum of three or 
> maximum of six directed edges. Directed clustering coefficient can then 
> shatter the triangles and emit a score of either 1 or 2 for each vertex.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3907) Directed Clustering Coefficient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318795#comment-15318795
 ] 

ASF GitHub Bot commented on FLINK-3907:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/2079

[FLINK-3907] [gelly] Directed Clustering Coefficient



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
3907_directed_clustering_coefficient

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2079


commit a4ff0b99d91f607d49803ce213a0bca1cff7e279
Author: Greg Hogan 
Date:   2016-05-25T10:41:12Z

[FLINK-3907] [gelly] Directed Clustering Coefficient




> Directed Clustering Coefficient
> ---
>
> Key: FLINK-3907
> URL: https://issues.apache.org/jira/browse/FLINK-3907
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> A directed clustering coefficient algorithm can be implemented using an 
> efficient triangle listing implementation which emits not only the three 
> vertex IDs forming the triangle but also a bitmask indicating which edges 
> form the triangle. A triangle can be formed with a minimum of three or 
> maximum of six directed edges. Directed clustering coefficient can then 
> shatter the triangles and emit a score of either 1 or 2 for each vertex.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2079: [FLINK-3907] [gelly] Directed Clustering Coefficie...

[greghogan]

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/2079

[FLINK-3907] [gelly] Directed Clustering Coefficient



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
3907_directed_clustering_coefficient

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2079


commit a4ff0b99d91f607d49803ce213a0bca1cff7e279
Author: Greg Hogan 
Date:   2016-05-25T10:41:12Z

[FLINK-3907] [gelly] Directed Clustering Coefficient




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-4028) AbstractAlignedProcessingTimeWindowOperator creates wrong TimeWindow

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen updated FLINK-4028:

Priority: Blocker  (was: Major)

> AbstractAlignedProcessingTimeWindowOperator creates wrong TimeWindow
> 
>
> Key: FLINK-4028
> URL: https://issues.apache.org/jira/browse/FLINK-4028
> Project: Flink
>  Issue Type: Bug
>  Components: Windowing Operators
>Affects Versions: 1.0.0, 1.1.0, 1.0.1, 1.0.2, 1.0.3
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.1.0
>
>
> In {{computeWindow}} we have this code: {{new TimeWindow(timestamp, timestamp 
> + windowSize)}}. This is wrong because {{timestamp}} is actually the end 
> timestamp of the window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4028) AbstractAlignedProcessingTimeWindowOperator creates wrong TimeWindow

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen updated FLINK-4028:

Fix Version/s: 1.1.0

> AbstractAlignedProcessingTimeWindowOperator creates wrong TimeWindow
> 
>
> Key: FLINK-4028
> URL: https://issues.apache.org/jira/browse/FLINK-4028
> Project: Flink
>  Issue Type: Bug
>  Components: Windowing Operators
>Affects Versions: 1.0.0, 1.1.0, 1.0.1, 1.0.2, 1.0.3
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.1.0
>
>
> In {{computeWindow}} we have this code: {{new TimeWindow(timestamp, timestamp 
> + windowSize)}}. This is wrong because {{timestamp}} is actually the end 
> timestamp of the window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318783#comment-15318783
 ] 

ASF GitHub Bot commented on FLINK-4000:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2062
  
Thanks, this is a critical fix.
+1

Merging this...


> Exception: Could not restore checkpointed state to operators and functions;  
> during Job Restart (Job restart is triggered due to one of the task manager 
> failure)
> -
>
> Key: FLINK-4000
> URL: https://issues.apache.org/jira/browse/FLINK-4000
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.0.3
> Environment: //Fault Tolerance Configuration of the Job
> env.enableCheckpointing(5000); 
> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
> env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1));
>Reporter: Aride Chettali
>
> java.lang.Exception: Could not restore checkpointed state to operators and 
> functions
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Failed to restore state to function: null
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449)
>   ... 3 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184)
>   at 
> org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80)
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165)
>   ... 4 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2062: [FLINK-4000] Fix for checkpoint state restore at MessageA...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2062
  
Thanks, this is a critical fix.
+1

Merging this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4011) Unable to access completed job in web frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318762#comment-15318762
 ] 

ASF GitHub Bot commented on FLINK-4011:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2065
  
Given the current status, this is a good pragmatic fif.

+1


> Unable to access completed job in web frontend
> --
>
> Key: FLINK-4011
> URL: https://issues.apache.org/jira/browse/FLINK-4011
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
>
> In the current master, I'm not able to access a finished job's detail page.
> The JobManager logs shows the following exception:
> {code}
> 2016-06-02 15:23:08,581 WARN  
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandler - Error while 
> handling request
> java.lang.RuntimeException: Couldn't deserialize ExecutionConfig.
> at 
> org.apache.flink.runtime.webmonitor.handlers.JobConfigHandler.handleRequest(JobConfigHandler.java:52)
> at 
> org.apache.flink.runtime.webmonitor.handlers.AbstractExecutionGraphRequestHandler.handleRequest(AbstractExecutionGraphRequestHandler.java:61)
> at 
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandler.respondAsLeader(RuntimeMonitorHandler.java:88)
> at 
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:84)
> at 
> org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:44)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.handler.codec.http.router.Handler.routed(Handler.java:62)
> at 
> io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:57)
> at 
> io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:20)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at 
> org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:105)
> at 
> org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:65)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at 
> io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at 
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:55)
> at 
> org.apache.flink.runtime.webmonitor.handlers.JobConfigHandler.handleRequest(JobConfigHandler.java:50)
> 

[GitHub] flink issue #2065: [FLINK-4011] Keep UserCodeClassLoader in archived Executi...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2065
  
Given the current status, this is a good pragmatic fif.

+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4017) [py] Add Aggregation support to Python API

[Geoffrey Mon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318713#comment-15318713
 ] 

Geoffrey Mon commented on FLINK-4017:
-

Thanks for the advice. The issue that was hindering progress was that the type 
of the data was rejected by the Aggregation functions, and I wasn't sure how to 
deal with that. I assumed that the nature of Python lambdas had to do with it 
after poking around with a debugger.

> [py] Add Aggregation support to Python API
> --
>
> Key: FLINK-4017
> URL: https://issues.apache.org/jira/browse/FLINK-4017
> Project: Flink
>  Issue Type: Improvement
>  Components: Python API
>Reporter: Geoffrey Mon
>Priority: Minor
>
> Aggregations are not currently supported in the Python API.
> I was getting started with setting up and working with Flink and figured this 
> would be a relatively simple task for me to get started with. Currently 
> working on this at https://github.com/geofbot/flink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4028) AbstractAlignedProcessingTimeWindowOperator creates wrong TimeWindow

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-4028:
---

 Summary: AbstractAlignedProcessingTimeWindowOperator creates wrong 
TimeWindow
 Key: FLINK-4028
 URL: https://issues.apache.org/jira/browse/FLINK-4028
 Project: Flink
  Issue Type: Bug
  Components: Windowing Operators
Affects Versions: 1.0.2, 1.0.1, 1.0.0, 1.1.0, 1.0.3
Reporter: Aljoscha Krettek


In {{computeWindow}} we have this code: {{new TimeWindow(timestamp, timestamp + 
windowSize)}}. This is wrong because {{timestamp}} is actually the end 
timestamp of the window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4017) [py] Add Aggregation support to Python API

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318695#comment-15318695
 ] 

Chesnay Schepler commented on FLINK-4017:
-

when using the python API data always exists as a byte array on the java side. 

you will either have to 
* implement the aggregation on the python side
* apply a deserializationMap, groupBy, aggregate, and serializationMap on the 
java side.

> [py] Add Aggregation support to Python API
> --
>
> Key: FLINK-4017
> URL: https://issues.apache.org/jira/browse/FLINK-4017
> Project: Flink
>  Issue Type: Improvement
>  Components: Python API
>Reporter: Geoffrey Mon
>Priority: Minor
>
> Aggregations are not currently supported in the Python API.
> I was getting started with setting up and working with Flink and figured this 
> would be a relatively simple task for me to get started with. Currently 
> working on this at https://github.com/geofbot/flink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4017) [py] Add Aggregation support to Python API

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318696#comment-15318696
 ] 

Chesnay Schepler commented on FLINK-4017:
-

what other issues did you come across?

> [py] Add Aggregation support to Python API
> --
>
> Key: FLINK-4017
> URL: https://issues.apache.org/jira/browse/FLINK-4017
> Project: Flink
>  Issue Type: Improvement
>  Components: Python API
>Reporter: Geoffrey Mon
>Priority: Minor
>
> Aggregations are not currently supported in the Python API.
> I was getting started with setting up and working with Flink and figured this 
> would be a relatively simple task for me to get started with. Currently 
> working on this at https://github.com/geofbot/flink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4017) [py] Add Aggregation support to Python API

[Geoffrey Mon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318676#comment-15318676
 ] 

Geoffrey Mon commented on FLINK-4017:
-

At least one of the issues I have come across is that aggregations are picky 
about the types of the data they are operating on, which is not explicit for 
Python lambdas. For example, in the WordCount example for Python, the return 
type of the lambda used in flat_map is assumed by Flink to be raw bytes, which 
are not Aggregator-able.

> [py] Add Aggregation support to Python API
> --
>
> Key: FLINK-4017
> URL: https://issues.apache.org/jira/browse/FLINK-4017
> Project: Flink
>  Issue Type: Improvement
>  Components: Python API
>Reporter: Geoffrey Mon
>Priority: Minor
>
> Aggregations are not currently supported in the Python API.
> I was getting started with setting up and working with Flink and figured this 
> would be a relatively simple task for me to get started with. Currently 
> working on this at https://github.com/geofbot/flink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-3530) Kafka09ITCase.testBigRecordJob fails on Travis

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger reassigned FLINK-3530:
-

Assignee: Robert Metzger

> Kafka09ITCase.testBigRecordJob fails on Travis
> --
>
> Key: FLINK-3530
> URL: https://issues.apache.org/jira/browse/FLINK-3530
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Robert Metzger
>  Labels: test-stability
>
> The test case {{Kafka09ITCase.testBigRecordJob}} failed on Travis.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/112049279/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3530) Kafka09ITCase.testBigRecordJob fails on Travis

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318631#comment-15318631
 ] 

Robert Metzger commented on FLINK-3530:
---

I have a similar failure like this: 
https://s3.amazonaws.com/archive.travis-ci.org/jobs/134215958/log.txt
I'll look into it now

> Kafka09ITCase.testBigRecordJob fails on Travis
> --
>
> Key: FLINK-3530
> URL: https://issues.apache.org/jira/browse/FLINK-3530
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>  Labels: test-stability
>
> The test case {{Kafka09ITCase.testBigRecordJob}} failed on Travis.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/112049279/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4013) GraphAlgorithms to simplify directed and undirected graphs

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-4013.
-
Resolution: Fixed

Implemented in 7160a681240deab693aa4d69c24c4a8a63bb58ba

> GraphAlgorithms to simplify directed and undirected graphs
> --
>
> Key: FLINK-4013
> URL: https://issues.apache.org/jira/browse/FLINK-4013
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> Create a directed {{GraphAlgorithm}} to remove self-loops and duplicate edges 
> and an undirected {{GraphAlgorithm}} to symmetrize and remove self-loops and 
> duplicate edges.
> Remove {{RMatGraph.setSimpleGraph}} and the associated logic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3980) Remove ExecutionConfig.PARALLELISM_UNKNOWN

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-3980.
-
Resolution: Implemented

Implemented in 21454973ed4edfb3e27bc50ed0279f5708d54fa2

> Remove ExecutionConfig.PARALLELISM_UNKNOWN
> --
>
> Key: FLINK-3980
> URL: https://issues.apache.org/jira/browse/FLINK-3980
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> FLINK-3589 added {{ExecutionConfig.PARALLELISM_DEFAULT}} and 
> {{ExecutionConfig.PARALLELISM_UNKNOWN}}. The former gave a name to the 
> contant {{-1}}  and the latter was used as a default no-op when setting the 
> parallelism.
> It's nice to keep these intents separate but given the current implementation 
> of Operator parallelism users can get by using {{PARALLELISM_DEFAULT}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4013) GraphAlgorithms to simplify directed and undirected graphs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318626#comment-15318626
 ] 

ASF GitHub Bot commented on FLINK-4013:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2067


> GraphAlgorithms to simplify directed and undirected graphs
> --
>
> Key: FLINK-4013
> URL: https://issues.apache.org/jira/browse/FLINK-4013
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> Create a directed {{GraphAlgorithm}} to remove self-loops and duplicate edges 
> and an undirected {{GraphAlgorithm}} to symmetrize and remove self-loops and 
> duplicate edges.
> Remove {{RMatGraph.setSimpleGraph}} and the associated logic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3925) GraphAlgorithm to filter by maximum degree

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318625#comment-15318625
 ] 

ASF GitHub Bot commented on FLINK-3925:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2005


> GraphAlgorithm to filter by maximum degree
> --
>
> Key: FLINK-3925
> URL: https://issues.apache.org/jira/browse/FLINK-3925
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> Filtering by minimum degree is K-Core which is iterative. Filtering by 
> maximum degree can be performed in constant time by filtering the set of 
> high-degree vertices then doing an anti-join against the original vertex set 
> and two anti-joins against the original edge set.
> Two reasons to remove high-degree vertices: 1) they may simply be noise in 
> the input data, and 2) speedup algorithms such as Adamic-Adar and Jaccard 
> Index which run quadratic in the vertex degree.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3980) Remove ExecutionConfig.PARALLELISM_UNKNOWN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318624#comment-15318624
 ] 

ASF GitHub Bot commented on FLINK-3980:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2064


> Remove ExecutionConfig.PARALLELISM_UNKNOWN
> --
>
> Key: FLINK-3980
> URL: https://issues.apache.org/jira/browse/FLINK-3980
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> FLINK-3589 added {{ExecutionConfig.PARALLELISM_DEFAULT}} and 
> {{ExecutionConfig.PARALLELISM_UNKNOWN}}. The former gave a name to the 
> contant {{-1}}  and the latter was used as a default no-op when setting the 
> parallelism.
> It's nice to keep these intents separate but given the current implementation 
> of Operator parallelism users can get by using {{PARALLELISM_DEFAULT}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2005: [FLINK-3925] GraphAlgorithm to filter by maximum d...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2005


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2064: [FLINK-3980] [core] Remove ExecutionConfig.PARALLE...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2064


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2067: [FLINK-4013] [gelly] GraphAlgorithms to simplify d...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2067


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3868) Specialized CopyableValue serializers and comparators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318592#comment-15318592
 ] 

ASF GitHub Bot commented on FLINK-3868:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/1983
  
@StephanEwen updated per your suggestions.


> Specialized CopyableValue serializers and comparators
> -
>
> Key: FLINK-3868
> URL: https://issues.apache.org/jira/browse/FLINK-3868
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.1.0
>
>
> This need was discussed on the mailing list [1] and will be obviated by code 
> generation for POJO serializers and comparators [2] (as I understand, i.e., 
> {{LongValue}} will now simply be treated as a POJO which happens to contain a 
> {{long}} and a specialized serializer and comparator will be generated).
> In the meantime, and since {{ValueTypeInfo}} will need to be reworked to use 
> the new generators, I think it is worthwhile to add specialized serializers 
> and comparators for the {{CopyableValue}} types.
> This will also provide another point of comparison for the performance of the 
> generated serializers and comparators.
> [1] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Tuple-performance-and-the-curious-JIT-compiler-td10666.html
> [2] https://issues.apache.org/jira/browse/FLINK-3599



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1983: [FLINK-3868] [core] Specialized CopyableValue serializers...

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/1983
  
@StephanEwen updated per your suggestions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-1736) Add CountVectorizer to machine learning library

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-1736:
-
Assignee: ROSHANI NAGMOTE  (was: Alexander Alexandrov)

> Add CountVectorizer to machine learning library
> ---
>
> Key: FLINK-1736
> URL: https://issues.apache.org/jira/browse/FLINK-1736
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Till Rohrmann
>Assignee: ROSHANI NAGMOTE
>  Labels: ML, Starter
>
> A {{CountVectorizer}} feature extractor [1] assigns each occurring word in a 
> corpus an unique identifier. With this mapping it can vectorize models such 
> as bag of words or ngrams in a efficient way. The unique identifier assigned 
> to a word acts as the index of a vector. The number of word occurrences is 
> represented as a vector value at a specific index. 
> The advantage of the {{CountVectorizer}} compared to the FeatureHasher is 
> that the mapping of words to indices can be obtained which makes it easier to 
> understand the resulting feature vectors.
> The {{CountVectorizer}} could be generalized to support arbitrary feature 
> values.
> The {{CountVectorizer}} should be implemented as a {{Transfomer}}.
> Resources:
> [1] 
> [http://scikit-learn.org/stable/modules/feature_extraction.html#common-vectorizer-usage]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3965) Delegating GraphAlgorithm

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318469#comment-15318469
 ] 

ASF GitHub Bot commented on FLINK-3965:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2032
  
@StephanEwen do you think this is a good and proper strategy?


> Delegating GraphAlgorithm
> -
>
> Key: FLINK-3965
> URL: https://issues.apache.org/jira/browse/FLINK-3965
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.1.0
>
>
> Complex and related algorithms often overlap in computation of data. Two such 
> examples are:
> 1) the local and global clustering coefficients each use a listing of 
> triangles
> 2) the local clustering coefficient joins on vertex degree, and the 
> underlying triangle listing annotates edge degree which uses vertex degree
> We can reuse and rewrite algorithm output by creating a {{ProxyObject}} as a 
> delegate for method calls to the {{DataSet}} returned by the algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2032
  
@StephanEwen do you think this is a good and proper strategy?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3978) Add hasBroadcastVariable method to RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318398#comment-15318398
 ] 

ASF GitHub Bot commented on FLINK-3978:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2039
  
This was closed in 5b0287971fa2beda360105d96e7bfbc7a110fae7


> Add hasBroadcastVariable method to RuntimeContext
> -
>
> Key: FLINK-3978
> URL: https://issues.apache.org/jira/browse/FLINK-3978
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> The javadocs for {{RuntimeContext}} state that {{getAccumulator}} "throws an 
> exception if the accumulator does not exist or if the accumulator exists, but 
> with different type", although {{AbstractRuntimeUDFContext}} does not throw 
> an exception but will return null.
> The javadocs for {{getBroadcastVariable}} do not mention throwing an 
> exception. Currently the only way to handle a broadcast variable that that 
> may or may not exist is to catch and ignore the exception. Adding a  
> {{containsBroadcastVariable}} method to {{RuntimeContext}} would make this 
> explicit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3978) Add hasBroadcastVariable method to RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318399#comment-15318399
 ] 

ASF GitHub Bot commented on FLINK-3978:
---

Github user greghogan closed the pull request at:

https://github.com/apache/flink/pull/2039


> Add hasBroadcastVariable method to RuntimeContext
> -
>
> Key: FLINK-3978
> URL: https://issues.apache.org/jira/browse/FLINK-3978
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> The javadocs for {{RuntimeContext}} state that {{getAccumulator}} "throws an 
> exception if the accumulator does not exist or if the accumulator exists, but 
> with different type", although {{AbstractRuntimeUDFContext}} does not throw 
> an exception but will return null.
> The javadocs for {{getBroadcastVariable}} do not mention throwing an 
> exception. Currently the only way to handle a broadcast variable that that 
> may or may not exist is to catch and ignore the exception. Adding a  
> {{containsBroadcastVariable}} method to {{RuntimeContext}} would make this 
> explicit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2039: [FLINK-3978] [core] Add hasBroadcastVariable method to Ru...

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2039
  
This was closed in 5b0287971fa2beda360105d96e7bfbc7a110fae7


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2039: [FLINK-3978] [core] Add hasBroadcastVariable metho...

[greghogan]

Github user greghogan closed the pull request at:

https://github.com/apache/flink/pull/2039


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3980) Remove ExecutionConfig.PARALLELISM_UNKNOWN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318385#comment-15318385
 ] 

ASF GitHub Bot commented on FLINK-3980:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2064
  
Will rebase and merge ...


> Remove ExecutionConfig.PARALLELISM_UNKNOWN
> --
>
> Key: FLINK-3980
> URL: https://issues.apache.org/jira/browse/FLINK-3980
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.1.0
>
>
> FLINK-3589 added {{ExecutionConfig.PARALLELISM_DEFAULT}} and 
> {{ExecutionConfig.PARALLELISM_UNKNOWN}}. The former gave a name to the 
> contant {{-1}}  and the latter was used as a default no-op when setting the 
> parallelism.
> It's nice to keep these intents separate but given the current implementation 
> of Operator parallelism users can get by using {{PARALLELISM_DEFAULT}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2064: [FLINK-3980] [core] Remove ExecutionConfig.PARALLELISM_UN...

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2064
  
Will rebase and merge ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3777) Add open and close methods to manage IF lifecycle

[Flavio Pompermaier (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318374#comment-15318374
 ] 

Flavio Pompermaier commented on FLINK-3777:
---

In our use case we have this very complex query that produce about 11 billions 
of records and we did some benchmark in order to determine the perfect size of 
the splits.
That best split size happened to be around 100k (per query), because as you 
stated, there's a trade-off between the complexity on the JobManager side but 
there's also a trade-off on the database server capability to answer to wide 
range of keys. 
Splitting the entire key-set into just a small number of splits causes the job 
to die because the queries never ends (i.e. timeout exceptions).

That was our "painful" experience..

> Add open and close methods to manage IF lifecycle
> -
>
> Key: FLINK-3777
> URL: https://issues.apache.org/jira/browse/FLINK-3777
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.1
>Reporter: Flavio Pompermaier
>Assignee: Flavio Pompermaier
>  Labels: inputformat, lifecycle
>
> At the moment the opening and closing of an inputFormat are not managed, 
> although open() could be (improperly IMHO) simulated by configure().
> This limits the possibility to reuse expensive resources (like database 
> connections) and manage their release. 
> Probably the best option would be to add 2 methods (i.e. openInputformat() 
> and closeInputFormat() ) to RichInputFormat*
> * NOTE: the best option from a "semantic" point of view would be to rename 
> the current open() and close() to openSplit() and closeSplit() respectively 
> while using open() and close() methods for the IF lifecycle management, but 
> this would cause a backward compatibility issue...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4024) FileSourceFunction not adjusted to new IF lifecycle

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318338#comment-15318338
 ] 

Stephan Ewen commented on FLINK-4024:
-

Related: I commented on FLINK-3777 voicing some concerns if that is a desirable 
functionality in the first place.

> FileSourceFunction not adjusted to new IF lifecycle
> ---
>
> Key: FLINK-4024
> URL: https://issues.apache.org/jira/browse/FLINK-4024
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Priority: Critical
> Fix For: 1.1.0
>
>
> The InputFormat lifecycle was extended in 
> ac2137cfa5e63bd4f53a4b7669dc591ab210093f, adding additional 
> open-/closeInputFormat() methods.
> The streaming FileSourceFunction was not adjusted for this change, and thus 
> will fail for every InputFormat that leverages these new methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3777) Add open and close methods to manage IF lifecycle

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318333#comment-15318333
 ] 

Stephan Ewen commented on FLINK-3777:
-

I understand that these allow initialization across input splits. My feeling is 
that if you need to initialize something a million times per input format, you 
are probably not creating the splits correctly.
You will still have overhead for the communication of the splits, complexity 
and potential memory issues on the JobManager for millions of splits.

The right fix for that would be to create the splits more correctly, rather 
than add workaround methods to make the per-split work in the input format 
less, at the cost of making the API harder to maintain.

> Add open and close methods to manage IF lifecycle
> -
>
> Key: FLINK-3777
> URL: https://issues.apache.org/jira/browse/FLINK-3777
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.1
>Reporter: Flavio Pompermaier
>Assignee: Flavio Pompermaier
>  Labels: inputformat, lifecycle
>
> At the moment the opening and closing of an inputFormat are not managed, 
> although open() could be (improperly IMHO) simulated by configure().
> This limits the possibility to reuse expensive resources (like database 
> connections) and manage their release. 
> Probably the best option would be to add 2 methods (i.e. openInputformat() 
> and closeInputFormat() ) to RichInputFormat*
> * NOTE: the best option from a "semantic" point of view would be to rename 
> the current open() and close() to openSplit() and closeSplit() respectively 
> while using open() and close() methods for the IF lifecycle management, but 
> this would cause a backward compatibility issue...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4016) FoldApplyWindowFunction is not properly initialized

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318320#comment-15318320
 ] 

Aljoscha Krettek commented on FLINK-4016:
-

I don't think so, FLINK-3977 describes a more general problem of which this 
(FLINK-4016) is a manifestation.

> FoldApplyWindowFunction is not properly initialized
> ---
>
> Key: FLINK-4016
> URL: https://issues.apache.org/jira/browse/FLINK-4016
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.0.3
>Reporter: RWenden
>Priority: Blocker
>  Labels: easyfix
> Fix For: 1.1.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> FoldApplyWindowFunction's outputtype is not set.
> We're using constructions like (excerpt):
>   .keyBy(0)
>   .countWindow(10, 5)
>   .fold(...)
> Running this stream gives an runtime exception in FoldApplyWindowFunction:
> "No initial value was serialized for the fold window function. Probably the 
> setOutputType method was not called."
> This can be easily fixed in WindowedStream.java by (around line# 449):
> FoldApplyWindowFunction foldApplyWindowFunction = new 
> FoldApplyWindowFunction<>(initialValue, foldFunction, function);
> foldApplyWindowFunction.setOutputType(resultType, 
> input.getExecutionConfig());
> operator = new EvictingWindowOperator<>(windowAssigner,
> 
> windowAssigner.getWindowSerializer(getExecutionEnvironment().getConfig()),
> keySel,
> 
> input.getKeyType().createSerializer(getExecutionEnvironment().getConfig()),
> stateDesc,
> new 
> InternalIterableWindowFunction<>(foldApplyWindowFunction),
> trigger,
> evictor);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318272#comment-15318272
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-3923:


The consumer isn't using the KCL at the moment, so default overriding wouldn't 
be a problem. As for KPL in the producer, I've heard from Robert that he's 
already talked with Amazon and came up with some conclusions. He's probably 
still a bit busy early this week, should update with the conclusions soon.

> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4016) FoldApplyWindowFunction is not properly initialized

[RWenden (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318226#comment-15318226
 ] 

RWenden commented on FLINK-4016:


Isn't it a better idea to close Jira 3977 and re-open 4016 ?


On Tue, Jun 7, 2016 at 10:54 AM, ASF GitHub Bot (JIRA) 



> FoldApplyWindowFunction is not properly initialized
> ---
>
> Key: FLINK-4016
> URL: https://issues.apache.org/jira/browse/FLINK-4016
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.0.3
>Reporter: RWenden
>Priority: Blocker
>  Labels: easyfix
> Fix For: 1.1.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> FoldApplyWindowFunction's outputtype is not set.
> We're using constructions like (excerpt):
>   .keyBy(0)
>   .countWindow(10, 5)
>   .fold(...)
> Running this stream gives an runtime exception in FoldApplyWindowFunction:
> "No initial value was serialized for the fold window function. Probably the 
> setOutputType method was not called."
> This can be easily fixed in WindowedStream.java by (around line# 449):
> FoldApplyWindowFunction foldApplyWindowFunction = new 
> FoldApplyWindowFunction<>(initialValue, foldFunction, function);
> foldApplyWindowFunction.setOutputType(resultType, 
> input.getExecutionConfig());
> operator = new EvictingWindowOperator<>(windowAssigner,
> 
> windowAssigner.getWindowSerializer(getExecutionEnvironment().getConfig()),
> keySel,
> 
> input.getKeyType().createSerializer(getExecutionEnvironment().getConfig()),
> stateDesc,
> new 
> InternalIterableWindowFunction<>(foldApplyWindowFunction),
> trigger,
> evictor);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4016) FoldApplyWindowFunction is not properly initialized

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318163#comment-15318163
 ] 

ASF GitHub Bot commented on FLINK-4016:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2070
  
Hi,
the clean way to go about this is to close this PR and open a new one for 
`FLINK-3977`. I wrote a new comment in the issue, outlining how the problem can 
be solved: https://issues.apache.org/jira/browse/FLINK-3977.


> FoldApplyWindowFunction is not properly initialized
> ---
>
> Key: FLINK-4016
> URL: https://issues.apache.org/jira/browse/FLINK-4016
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.0.3
>Reporter: RWenden
>Priority: Blocker
>  Labels: easyfix
> Fix For: 1.1.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> FoldApplyWindowFunction's outputtype is not set.
> We're using constructions like (excerpt):
>   .keyBy(0)
>   .countWindow(10, 5)
>   .fold(...)
> Running this stream gives an runtime exception in FoldApplyWindowFunction:
> "No initial value was serialized for the fold window function. Probably the 
> setOutputType method was not called."
> This can be easily fixed in WindowedStream.java by (around line# 449):
> FoldApplyWindowFunction foldApplyWindowFunction = new 
> FoldApplyWindowFunction<>(initialValue, foldFunction, function);
> foldApplyWindowFunction.setOutputType(resultType, 
> input.getExecutionConfig());
> operator = new EvictingWindowOperator<>(windowAssigner,
> 
> windowAssigner.getWindowSerializer(getExecutionEnvironment().getConfig()),
> keySel,
> 
> input.getKeyType().createSerializer(getExecutionEnvironment().getConfig()),
> stateDesc,
> new 
> InternalIterableWindowFunction<>(foldApplyWindowFunction),
> trigger,
> evictor);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2070: [FLINK-4016] initialize FoldApplyWindowFunction properly

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2070
  
Hi,
the clean way to go about this is to close this PR and open a new one for 
`FLINK-3977`. I wrote a new comment in the issue, outlining how the problem can 
be solved: https://issues.apache.org/jira/browse/FLINK-3977.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3977) Subclasses of InternalWindowFunction must support OutputTypeConfigurable

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318161#comment-15318161
 ] 

Aljoscha Krettek commented on FLINK-3977:
-

The way to solve this is to make subclasses of {{InternalWindowFunction}} 
extends {{OutputTypeConfigurable}}. When {{setOutputType()}} is called these 
must check whether the wrapped function is {{OutputTypeConfigurable}} and 
forward the call.

> Subclasses of InternalWindowFunction must support OutputTypeConfigurable
> 
>
> Key: FLINK-3977
> URL: https://issues.apache.org/jira/browse/FLINK-3977
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Priority: Critical
>
> Right now, if they wrap functions and a wrapped function implements 
> {{OutputTypeConfigurable}}, {{setOutputType}} is never called. This manifests 
> itself, for example, in FoldFunction on a window with evictor not working.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[Abdullah Ozturk (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318155#comment-15318155
 ] 

Abdullah Ozturk commented on FLINK-3923:


I think it is really good idea. Except, I am not sure about overriding the 
default values already set by KPL or KCL with our defaults if it has no 
definite purpose. But, how about the state of the decision for whether using 
Amazon licensed libraries or not in Flink? According to my last knowledge, 
enhancing these configurations were up to that decision. 

> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2077: [FLINK-4026] Fix code, grammar, and link issues in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2077


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-4026) Fix code, grammar, and link issues in the Streaming documentation

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek resolved FLINK-4026.
-
   Resolution: Fixed
Fix Version/s: 1.1.0

Fixed

> Fix code, grammar, and link issues in the Streaming documentation
> -
>
> Key: FLINK-4026
> URL: https://issues.apache.org/jira/browse/FLINK-4026
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Dyana Rose
>Priority: Trivial
> Fix For: 1.1.0
>
>
> The streaming API section of the documentation has issues with grammar that 
> make it hard to follow in places. As well as an incorrect code example, and 
> places of unnecessary parentheses on the Windows page, and a missing link for 
> Kinesis Streams on the Connectors index page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4026) Fix code, grammar, and link issues in the Streaming documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318152#comment-15318152
 ] 

ASF GitHub Bot commented on FLINK-4026:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2077


> Fix code, grammar, and link issues in the Streaming documentation
> -
>
> Key: FLINK-4026
> URL: https://issues.apache.org/jira/browse/FLINK-4026
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Dyana Rose
>Priority: Trivial
> Fix For: 1.1.0
>
>
> The streaming API section of the documentation has issues with grammar that 
> make it hard to follow in places. As well as an incorrect code example, and 
> places of unnecessary parentheses on the Windows page, and a missing link for 
> Kinesis Streams on the Connectors index page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4026) Fix code, grammar, and link issues in the Streaming documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318148#comment-15318148
 ] 

ASF GitHub Bot commented on FLINK-4026:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2077
  
Thanks for your fixes! I'm merging.  


> Fix code, grammar, and link issues in the Streaming documentation
> -
>
> Key: FLINK-4026
> URL: https://issues.apache.org/jira/browse/FLINK-4026
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Dyana Rose
>Priority: Trivial
>
> The streaming API section of the documentation has issues with grammar that 
> make it hard to follow in places. As well as an incorrect code example, and 
> places of unnecessary parentheses on the Windows page, and a missing link for 
> Kinesis Streams on the Connectors index page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2077: [FLINK-4026] Fix code, grammar, and link issues in the St...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2077
  
Thanks for your fixes! I'm merging. 😃 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---