[jira] [Created] (FLINK-7003) "select * from" in Flink SQL should not flatten all fields in the table by default

[Shuyi Chen (JIRA)]

Shuyi Chen created FLINK-7003:
-

 Summary: "select * from" in Flink SQL should not flatten all 
fields in the table by default
 Key: FLINK-7003
 URL: https://issues.apache.org/jira/browse/FLINK-7003
 Project: Flink
  Issue Type: Bug
Reporter: Shuyi Chen


Currently, CompositeRelDataType is extended from 
RelRecordType(StructKind.PEEK_FIELDS, ...).  In Calcite, StructKind.PEEK_FIELDS 
would allow us to peek fields for nested types. However, when we use "select * 
from", calcite will flatten all nested fields that is marked as 
StructKind.PEEK_FIELDS in the table. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6764) Deduplicate stateless TypeSerializers when serializing composite TypeSerializers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062547#comment-16062547
 ] 

ASF GitHub Bot commented on FLINK-6764:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4026
  
@zentol there was a recent offline discussion regarding the savepoint 
formats that will probably void this change. Will close this PR for now.


> Deduplicate stateless TypeSerializers when serializing composite 
> TypeSerializers
> 
>
> Key: FLINK-6764
> URL: https://issues.apache.org/jira/browse/FLINK-6764
> Project: Flink
>  Issue Type: Improvement
>  Components: Type Serialization System
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Till Rohrmann
>Assignee: Tzu-Li (Gordon) Tai
>
> Composite type serializer, such as the {{PojoSerializer}}, could be improved 
> by deduplicating stateless {{TypeSerializer}} when being serialized. This 
> would decrease their serialization size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6764) Deduplicate stateless TypeSerializers when serializing composite TypeSerializers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062548#comment-16062548
 ] 

ASF GitHub Bot commented on FLINK-6764:
---

Github user tzulitai closed the pull request at:

https://github.com/apache/flink/pull/4026


> Deduplicate stateless TypeSerializers when serializing composite 
> TypeSerializers
> 
>
> Key: FLINK-6764
> URL: https://issues.apache.org/jira/browse/FLINK-6764
> Project: Flink
>  Issue Type: Improvement
>  Components: Type Serialization System
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Till Rohrmann
>Assignee: Tzu-Li (Gordon) Tai
>
> Composite type serializer, such as the {{PojoSerializer}}, could be improved 
> by deduplicating stateless {{TypeSerializer}} when being serialized. This 
> would decrease their serialization size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4026: [FLINK-6764] Deduplicate stateless serializers in ...

[tzulitai]

Github user tzulitai closed the pull request at:

https://github.com/apache/flink/pull/4026


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4026: [FLINK-6764] Deduplicate stateless serializers in checkpo...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4026
  
@zentol there was a recent offline discussion regarding the savepoint 
formats that will probably void this change. Will close this PR for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6939) Not store IterativeCondition with NFA state

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062508#comment-16062508
 ] 

Jark Wu commented on FLINK-6939:


Hi [~kkl0u], thanks for your response.  The purpose of this issue is that, in 
order to integrate CEP and SQL, an important step is code generation of 
IterativeCondition and compile it in runtime. But the conditions are 
de/serialized on every input elements, so that we have to compile the code on 
every input elements which is not we want. We hope to compile the code only in 
the setup phase of CEP operator. In order to achieve this goal, we should not 
store conditions with NFA states.

I think store conditions upon creation of the NFA is a good idea. And 
[~dian.fu] and [~Dawid Wysakowicz had a discussion about this last week.  
Storing NFA meta data (I think it should include conditions) separately with 
NFA states. See FLINK-6983

> Not store IterativeCondition with NFA state
> ---
>
> Key: FLINK-6939
> URL: https://issues.apache.org/jira/browse/FLINK-6939
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Jark Wu
>Assignee: Jark Wu
> Fix For: 1.4.0
>
>
> Currently, the IterativeCondition is stored with the total NFA state. And 
> de/serialized every time when update/get the NFA state. It is a heavy 
> operation and not necessary. In addition it is a required feature for 
> FLINK-6938.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6960) Add E() supported in SQL

[Jark Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu closed FLINK-6960.
--
   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed in 524a1fa0fecbcbd14d3cb6b9e99d20f424d58004

> Add E() supported in SQL
> 
>
> Key: FLINK-6960
> URL: https://issues.apache.org/jira/browse/FLINK-6960
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>  Labels: starter
> Fix For: 1.4.0
>
>
> E=Math.E 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6960) Add E() supported in SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062476#comment-16062476
 ] 

ASF GitHub Bot commented on FLINK-6960:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4152


> Add E() supported in SQL
> 
>
> Key: FLINK-6960
> URL: https://issues.apache.org/jira/browse/FLINK-6960
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>  Labels: starter
> Fix For: 1.4.0
>
>
> E=Math.E 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4152: [FLINK-6960][table] Add E supported in SQL.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4152


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6991) Inaccessible link under Gelly document

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062471#comment-16062471
 ] 

mingleizhang commented on FLINK-6991:
-

[~Zentol] Could you give him a permission that contribute to Flink ? Thanks.

> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: mingleizhang
>
> When I visited the web 
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top, 
> then at the bottom of the page there is a Gelly Documentation link. It 
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6991) Inaccessible link under Gelly document

[njzhuyuqi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062466#comment-16062466
 ] 

njzhuyuqi commented on FLINK-6991:
--

I am a new to Flink.  Can anyone give me a permission that contribute to Flink?

> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: mingleizhang
>
> When I visited the web 
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top, 
> then at the bottom of the page there is a Gelly Documentation link. It 
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7000) Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062459#comment-16062459
 ] 

ASF GitHub Bot commented on FLINK-7000:
---

Github user ch33hau commented on a diff in the pull request:

https://github.com/apache/flink/pull/4178#discussion_r123912353
  
--- Diff: 
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/StreamExecutionEnvironment.scala
 ---
@@ -768,6 +768,18 @@ object StreamExecutionEnvironment {
   }
 
   /**
+   * Creates a local execution environment. The local execution 
environment will run the
+   * program in a multi-threaded fashion in the same JVM as the 
environment was created in.
+   *
+   * @param parallelism   The parallelism for the local environment.
+   * @param configuration Pass a custom configuration into the cluster.
+   */
+  def createLocalEnvironment(parallelism: Int, configuration: 
Configuration):
--- End diff --

No it can't. I have think about it before, again compiler complains.

It could be related to this [scala 
issue](https://stackoverflow.com/questions/4652095/why-does-the-scala-compiler-disallow-overloaded-methods-with-default-arguments)

If you have two overloads with defaults on the same parameter position, 
we would need a different naming scheme. 
But we want to keep the generated byte-code stable over multiple 
compiler runs.


> Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment
> --
>
> Key: FLINK-7000
> URL: https://issues.apache.org/jira/browse/FLINK-7000
> Project: Flink
>  Issue Type: Improvement
>Reporter: Lim Chee Hau
>
> I was doing some local testings in {{Scala}} environment, however I found 
> that there is no straightforward way to add custom configuration to 
> {{StreamExecutionEnvironment}} by using {{createLocalEnvironment}} method. 
> This could be easily achieve in {{Java}} environment since 
> {{StreamExecutionEnvironment}} in {{Java}} has 
> - {{createLocalEnvironment()}}
> - {{createLocalEnvironment(Int)}}
> - {{createLocalEnvironment(Int, Configuration)}}
> Whereas Scala only has 2 out of 3 of these methods.
> Not sure if this is a missing implementation, if yes I could create a PR for 
> this.
> Therefore the example in [Local 
> Execution|https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/local_execution.html]
>  could be making sense for Scala users as well:
> bq. The LocalEnvironment allows also to pass custom configuration values to 
> Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4178: [FLINK-7000] Add custom configuration local enviro...

[ch33hau]

Github user ch33hau commented on a diff in the pull request:

https://github.com/apache/flink/pull/4178#discussion_r123912353
  
--- Diff: 
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/StreamExecutionEnvironment.scala
 ---
@@ -768,6 +768,18 @@ object StreamExecutionEnvironment {
   }
 
   /**
+   * Creates a local execution environment. The local execution 
environment will run the
+   * program in a multi-threaded fashion in the same JVM as the 
environment was created in.
+   *
+   * @param parallelism   The parallelism for the local environment.
+   * @param configuration Pass a custom configuration into the cluster.
+   */
+  def createLocalEnvironment(parallelism: Int, configuration: 
Configuration):
--- End diff --

No it can't. I have think about it before, again compiler complains.

It could be related to this [scala 
issue](https://stackoverflow.com/questions/4652095/why-does-the-scala-compiler-disallow-overloaded-methods-with-default-arguments)

If you have two overloads with defaults on the same parameter position, 
we would need a different naming scheme. 
But we want to keep the generated byte-code stable over multiple 
compiler runs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6965) Avro is missing snappy dependency

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062426#comment-16062426
 ] 

ASF GitHub Bot commented on FLINK-6965:
---

Github user packet23 commented on a diff in the pull request:

https://github.com/apache/flink/pull/4160#discussion_r123909436
  
--- Diff: tools/travis_mvn_watchdog.sh ---
@@ -182,6 +182,13 @@ check_shaded_artifacts() {
exit 1
fi
 
+   SNAPPY=`cat allClasses | grep '^org/xerial/snappy' | wc -l`
+   if [ AVRO == "0" ]; then
--- End diff --

Should it not read
```bash
if [ $SNAPPY == "0" ]; then
   # bail out
fi
```


> Avro is missing snappy dependency
> -
>
> Key: FLINK-6965
> URL: https://issues.apache.org/jira/browse/FLINK-6965
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.3.2
>
>
> The shading rework made before 1.3 removed a snappy dependency that was 
> accidentally pulled in through hadoop. This is technically alright, until 
> class-loaders rear their ugly heads.
> Our kafka connector can read avro records, which may or may not require 
> snappy. Usually this _should_ be solvable by including the snappy dependency 
> in the user-jar if necessary, however since the kafka connector loads classes 
> that it requires using the system class loader this doesn't work.
> As such we have to add a separate snappy dependency to flink-core.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4160: {FLINK-6965] Include snappy-java in flink-dist

[packet23]

Github user packet23 commented on a diff in the pull request:

https://github.com/apache/flink/pull/4160#discussion_r123909436
  
--- Diff: tools/travis_mvn_watchdog.sh ---
@@ -182,6 +182,13 @@ check_shaded_artifacts() {
exit 1
fi
 
+   SNAPPY=`cat allClasses | grep '^org/xerial/snappy' | wc -l`
+   if [ AVRO == "0" ]; then
--- End diff --

Should it not read
```bash
if [ $SNAPPY == "0" ]; then
   # bail out
fi
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-7002) Partitioning broken if enum is used in compound key specified using field expression

[Sebastian Klemke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebastian Klemke updated FLINK-7002:

Attachment: TestJob.java
WorkingTestJob.java
testdata.avro

> Partitioning broken if enum is used in compound key specified using field 
> expression
> 
>
> Key: FLINK-7002
> URL: https://issues.apache.org/jira/browse/FLINK-7002
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.2.0, 1.3.1
>Reporter: Sebastian Klemke
> Attachments: testdata.avro, TestJob.java, WorkingTestJob.java
>
>
> When groupBy() or keyBy() is used with multiple field expressions, at least 
> one of them being an enum type serialized using EnumTypeInfo, partitioning 
> seems random, resulting in incorrectly grouped/keyed output 
> datasets/datastreams.
> The attached Flink DataSet API jobs and the test dataset detail the issue: 
> Both jobs count (id, type) occurrences, TestJob uses field expressions to 
> group, WorkingTestJob uses a KeySelector function.
> Expected output for both is 6 records, with frequency value 100_000 each. If 
> you run in LocalEnvironment, results are in fact equivalent. But when run on 
> a cluster with 5 TaskManagers, only KeySelector function with String key 
> produces correct results whereas field expressions produce random, 
> non-repeatable, wrong results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7002) Partitioning broken if enum is used in compound key specified using field expression

[Sebastian Klemke (JIRA)]

Sebastian Klemke created FLINK-7002:
---

 Summary: Partitioning broken if enum is used in compound key 
specified using field expression
 Key: FLINK-7002
 URL: https://issues.apache.org/jira/browse/FLINK-7002
 Project: Flink
  Issue Type: Bug
  Components: Core
Affects Versions: 1.3.1, 1.2.0
Reporter: Sebastian Klemke


When groupBy() or keyBy() is used with multiple field expressions, at least one 
of them being an enum type serialized using EnumTypeInfo, partitioning seems 
random, resulting in incorrectly grouped/keyed output datasets/datastreams.

The attached Flink DataSet API jobs and the test dataset detail the issue: Both 
jobs count (id, type) occurrences, TestJob uses field expressions to group, 
WorkingTestJob uses a KeySelector function.

Expected output for both is 6 records, with frequency value 100_000 each. If 
you run in LocalEnvironment, results are in fact equivalent. But when run on a 
cluster with 5 TaskManagers, only KeySelector function with String key produces 
correct results whereas field expressions produce random, non-repeatable, wrong 
results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7000) Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062406#comment-16062406
 ] 

ASF GitHub Bot commented on FLINK-7000:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4178#discussion_r123906377
  
--- Diff: 
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/StreamExecutionEnvironment.scala
 ---
@@ -768,6 +768,18 @@ object StreamExecutionEnvironment {
   }
 
   /**
+   * Creates a local execution environment. The local execution 
environment will run the
+   * program in a multi-threaded fashion in the same JVM as the 
environment was created in.
+   *
+   * @param parallelism   The parallelism for the local environment.
+   * @param configuration Pass a custom configuration into the cluster.
+   */
+  def createLocalEnvironment(parallelism: Int, configuration: 
Configuration):
--- End diff --

can the parallelism have a default value?


> Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment
> --
>
> Key: FLINK-7000
> URL: https://issues.apache.org/jira/browse/FLINK-7000
> Project: Flink
>  Issue Type: Improvement
>Reporter: Lim Chee Hau
>
> I was doing some local testings in {{Scala}} environment, however I found 
> that there is no straightforward way to add custom configuration to 
> {{StreamExecutionEnvironment}} by using {{createLocalEnvironment}} method. 
> This could be easily achieve in {{Java}} environment since 
> {{StreamExecutionEnvironment}} in {{Java}} has 
> - {{createLocalEnvironment()}}
> - {{createLocalEnvironment(Int)}}
> - {{createLocalEnvironment(Int, Configuration)}}
> Whereas Scala only has 2 out of 3 of these methods.
> Not sure if this is a missing implementation, if yes I could create a PR for 
> this.
> Therefore the example in [Local 
> Execution|https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/local_execution.html]
>  could be making sense for Scala users as well:
> bq. The LocalEnvironment allows also to pass custom configuration values to 
> Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4178: [FLINK-7000] Add custom configuration local enviro...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4178#discussion_r123906377
  
--- Diff: 
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/StreamExecutionEnvironment.scala
 ---
@@ -768,6 +768,18 @@ object StreamExecutionEnvironment {
   }
 
   /**
+   * Creates a local execution environment. The local execution 
environment will run the
+   * program in a multi-threaded fashion in the same JVM as the 
environment was created in.
+   *
+   * @param parallelism   The parallelism for the local environment.
+   * @param configuration Pass a custom configuration into the cluster.
+   */
+  def createLocalEnvironment(parallelism: Int, configuration: 
Configuration):
--- End diff --

can the parallelism have a default value?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4145: [FLINK-6938][FLINK-6939] [cep] Not store IterativeConditi...

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4145
  
Hi @wuchong ! Nice work! 

I have some changes to propose in the code itself, but before moving on 
with this, I have some questions about the need of the change. 

For the second, I have commented on the JIRA (FLINK-6939) and it would be 
nice to finish the discussion there first, before checking the following 
comments.

Now for the comments on the code I would suggest the following:

1) Remove the `TransitionInfo` from the `ConditionRegistry` and replace it 
with the `StateTransition`, as the `TransitionInfo` is just a copy of the 
state transition. Also make the
`ConditionRegistry`, `ConditionRegistry`.
2)  Pass the `ConditionRegistry` to the `State` itself in the constructor. 
This will simplify all the changes of the `addStateTransition()`, 
`addProceed()`... as they do not need the registry as an additional argument 
anymore.
3) If we agree to keep the `RichIterativeFunction`, then we should probably 
think if we can remove the non-rich `IterativeCondition` (although this is 
optional).

For these changes, I have a branch where I tested them a bit but of course 
feel free to make the code nicer, and also tell me if I did any change that 
does not seem reasonable to you. This branch was just a fast testing branch on 
my side. You can find it here:

https://github.com/kl0u/flink/tree/cep-iter-pr

Again thanks a lot for the work and I am looking forward to you comments!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6938) IterativeCondition should support RichFunction interface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062403#comment-16062403
 ] 

ASF GitHub Bot commented on FLINK-6938:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4145
  
Hi @wuchong ! Nice work! 

I have some changes to propose in the code itself, but before moving on 
with this, I have some questions about the need of the change. 

For the second, I have commented on the JIRA (FLINK-6939) and it would be 
nice to finish the discussion there first, before checking the following 
comments.

Now for the comments on the code I would suggest the following:

1) Remove the `TransitionInfo` from the `ConditionRegistry` and replace it 
with the `StateTransition`, as the `TransitionInfo` is just a copy of the 
state transition. Also make the
`ConditionRegistry`, `ConditionRegistry`.
2)  Pass the `ConditionRegistry` to the `State` itself in the constructor. 
This will simplify all the changes of the `addStateTransition()`, 
`addProceed()`... as they do not need the registry as an additional argument 
anymore.
3) If we agree to keep the `RichIterativeFunction`, then we should probably 
think if we can remove the non-rich `IterativeCondition` (although this is 
optional).

For these changes, I have a branch where I tested them a bit but of course 
feel free to make the code nicer, and also tell me if I did any change that 
does not seem reasonable to you. This branch was just a fast testing branch on 
my side. You can find it here:

https://github.com/kl0u/flink/tree/cep-iter-pr

Again thanks a lot for the work and I am looking forward to you comments!



> IterativeCondition should support RichFunction interface
> 
>
> Key: FLINK-6938
> URL: https://issues.apache.org/jira/browse/FLINK-6938
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Jark Wu
>Assignee: Jark Wu
> Fix For: 1.4.0
>
>
> In FLIP-20, we need IterativeCondition to support an {{open()}} method to 
> compile the generated code once. We do not want to insert a if condition  in 
> the {{filter()}} method. So I suggest make IterativeCondition support 
> {{RichFunction}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6939) Not store IterativeCondition with NFA state

[Kostas Kloudas (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062398#comment-16062398
 ] 

Kostas Kloudas commented on FLINK-6939:
---

Hi [~jark], I reviewed your PR and have some comments on the code itself, but 
before publishing them I would like to understand the need of this change. 

Although it seems nice, and, in general, I liked the code,  the problem is that 
it is not aligned with another change that we have in the pipeline, which is 
dynamically adding patterns to the CEP operator. In a nutshell, you can imagine 
a co-stream where the patterns will come from, and then be translated into NFAs 
and be stored. In this case, not storing the conditions can result in problems, 
as the Patterns are not hardcoded in the program anymore, and they cannot be 
necessarily re-produced upon recovery. 

We may be able to find a middle ground, where conditions are not always 
serialized and de-serialized, but they are still stored upon creation of the 
NFA, but before going down this road, could you please explain me a bit if this 
is just an optimization or it serves any other purpose?

> Not store IterativeCondition with NFA state
> ---
>
> Key: FLINK-6939
> URL: https://issues.apache.org/jira/browse/FLINK-6939
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Jark Wu
>Assignee: Jark Wu
> Fix For: 1.4.0
>
>
> Currently, the IterativeCondition is stored with the total NFA state. And 
> de/serialized every time when update/get the NFA state. It is a heavy 
> operation and not necessary. In addition it is a required feature for 
> FLINK-6938.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6742) Improve error message when savepoint migration fails due to task removal

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062364#comment-16062364
 ] 

Chesnay Schepler commented on FLINK-6742:
-

I merged it to 1.3 after your comment, so it was ok ;)

I'll think about your suggestion regarding the parallelism tomorrow.

> Improve error message when savepoint migration fails due to task removal
> 
>
> Key: FLINK-6742
> URL: https://issues.apache.org/jira/browse/FLINK-6742
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Gyula Fora
>Assignee: Chesnay Schepler
>Priority: Minor
>  Labels: flink-rel-1.3.1-blockers
>
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointV2.convertToOperatorStateSavepointV2(SavepointV2.java:171)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:75)
>   at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1090)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4026: [FLINK-6764] Deduplicate stateless serializers in checkpo...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4026
  
@tzulitai What's the state of this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6764) Deduplicate stateless TypeSerializers when serializing composite TypeSerializers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062359#comment-16062359
 ] 

ASF GitHub Bot commented on FLINK-6764:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4026
  
@tzulitai What's the state of this PR?


> Deduplicate stateless TypeSerializers when serializing composite 
> TypeSerializers
> 
>
> Key: FLINK-6764
> URL: https://issues.apache.org/jira/browse/FLINK-6764
> Project: Flink
>  Issue Type: Improvement
>  Components: Type Serialization System
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Till Rohrmann
>Assignee: Tzu-Li (Gordon) Tai
>
> Composite type serializer, such as the {{PojoSerializer}}, could be improved 
> by deduplicating stateless {{TypeSerializer}} when being serialized. This 
> would decrease their serialization size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3554: Detached (Remote)StreamEnvironment execution

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3554#discussion_r123901226
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/RemoteStreamEnvironment.java
 ---
@@ -25,229 +25,205 @@
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
-
 import org.apache.flink.annotation.Public;
 import org.apache.flink.api.common.InvalidProgramException;
 import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.JobSubmissionResult;
 import org.apache.flink.api.java.ExecutionEnvironment;
 import org.apache.flink.client.program.ClusterClient;
 import org.apache.flink.client.program.JobWithJars;
 import org.apache.flink.client.program.ProgramInvocationException;
 import org.apache.flink.client.program.StandaloneClusterClient;
 import org.apache.flink.configuration.ConfigConstants;
 import org.apache.flink.configuration.Configuration;
-
 import org.apache.flink.streaming.api.graph.StreamGraph;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 @Public
 public class RemoteStreamEnvironment extends StreamExecutionEnvironment {
-   
-   private static final Logger LOG = 
LoggerFactory.getLogger(RemoteStreamEnvironment.class);
-
-   /** The hostname of the JobManager */
-   private final String host;
-
-   /** The port of the JobManager main actor system */
--- End diff --

please revert the auto-formatting here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-6958) Async I/O timeout not work

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062351#comment-16062351
 ] 

Jark Wu edited comment on FLINK-6958 at 6/25/17 3:07 PM:
-

[~till.rohrmann] I'm afraid it doesn't work, because the aync io operator hang 
on the {{checkpointingLock.wait()}} not {{super.close()}} in 
{{AsyncWaitOperator.close()}} method.  


was (Author: jark):
[~till.rohrmann] I'm afraid it doesn't work, because the aync io operator hang 
on the {{checkpointingLock.wait()}} not {{super.close()}} in {{close()}} 
method.  

> Async I/O timeout not work
> --
>
> Key: FLINK-6958
> URL: https://issues.apache.org/jira/browse/FLINK-6958
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: feng xiaojie
>Assignee: Jark Wu
>
> when use Async I/O with UnorderedStreamElementQueue, the queue will always 
> full if you don't  call the AsyncCollector.collect to ack them.
> Timeout shall collect these entries when the timeout trigger,but it isn't work
> I debug find,
> when time out, it will call resultFuture.completeExceptionally(error);
> but not call  UnorderedStreamElementQueue.onCompleteHandler
> it will cause that async i/o hang always



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6958) Async I/O timeout not work

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062351#comment-16062351
 ] 

Jark Wu commented on FLINK-6958:


[~till.rohrmann] I'm afraid it doesn't work, because the aync io operator hang 
on the {{checkpointingLock.wait()}} not {{super.close()}} in {{close()}} 
method.  

> Async I/O timeout not work
> --
>
> Key: FLINK-6958
> URL: https://issues.apache.org/jira/browse/FLINK-6958
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: feng xiaojie
>Assignee: Jark Wu
>
> when use Async I/O with UnorderedStreamElementQueue, the queue will always 
> full if you don't  call the AsyncCollector.collect to ack them.
> Timeout shall collect these entries when the timeout trigger,but it isn't work
> I debug find,
> when time out, it will call resultFuture.completeExceptionally(error);
> but not call  UnorderedStreamElementQueue.onCompleteHandler
> it will cause that async i/o hang always



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4178: [FLINK-7000] Add custom configuration local environment i...

[ch33hau]

Github user ch33hau commented on the issue:

https://github.com/apache/flink/pull/4178
  
Hi @zentol ,

I have updated PR by adding a new method without default values.

Thanks =)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7000) Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062348#comment-16062348
 ] 

ASF GitHub Bot commented on FLINK-7000:
---

Github user ch33hau commented on the issue:

https://github.com/apache/flink/pull/4178
  
Hi @zentol ,

I have updated PR by adding a new method without default values.

Thanks =)


> Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment
> --
>
> Key: FLINK-7000
> URL: https://issues.apache.org/jira/browse/FLINK-7000
> Project: Flink
>  Issue Type: Improvement
>Reporter: Lim Chee Hau
>
> I was doing some local testings in {{Scala}} environment, however I found 
> that there is no straightforward way to add custom configuration to 
> {{StreamExecutionEnvironment}} by using {{createLocalEnvironment}} method. 
> This could be easily achieve in {{Java}} environment since 
> {{StreamExecutionEnvironment}} in {{Java}} has 
> - {{createLocalEnvironment()}}
> - {{createLocalEnvironment(Int)}}
> - {{createLocalEnvironment(Int, Configuration)}}
> Whereas Scala only has 2 out of 3 of these methods.
> Not sure if this is a missing implementation, if yes I could create a PR for 
> this.
> Therefore the example in [Local 
> Execution|https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/local_execution.html]
>  could be making sense for Scala users as well:
> bq. The LocalEnvironment allows also to pass custom configuration values to 
> Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6275) Unprotected access to resourceManager in YarnFlinkApplicationMasterRunner#runApplicationMaster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062346#comment-16062346
 ] 

ASF GitHub Bot commented on FLINK-6275:
---

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/3724
  
@zentol Sorry. I just forgot. I close it right now.


> Unprotected access to resourceManager in 
> YarnFlinkApplicationMasterRunner#runApplicationMaster
> --
>
> Key: FLINK-6275
> URL: https://issues.apache.org/jira/browse/FLINK-6275
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
>   // wait for resource manager to finish
>   resourceManager.getTerminationFuture().get();
> {code}
> The above is outside synchronized block.
> @GuardedBy indicates that access should be protected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3724: [FLINK-6275] [yarn] Fix unprotected access to reso...

[zhangminglei]

Github user zhangminglei closed the pull request at:

https://github.com/apache/flink/pull/3724


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6275) Unprotected access to resourceManager in YarnFlinkApplicationMasterRunner#runApplicationMaster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062347#comment-16062347
 ] 

ASF GitHub Bot commented on FLINK-6275:
---

Github user zhangminglei closed the pull request at:

https://github.com/apache/flink/pull/3724


> Unprotected access to resourceManager in 
> YarnFlinkApplicationMasterRunner#runApplicationMaster
> --
>
> Key: FLINK-6275
> URL: https://issues.apache.org/jira/browse/FLINK-6275
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
>   // wait for resource manager to finish
>   resourceManager.getTerminationFuture().get();
> {code}
> The above is outside synchronized block.
> @GuardedBy indicates that access should be protected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #3724: [FLINK-6275] [yarn] Fix unprotected access to resourceMan...

[zhangminglei]

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/3724
  
@zentol Sorry. I just forgot. I close it right now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-6362) Unprotected access to registeredStates in SharedStateRegistry#getReferenceCount

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved FLINK-6362.
---
Resolution: Cannot Reproduce

> Unprotected access to registeredStates in 
> SharedStateRegistry#getReferenceCount
> ---
>
> Key: FLINK-6362
> URL: https://issues.apache.org/jira/browse/FLINK-6362
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: Ted Yu
>Assignee: mingleizhang
>
> Here is related code:
> {code}
>   public int getReferenceCount(SharedStateHandle state) {
> if (state == null) {
>   return 0;
> }
> SharedStateRegistry.SharedStateEntry entry =
>   registeredStates.get(state.getRegistrationKey());
> {code}
> Access should be protected by holding lock on registeredStates



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (FLINK-6433) Unreachable code in SqlToRelConverter#visitCall()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved FLINK-6433.
---
Resolution: Won't Fix

> Unreachable code in SqlToRelConverter#visitCall()
> -
>
> Key: FLINK-6433
> URL: https://issues.apache.org/jira/browse/FLINK-6433
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
>   SqlFunction histogramOp = !ENABLE_HISTOGRAM_AGG
> ? null
> : getHistogramOp(aggOp);
>   if (histogramOp != null) {
> {code}
> Since ENABLE_HISTOGRAM_AGG is hardcoded as false, the if block wouldn't be 
> executed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6275) Unprotected access to resourceManager in YarnFlinkApplicationMasterRunner#runApplicationMaster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062344#comment-16062344
 ] 

ASF GitHub Bot commented on FLINK-6275:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3724
  
@zhangminglei could you close this PR?


> Unprotected access to resourceManager in 
> YarnFlinkApplicationMasterRunner#runApplicationMaster
> --
>
> Key: FLINK-6275
> URL: https://issues.apache.org/jira/browse/FLINK-6275
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
>   // wait for resource manager to finish
>   resourceManager.getTerminationFuture().get();
> {code}
> The above is outside synchronized block.
> @GuardedBy indicates that access should be protected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #3724: [FLINK-6275] [yarn] Fix unprotected access to resourceMan...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3724
  
@zhangminglei could you close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4178: [FLINK-7000] Add custom configuration local environment i...

[ch33hau]

Github user ch33hau commented on the issue:

https://github.com/apache/flink/pull/4178
  
Do you mean a new method with `createLocalEnvironment(Int, conf)`?

By this way compiler won't complain, just both the value have to be 
entered. 
However, I do agree that the way that you have suggested could allows user 
to pass in custom configuration without modifying public API

I could update the PR in a moment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7000) Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062342#comment-16062342
 ] 

ASF GitHub Bot commented on FLINK-7000:
---

Github user ch33hau commented on the issue:

https://github.com/apache/flink/pull/4178
  
Do you mean a new method with `createLocalEnvironment(Int, conf)`?

By this way compiler won't complain, just both the value have to be 
entered. 
However, I do agree that the way that you have suggested could allows user 
to pass in custom configuration without modifying public API

I could update the PR in a moment.


> Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment
> --
>
> Key: FLINK-7000
> URL: https://issues.apache.org/jira/browse/FLINK-7000
> Project: Flink
>  Issue Type: Improvement
>Reporter: Lim Chee Hau
>
> I was doing some local testings in {{Scala}} environment, however I found 
> that there is no straightforward way to add custom configuration to 
> {{StreamExecutionEnvironment}} by using {{createLocalEnvironment}} method. 
> This could be easily achieve in {{Java}} environment since 
> {{StreamExecutionEnvironment}} in {{Java}} has 
> - {{createLocalEnvironment()}}
> - {{createLocalEnvironment(Int)}}
> - {{createLocalEnvironment(Int, Configuration)}}
> Whereas Scala only has 2 out of 3 of these methods.
> Not sure if this is a missing implementation, if yes I could create a PR for 
> this.
> Therefore the example in [Local 
> Execution|https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/local_execution.html]
>  could be making sense for Scala users as well:
> bq. The LocalEnvironment allows also to pass custom configuration values to 
> Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062341#comment-16062341
 ] 

ASF GitHub Bot commented on FLINK-5892:
---

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/3844


> Recover job state at the granularity of operator
> 
>
> Key: FLINK-5892
> URL: https://issues.apache.org/jira/browse/FLINK-5892
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Reporter: Guowei Ma
>Assignee: Guowei Ma
> Fix For: 1.3.0
>
>
> JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the 
> granularity of task.
> This leads to the result that the operator of the job may not recover the 
> state from a save point even if the save point has the state of operator. 
>  
> https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3844: [FLINK-5892] Enable 1.2 keyed state test

[zentol]

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/3844


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7000) Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062339#comment-16062339
 ] 

ASF GitHub Bot commented on FLINK-7000:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4178
  
This method is part of the public API and thus must not be changed.

Does the compiler still complain if there is no default value for the 
configuration?


> Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment
> --
>
> Key: FLINK-7000
> URL: https://issues.apache.org/jira/browse/FLINK-7000
> Project: Flink
>  Issue Type: Improvement
>Reporter: Lim Chee Hau
>
> I was doing some local testings in {{Scala}} environment, however I found 
> that there is no straightforward way to add custom configuration to 
> {{StreamExecutionEnvironment}} by using {{createLocalEnvironment}} method. 
> This could be easily achieve in {{Java}} environment since 
> {{StreamExecutionEnvironment}} in {{Java}} has 
> - {{createLocalEnvironment()}}
> - {{createLocalEnvironment(Int)}}
> - {{createLocalEnvironment(Int, Configuration)}}
> Whereas Scala only has 2 out of 3 of these methods.
> Not sure if this is a missing implementation, if yes I could create a PR for 
> this.
> Therefore the example in [Local 
> Execution|https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/local_execution.html]
>  could be making sense for Scala users as well:
> bq. The LocalEnvironment allows also to pass custom configuration values to 
> Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6990) Poor performance with Sliding Time Windows

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062338#comment-16062338
 ] 

Jark Wu commented on FLINK-6990:


[~BriceBingman] [~fhueske] the poor performance of sliding time window is 
obvious in our production. We have planed to optimize it with window pane 
optimization. I have created a issue FLINK-7001 about it and will attach a more 
detail design doc on it. 

> Poor performance with Sliding Time Windows
> --
>
> Key: FLINK-6990
> URL: https://issues.apache.org/jira/browse/FLINK-6990
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API, Streaming
>Affects Versions: 1.3.0
> Environment: OSX 10.11.4
> 2.8 GHz Intel Core i7
> 16 GB 1600 MHz DDR3
>Reporter: Brice Bingman
>
> I'm experiencing poor performance when using sliding time windows.  Here is a 
> simple example that performs poorly for me:
> {code:java}
> public class FlinkPerfTest {
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment see = 
> StreamExecutionEnvironment.getExecutionEnvironment();
> //Streaming 10,000 events per second
> see.addSource(new SourceFunction() {
> transient ScheduledExecutorService executor;
> @Override
> public synchronized void run(final SourceContext ctx) 
> throws Exception {
> executor = Executors.newSingleThreadScheduledExecutor();
> executor.scheduleAtFixedRate(new Runnable() {
> @Override
> public void run() {
> for (int k = 0; k < 10; k++) {
> for (int i = 0; i < 1000; i++) {
> TestObject obj = new TestObject();
> obj.setKey(k);
> ctx.collect(obj);
> }
> }
> }
> }, 0, 1, TimeUnit.SECONDS);
> this.wait();
> }
> @Override
> public synchronized void cancel() {
> executor.shutdown();
> this.notify();
> }
> }).keyBy("key")
> .window(SlidingProcessingTimeWindows.of(Time.minutes(10), 
> Time.seconds(1))).apply(new WindowFunction TimeWindow>() {
> @Override
> public void apply(Tuple key, TimeWindow window, 
> Iterable input, Collector out) throws Exception {
> int count = 0;
> for (Object obj : input) {
> count++;
> }
> out.collect(key.getField(0) + ": " + count);
> }
> })
> .print();
> see.execute();
> }
> public static class TestObject {
> private Integer key;
> public Integer getKey() {
> return key;
> }
> public void setKey(Integer key) {
> this.key = key;
> }
> }
> }
> {code}
> When running this, flink periodically pauses for long periods of time.  I 
> would expect a steady stream of output at 1 second intervals.  For 
> comparison, you can switch to a count window of similar size which peforms 
> just fine:
> {code:java}
>.countWindow(60, 1000).apply(new 
> WindowFunction() {
> @Override
> public void apply(Tuple key, GlobalWindow window, 
> Iterable input, Collector out) throws Exception {
> int count = 0;
> for (Object obj : input) {
> count++;
> }
> out.collect(key.getField(0) + ": " + count);
> }
> })
> {code}
> I would expect the sliding time window to perform similarly to a count 
> window.  The sliding time window also uses significantly more cpu and memory 
> than the count window.  I would also expect resource consumption to be 
> similar.
> A possible cause could be that the SystemProcessingTimeService.TriggerTask is 
> locking with the checkpointLock which acts like a global lock.  There should 
> be a lock per key or preferably a lock-less solution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4178: [FLINK-7000] Add custom configuration local environment i...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4178
  
This method is part of the public API and thus must not be changed.

Does the compiler still complain if there is no default value for the 
configuration?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-7001) Improve performance of Sliding Time Window with pane optimization

[Jark Wu (JIRA)]

Jark Wu created FLINK-7001:
--

 Summary: Improve performance of Sliding Time Window with pane 
optimization
 Key: FLINK-7001
 URL: https://issues.apache.org/jira/browse/FLINK-7001
 Project: Flink
  Issue Type: Improvement
  Components: DataStream API
Reporter: Jark Wu
Assignee: Jark Wu
 Fix For: 1.4.0


Currently, the implementation of time-based sliding windows treats each window 
individually and replicates records to each window. For a window of 10 minute 
size that slides by 1 second the data is replicated 600 fold (10 minutes / 1 
second). We can optimize sliding window by divide windows into panes (aligned 
with slide), so that we can avoid record duplication and leverage the 
checkpoint.

I will attach a more detail design doc to the issue.

The following issues are similar to this issue: FLINK-5387, FLINK-6990



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6742) Improve error message when savepoint migration fails due to task removal

[Gyula Fora (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062308#comment-16062308
 ] 

Gyula Fora commented on FLINK-6742:
---

Ah sorry Chesnay I missed it on the 1.3 branch :/

> Improve error message when savepoint migration fails due to task removal
> 
>
> Key: FLINK-6742
> URL: https://issues.apache.org/jira/browse/FLINK-6742
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Gyula Fora
>Assignee: Chesnay Schepler
>Priority: Minor
>  Labels: flink-rel-1.3.1-blockers
>
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointV2.convertToOperatorStateSavepointV2(SavepointV2.java:171)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:75)
>   at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1090)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062305#comment-16062305
 ] 

Chesnay Schepler commented on FLINK-5892:
-

The KeyedComplexChainTest was updated to properly use a 1.2 snapshot (at the 
time of merging restoration of keyed state was broken).

1.3: 0fd3683c3489d98c5c82d21b9a7a5ee93c0d6b2e
1.4: 7a0fff31d483633f88026650956b97cc451f19df

> Recover job state at the granularity of operator
> 
>
> Key: FLINK-5892
> URL: https://issues.apache.org/jira/browse/FLINK-5892
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Reporter: Guowei Ma
>Assignee: Guowei Ma
> Fix For: 1.3.0
>
>
> JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the 
> granularity of task.
> This leads to the result that the operator of the job may not recover the 
> state from a save point even if the save point has the state of operator. 
>  
> https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6774) build-helper-maven-plugin version not set

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6774.
---
   Resolution: Fixed
Fix Version/s: 1.3.2
   1.4.0

1.3: b20ae53a9957683f3b7fe44c0a799422021fbb69
1.4: 1ae4ac28b102c1a33d071dd1184782eb12625bbc

> build-helper-maven-plugin version not set
> -
>
> Key: FLINK-6774
> URL: https://issues.apache.org/jira/browse/FLINK-6774
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Minor
> Fix For: 1.4.0, 1.3.2
>
>
> some modules forgot to specify the version of their 
> {{build-helper-maven-plugin}} which causes the following warning in a maven 
> build:
> {code}
> [WARNING] 'build.plugins.plugin.version' for 
> org.codehaus.mojo:build-helper-maven-plugin is missing. @ 
> org.apache.flink:flink-connector-kafka-base_${scala.binary.version}:[unknown-version],
>  
> /home/nico/Projects/flink/flink-connectors/flink-connector-kafka-base/pom.xml,
>  line 216, column 12
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6682) Improve error message in case parallelism exceeds maxParallelism

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6682.
---
   Resolution: Fixed
Fix Version/s: 1.3.2
   1.4.0

1.3: b1f762127234e323b947aa4a363935f87be1994f
1.4: c736ba2ef15e9e81a54a3fc02ccffadcbf594767

> Improve error message in case parallelism exceeds maxParallelism
> 
>
> Key: FLINK-6682
> URL: https://issues.apache.org/jira/browse/FLINK-6682
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Chesnay Schepler
>Assignee: mingleizhang
> Fix For: 1.4.0, 1.3.2
>
>
> When restoring a job with a parallelism that exceeds the maxParallelism we're 
> not providing a useful error message, as all you get is an 
> IllegalArgumentException:
> {code}
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
> at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:343)
> at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:396)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:467)
> ... 22 more
> Caused by: java.lang.IllegalArgumentException
> at 
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.createKeyGroupPartitions(StateAssignmentOperation.java:449)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignAttemptState(StateAssignmentOperation.java:117)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignStates(StateAssignmentOperation.java:102)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedState(CheckpointCoordinator.java:1038)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1101)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1386)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6541) Jar upload directory not created

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6541.
---
   Resolution: Fixed
Fix Version/s: 1.3.2
   1.4.0

1.3: ff2bb718cd9e87dfe7ab5601e9ff9f7293eb3cfe
1.4: 721640778aa88b6491baf837278a873493f283f0

> Jar upload directory not created
> 
>
> Key: FLINK-6541
> URL: https://issues.apache.org/jira/browse/FLINK-6541
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0, 1.4.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.4.0, 1.3.2
>
>
> Steps to reproduce:
> * setup configuration property: jobmanager.web.tmpdir = /mnt/flink/web
> * this directory should not exist
> * Run flink job manager.
> * in logs: 
> {code}
> 2017-05-11 12:07:58,397 ERROR 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils   - WebServer 
> could not be created [main]
> java.io.IOException: Jar upload directory 
> /mnt/flink/web/flink-web-3f2733c3-6f4c-4311-b617-1e93d9535421 cannot be 
> created or is not writable.
> {code}
> Expected:
> * create parent directories if they do not exit. i.e. use 
> "uploadDir.mkdirs()" instead of "uploadDir.mkdir()"
> Note:
> * BlobServer create parent directories (See BlobUtils storageDir.mkdirs())



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (FLINK-6682) Improve error message in case parallelism exceeds maxParallelism

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reopened FLINK-6682:
-

> Improve error message in case parallelism exceeds maxParallelism
> 
>
> Key: FLINK-6682
> URL: https://issues.apache.org/jira/browse/FLINK-6682
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Chesnay Schepler
>Assignee: mingleizhang
>
> When restoring a job with a parallelism that exceeds the maxParallelism we're 
> not providing a useful error message, as all you get is an 
> IllegalArgumentException:
> {code}
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
> at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:343)
> at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:396)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:467)
> ... 22 more
> Caused by: java.lang.IllegalArgumentException
> at 
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.createKeyGroupPartitions(StateAssignmentOperation.java:449)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignAttemptState(StateAssignmentOperation.java:117)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignStates(StateAssignmentOperation.java:102)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedState(CheckpointCoordinator.java:1038)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1101)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1386)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6784) Add some notes about externalized checkpoints and the difference to savepoints

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6784.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.3: c815ada2b68fd0692b1ccd77c4bcff2a3e9175ea
1.4: e1269ed8fa2d66978e2816697d6e3fe3dfabf7f6

> Add some notes about externalized checkpoints and the difference to savepoints
> --
>
> Key: FLINK-6784
> URL: https://issues.apache.org/jira/browse/FLINK-6784
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.4.0, 1.3.0
>
>
> while externalized checkpoints are described somehow, there does not seem to 
> be any paragraph explaining the difference to savepoints, also there are two 
> checkpointing docs which could at least be linked somehow



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6798) Remove/update documentation about network buffer tuning

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6798.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.3: 31fd58258b1346f3c9adcf397ffdec316333a4bd
1.4: d713ed154240c1678505b22ae260bdc8b7b652d7

> Remove/update documentation about network buffer tuning
> ---
>
> Key: FLINK-6798
> URL: https://issues.apache.org/jira/browse/FLINK-6798
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Network
>Reporter: Robert Metzger
>Assignee: Nico Kruber
> Fix For: 1.4.0, 1.3.0
>
>
> {quote}The number of network buffers is a parameter that can currently have 
> an effect on checkpointing at large scale. The Flink community is working on 
> eliminating that parameter in the next versions of Flink.
> {quote} 
> in 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/large_state_tuning.html#tuning-network-buffers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6782) Update savepoint documentation

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6782.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.3: bfba2b1d453266f28523c47682d3cfe5ae67ce74
1.4: 60222a24123379bbbd044a6ca25817e64a9348b9

> Update savepoint documentation
> --
>
> Key: FLINK-6782
> URL: https://issues.apache.org/jira/browse/FLINK-6782
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.4.0, 1.3.0
>
>
> Savepoint documentation is a bit outdated regarding full data being stored in 
> the savepoint path, not just a metadata file



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6742) Improve error message when savepoint migration fails due to task removal

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062297#comment-16062297
 ] 

Chesnay Schepler commented on FLINK-6742:
-

1.3: 2bbfe0292c13d875b531a6c168ea78bfc7f21f0b
1.4: 72b0ae069f8404a2f8a952e1a20004b9d340c445

> Improve error message when savepoint migration fails due to task removal
> 
>
> Key: FLINK-6742
> URL: https://issues.apache.org/jira/browse/FLINK-6742
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Gyula Fora
>Assignee: Chesnay Schepler
>Priority: Minor
>  Labels: flink-rel-1.3.1-blockers
>
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointV2.convertToOperatorStateSavepointV2(SavepointV2.java:171)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:75)
>   at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1090)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4180: Merge PR for 1.3

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4180


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-6965) Avro is missing snappy dependency

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-6965:

Priority: Blocker  (was: Major)

> Avro is missing snappy dependency
> -
>
> Key: FLINK-6965
> URL: https://issues.apache.org/jira/browse/FLINK-6965
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.3.2
>
>
> The shading rework made before 1.3 removed a snappy dependency that was 
> accidentally pulled in through hadoop. This is technically alright, until 
> class-loaders rear their ugly heads.
> Our kafka connector can read avro records, which may or may not require 
> snappy. Usually this _should_ be solvable by including the snappy dependency 
> in the user-jar if necessary, however since the kafka connector loads classes 
> that it requires using the system class loader this doesn't work.
> As such we have to add a separate snappy dependency to flink-core.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-6843) ClientConnectionTest fails on travis

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-6843:

Priority: Blocker  (was: Major)

> ClientConnectionTest fails on travis
> 
>
> Key: FLINK-6843
> URL: https://issues.apache.org/jira/browse/FLINK-6843
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.3.0
>Reporter: Chesnay Schepler
>Priority: Blocker
>
> jdk7, hadoop 2.4.1, scala 2.11
> {code}
> testJobManagerRetrievalWithHAServices(org.apache.flink.client.program.ClientConnectionTest)
>   Time elapsed: 0.013 sec  <<< ERROR!
> java.lang.UnsupportedClassVersionError: 
> org/apache/flink/runtime/highavailability/TestingHighAvailabilityServices : 
> Unsupported major.minor version 52.0
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   at 
> org.apache.flink.client.program.ClientConnectionTest.testJobManagerRetrievalWithHAServices(ClientConnectionTest.java:122)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6999) Print TravisCI CPU, memory, and filesystem info

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-6999.
-
Resolution: Implemented

master: a13a98f6570fb8648e3780e90407c976178aff7d

> Print TravisCI CPU, memory, and filesystem info
> ---
>
> Key: FLINK-6999
> URL: https://issues.apache.org/jira/browse/FLINK-6999
> Project: Flink
>  Issue Type: New Feature
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.4.0
>
>
> Print and fold the output from {{lscpu}}, {{/proc/meminfo}}, and {{df}} in 
> the TravisCI logs.
> For help investigating changes such as the recent doubling of and highly 
> variable job times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6999) Print TravisCI CPU, memory, and filesystem info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062287#comment-16062287
 ] 

ASF GitHub Bot commented on FLINK-6999:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4177


> Print TravisCI CPU, memory, and filesystem info
> ---
>
> Key: FLINK-6999
> URL: https://issues.apache.org/jira/browse/FLINK-6999
> Project: Flink
>  Issue Type: New Feature
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.4.0
>
>
> Print and fold the output from {{lscpu}}, {{/proc/meminfo}}, and {{df}} in 
> the TravisCI logs.
> For help investigating changes such as the recent doubling of and highly 
> variable job times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4177: [FLINK-6999] [build] Print TravisCI CPU, memory, a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4177


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6994) Wrong base url in master docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062286#comment-16062286
 ] 

ASF GitHub Bot commented on FLINK-6994:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/4175
  
Why do we have both `flink-docs-master` and `flink-docs-release-1.4`? The 
main page only links to 1.3 and 1.4 but the flink-docs-master documentation 
links to 1.3, 1.2, 1.1, and 1.0. But each of those packages only links to older 
releases so if a user arrives on a 1.2 page they won't see 1.3.


> Wrong base url in master docs
> -
>
> Key: FLINK-6994
> URL: https://issues.apache.org/jira/browse/FLINK-6994
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> The base url of the master docs point to 1.3 instead of 1.4. At the moment 
> the menu items point to the latest stable release docs instead of the nightly 
> master docs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4175: [FLINK-6994] [docs] Wrong base url in master docs

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/4175
  
Why do we have both `flink-docs-master` and `flink-docs-release-1.4`? The 
main page only links to 1.3 and 1.4 but the flink-docs-master documentation 
links to 1.3, 1.2, 1.1, and 1.0. But each of those packages only links to older 
releases so if a user arrives on a 1.2 page they won't see 1.3.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6742) Improve error message when savepoint migration fails due to task removal

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062282#comment-16062282
 ] 

ASF GitHub Bot commented on FLINK-6742:
---

Github user gyfora commented on a diff in the pull request:

https://github.com/apache/flink/pull/4083#discussion_r123895607
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointV2.java
 ---
@@ -168,10 +168,27 @@ public static Savepoint 
convertToOperatorStateSavepointV2(
expandedToLegacyIds = true;
}
 
+   if (jobVertex == null) {
+   throw new IllegalStateException(
+   "Could not find task for state with ID 
" + taskState.getJobVertexID() + ". " +
+   "When migrating a savepoint from a 
version < 1.3 please make sure that the topology was not " +
+   "changed through removal of a stateful 
operator or modification of a chain containing a stateful " +
+   "operator.");
+   }
+
List operatorIDs = 
jobVertex.getOperatorIDs();
 
for (int subtaskIndex = 0; subtaskIndex < 
jobVertex.getParallelism(); subtaskIndex++) {
-   SubtaskState subtaskState = 
taskState.getState(subtaskIndex);
+   SubtaskState subtaskState;
+   try {
+   subtaskState = 
taskState.getState(subtaskIndex);
--- End diff --

Sorry for commenting late on this but I have had some major migration 
issues in the last few days :D 
I think we should explicitly compare parallelism instead of relying on the 
error:
if (taskState.getStates().size() != jobVertex.getParallelism()) --> error

Otherwise this will not fail on lower parallelism.


> Improve error message when savepoint migration fails due to task removal
> 
>
> Key: FLINK-6742
> URL: https://issues.apache.org/jira/browse/FLINK-6742
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Gyula Fora
>Assignee: Chesnay Schepler
>Priority: Minor
>  Labels: flink-rel-1.3.1-blockers
>
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointV2.convertToOperatorStateSavepointV2(SavepointV2.java:171)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:75)
>   at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1090)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4083: [FLINK-6742] Improve savepoint migration failure e...

[gyfora]

Github user gyfora commented on a diff in the pull request:

https://github.com/apache/flink/pull/4083#discussion_r123895607
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointV2.java
 ---
@@ -168,10 +168,27 @@ public static Savepoint 
convertToOperatorStateSavepointV2(
expandedToLegacyIds = true;
}
 
+   if (jobVertex == null) {
+   throw new IllegalStateException(
+   "Could not find task for state with ID 
" + taskState.getJobVertexID() + ". " +
+   "When migrating a savepoint from a 
version < 1.3 please make sure that the topology was not " +
+   "changed through removal of a stateful 
operator or modification of a chain containing a stateful " +
+   "operator.");
+   }
+
List operatorIDs = 
jobVertex.getOperatorIDs();
 
for (int subtaskIndex = 0; subtaskIndex < 
jobVertex.getParallelism(); subtaskIndex++) {
-   SubtaskState subtaskState = 
taskState.getState(subtaskIndex);
+   SubtaskState subtaskState;
+   try {
+   subtaskState = 
taskState.getState(subtaskIndex);
--- End diff --

Sorry for commenting late on this but I have had some major migration 
issues in the last few days :D 
I think we should explicitly compare parallelism instead of relying on the 
error:
if (taskState.getStates().size() != jobVertex.getParallelism()) --> error

Otherwise this will not fail on lower parallelism.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5595) Add links to sub-sections in the left-hand navigation bar

[Mike Winters (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062270#comment-16062270
 ] 

Mike Winters commented on FLINK-5595:
-

Yes, please do! Thanks [~nssalian]

> Add links to sub-sections in the left-hand navigation bar
> -
>
> Key: FLINK-5595
> URL: https://issues.apache.org/jira/browse/FLINK-5595
> Project: Flink
>  Issue Type: Improvement
>  Components: Project Website
>Reporter: Mike Winters
>Assignee: Mike Winters
>Priority: Minor
>  Labels: newbie, website
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Some pages on the Flink project site (such as 
> http://flink.apache.org/introduction.html) include a table of contents at the 
> top. The sections from the ToC are not exposed in the left-hand nav when the 
> page is active, but this could be a useful addition, especially for longer, 
> content-heavy pages. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7000) Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062263#comment-16062263
 ] 

ASF GitHub Bot commented on FLINK-7000:
---

Github user ch33hau commented on the issue:

https://github.com/apache/flink/pull/4178
  
Hi @zentol , thanks for taking time for reviewing :)

It was my original thought. 
However turns out due to `Scala` supports `default value` in parameters, in 
order to achieve same behaviour as Java, adding a new method is not working 
here.

 This is what currently Java has:

1. createLocalEnvironment()
2. createLocalEnvironment(Int)
3. createLocalEnvironment(Int, Configuration)

Before this PR, in Scala we have:

createLocalEnvironment(Int = defaultValue)

So a Scala user can access this method by `createLocalEnvironment()` or 
`createLocalEnvironment(someValue)`.

If I add another method `createLocalEnvironment(Int = defaultValue, conf = 
defaultValue)`, compiler will complain because at this moment 
`createLocalEnvironment(someValue)` is ambiguous.

My opinion is by just adding a parameter with default value to existing 
method, it doesn't change the existing way to access the method. 

How do you think?


> Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment
> --
>
> Key: FLINK-7000
> URL: https://issues.apache.org/jira/browse/FLINK-7000
> Project: Flink
>  Issue Type: Improvement
>Reporter: Lim Chee Hau
>
> I was doing some local testings in {{Scala}} environment, however I found 
> that there is no straightforward way to add custom configuration to 
> {{StreamExecutionEnvironment}} by using {{createLocalEnvironment}} method. 
> This could be easily achieve in {{Java}} environment since 
> {{StreamExecutionEnvironment}} in {{Java}} has 
> - {{createLocalEnvironment()}}
> - {{createLocalEnvironment(Int)}}
> - {{createLocalEnvironment(Int, Configuration)}}
> Whereas Scala only has 2 out of 3 of these methods.
> Not sure if this is a missing implementation, if yes I could create a PR for 
> this.
> Therefore the example in [Local 
> Execution|https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/local_execution.html]
>  could be making sense for Scala users as well:
> bq. The LocalEnvironment allows also to pass custom configuration values to 
> Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4178: [FLINK-7000] Add custom configuration local environment i...

[ch33hau]

Github user ch33hau commented on the issue:

https://github.com/apache/flink/pull/4178
  
Hi @zentol , thanks for taking time for reviewing :)

It was my original thought. 
However turns out due to `Scala` supports `default value` in parameters, in 
order to achieve same behaviour as Java, adding a new method is not working 
here.

 This is what currently Java has:

1. createLocalEnvironment()
2. createLocalEnvironment(Int)
3. createLocalEnvironment(Int, Configuration)

Before this PR, in Scala we have:

createLocalEnvironment(Int = defaultValue)

So a Scala user can access this method by `createLocalEnvironment()` or 
`createLocalEnvironment(someValue)`.

If I add another method `createLocalEnvironment(Int = defaultValue, conf = 
defaultValue)`, compiler will complain because at this moment 
`createLocalEnvironment(someValue)` is ambiguous.

My opinion is by just adding a parameter with default value to existing 
method, it doesn't change the existing way to access the method. 

How do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6997) SavepointITCase fails in master branch sometimes

[Shashank Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062257#comment-16062257
 ] 

Shashank Agarwal commented on FLINK-6997:
-

I have tried savepoint on release-1.3 getting same error.
{code}
flink cancel -s hdfs:///fl/savepoint-130 f6bfa9e01be030d5d144a6aa680ff3ed
{code}

I am getting error
{code}
The program finished with the following exception:

java.lang.Exception: Canceling the job with ID f6bfa9e01be030d5d144a6aa680ff3ed 
failed.
at org.apache.flink.client.CliFrontend.cancel(CliFrontend.java:637)
at 
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1092)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
Caused by: java.lang.Exception: Failed to trigger savepoint.
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$6.apply(JobManager.scala:639)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$6.apply(JobManager.scala:629)
at 
org.apache.flink.runtime.concurrent.impl.FlinkFuture$5.onComplete(FlinkFuture.java:272)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:245)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at 
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at 
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at 
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Failed to trigger savepoint: Not all required 
tasks are currently running.
at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.triggerSavepoint(CheckpointCoordinator.java:382)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:625)
at 
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at 
org.apache.flink.runtime.clusterframework.ContaineredJobManager$$anonfun$handleContainerMessage$1.applyOrElse(ContaineredJobManager.scala:100)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at 
org.apache.flink.yarn.YarnJobManager$$anonfun$handleYarnShutdown$1.applyOrElse(YarnJobManager.scala:103)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at 
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
at 
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at 
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at 
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:125)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at 

[jira] [Closed] (FLINK-6682) Improve error message in case parallelism exceeds maxParallelism

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang closed FLINK-6682.
---
Resolution: Fixed

> Improve error message in case parallelism exceeds maxParallelism
> 
>
> Key: FLINK-6682
> URL: https://issues.apache.org/jira/browse/FLINK-6682
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Chesnay Schepler
>Assignee: mingleizhang
>
> When restoring a job with a parallelism that exceeds the maxParallelism we're 
> not providing a useful error message, as all you get is an 
> IllegalArgumentException:
> {code}
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
> at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:343)
> at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:396)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:467)
> ... 22 more
> Caused by: java.lang.IllegalArgumentException
> at 
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.createKeyGroupPartitions(StateAssignmentOperation.java:449)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignAttemptState(StateAssignmentOperation.java:117)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignStates(StateAssignmentOperation.java:102)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedState(CheckpointCoordinator.java:1038)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1101)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1386)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6999) Print TravisCI CPU, memory, and filesystem info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062254#comment-16062254
 ] 

ASF GitHub Bot commented on FLINK-6999:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4177
  
As for printing it for each module, this should be easy if we move the code 
into a separate script and use the ´exec-maven-plugin` or smth.


> Print TravisCI CPU, memory, and filesystem info
> ---
>
> Key: FLINK-6999
> URL: https://issues.apache.org/jira/browse/FLINK-6999
> Project: Flink
>  Issue Type: New Feature
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.4.0
>
>
> Print and fold the output from {{lscpu}}, {{/proc/meminfo}}, and {{df}} in 
> the TravisCI logs.
> For help investigating changes such as the recent doubling of and highly 
> variable job times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6999) Print TravisCI CPU, memory, and filesystem info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062255#comment-16062255
 ] 

ASF GitHub Bot commented on FLINK-6999:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4177
  
We can put the plugin into a separate "travis" profile.


> Print TravisCI CPU, memory, and filesystem info
> ---
>
> Key: FLINK-6999
> URL: https://issues.apache.org/jira/browse/FLINK-6999
> Project: Flink
>  Issue Type: New Feature
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.4.0
>
>
> Print and fold the output from {{lscpu}}, {{/proc/meminfo}}, and {{df}} in 
> the TravisCI logs.
> For help investigating changes such as the recent doubling of and highly 
> variable job times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4177: [FLINK-6999] [build] Print TravisCI CPU, memory, and file...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4177
  
We can put the plugin into a separate "travis" profile.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4177: [FLINK-6999] [build] Print TravisCI CPU, memory, and file...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4177
  
As for printing it for each module, this should be easy if we move the code 
into a separate script and use the ´exec-maven-plugin` or smth.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7000) Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062253#comment-16062253
 ] 

ASF GitHub Bot commented on FLINK-7000:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4178
  
you're modifying an existing method, didn't you mean to _add_ a new one?


> Add custom configuration for StreamExecutionEnvironment#createLocalEnvironment
> --
>
> Key: FLINK-7000
> URL: https://issues.apache.org/jira/browse/FLINK-7000
> Project: Flink
>  Issue Type: Improvement
>Reporter: Lim Chee Hau
>
> I was doing some local testings in {{Scala}} environment, however I found 
> that there is no straightforward way to add custom configuration to 
> {{StreamExecutionEnvironment}} by using {{createLocalEnvironment}} method. 
> This could be easily achieve in {{Java}} environment since 
> {{StreamExecutionEnvironment}} in {{Java}} has 
> - {{createLocalEnvironment()}}
> - {{createLocalEnvironment(Int)}}
> - {{createLocalEnvironment(Int, Configuration)}}
> Whereas Scala only has 2 out of 3 of these methods.
> Not sure if this is a missing implementation, if yes I could create a PR for 
> this.
> Therefore the example in [Local 
> Execution|https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/local_execution.html]
>  could be making sense for Scala users as well:
> bq. The LocalEnvironment allows also to pass custom configuration values to 
> Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4178: [FLINK-7000] Add custom configuration local environment i...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4178
  
you're modifying an existing method, didn't you mean to _add_ a new one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6999) Print TravisCI CPU, memory, and filesystem info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062252#comment-16062252
 ] 

ASF GitHub Bot commented on FLINK-6999:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4177
  
cool stuff, +1.


> Print TravisCI CPU, memory, and filesystem info
> ---
>
> Key: FLINK-6999
> URL: https://issues.apache.org/jira/browse/FLINK-6999
> Project: Flink
>  Issue Type: New Feature
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.4.0
>
>
> Print and fold the output from {{lscpu}}, {{/proc/meminfo}}, and {{df}} in 
> the TravisCI logs.
> For help investigating changes such as the recent doubling of and highly 
> variable job times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4177: [FLINK-6999] [build] Print TravisCI CPU, memory, and file...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4177
  
cool stuff, +1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4180: Merge PR for 1.3

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/4180

Merge PR for 1.3

Please ignore.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink m3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4180.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4180


commit ff2bb718cd9e87dfe7ab5601e9ff9f7293eb3cfe
Author: zentol 
Date:   2017-05-13T16:09:30Z

[FLINK-6541] Improve tmp dir setup in TM/WebMonitor

commit b954eda8763dd030b0325e0e001aa930f64ef72f
Author: Nico Kruber 
Date:   2017-05-31T08:50:39Z

[hotfix][docs] update Checkpoint docs with correct code example

commit bfba2b1d453266f28523c47682d3cfe5ae67ce74
Author: Nico Kruber 
Date:   2017-05-31T11:51:12Z

[FLINK-6782][docs] update snapshot documentation to reflect flink 1.3

This closes #4024.

commit c815ada2b68fd0692b1ccd77c4bcff2a3e9175ea
Author: Nico Kruber 
Date:   2017-05-31T14:22:22Z

[FLINK-6784][docs] update externalized checkpoints documentation

This closes #4033.

commit 31fd58258b1346f3c9adcf397ffdec316333a4bd
Author: Nico Kruber 
Date:   2017-06-06T12:24:14Z

[FLINK-6798][docs] update old network buffer notices

This closes #4080.

commit 2bbfe0292c13d875b531a6c168ea78bfc7f21f0b
Author: zentol 
Date:   2017-06-07T10:03:21Z

[FLINK-6742] Improve savepoint migration failure error message

commit 3d5cee2dcc6b764556f430404b5b7936ee1cd009
Author: zentol 
Date:   2017-06-07T10:03:57Z

[hotfix] Improve readability in SPV2#convertToOperatorStateSavepointV2

commit b1f762127234e323b947aa4a363935f87be1994f
Author: zhangminglei 
Date:   2017-06-20T11:43:44Z

[FLINK-6682] [checkpoints] Improve error message in case parallelism 
exceeds maxParallelism

This closes #4125.

commit b20ae53a9957683f3b7fe44c0a799422021fbb69
Author: Nico Kruber 
Date:   2017-05-30T12:52:23Z

[FLINK-6774][build] set missing build-helper-maven-plugin version

This closes #4017.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-6943) Improve exceptions within TypeExtractionUtils#getSingleAbstractMethod

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6943.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.4: 2274bf74f0ac84fdcf6a6109426ada3c61b4454e

> Improve exceptions within TypeExtractionUtils#getSingleAbstractMethod
> -
>
> Key: FLINK-6943
> URL: https://issues.apache.org/jira/browse/FLINK-6943
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
> Fix For: 1.4.0
>
>
> Exception message seems to be inexact. 
> Also if there is no SAM, sam would be null upon returning from the method.
> The suggestion from a review was to change the message and add a check (for 
> null sam) prior to returning.
> Another suggestion is to check if the given method is an interface, as only 
> for interface it is possible to pass lambda.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6389) Upgrade hbase dependency to 1.3.1

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6389.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.4: 5604930c010b32200facc4c86dfb26c64120349e

> Upgrade hbase dependency to 1.3.1
> -
>
> Key: FLINK-6389
> URL: https://issues.apache.org/jira/browse/FLINK-6389
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
> Fix For: 1.4.0
>
>
> hbase 1.3.1 has been released.
> It fixes compatibility issue in 1.3.0 release, among other bug fixes.
> We should upgrade to hbase 1.3.1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6786) Remove duplicate QueryScopeInfoTest

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6786.
---
Resolution: Fixed

1.4: b863b90c80a63e950106b4460bb81f733819dbc2

> Remove duplicate QueryScopeInfoTest
> ---
>
> Key: FLINK-6786
> URL: https://issues.apache.org/jira/browse/FLINK-6786
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics, Tests
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> The {{QueryScopeInfoTest}} exists twice in {{runtime/metrics}}, under 
> {{groups/}} and {{dump/}}.
> These should be merged together.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6956) Table examples broken

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6956.
---
Resolution: Fixed

1.4: 1f6be8372706c1531844857291bc166bf091176b

> Table examples broken
> -
>
> Key: FLINK-6956
> URL: https://issues.apache.org/jira/browse/FLINK-6956
> Project: Flink
>  Issue Type: Bug
>  Components: Examples, Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> When running the examples you get this exception:
> {code}
> Caused by: org.apache.flink.table.api.TableException: Class 'class 
> org.apache.flink.table.examples.java.WordCountSQL$WC' described in type 
> information 
> 'GenericType' must be 
> static and globally accessible.
> at 
> org.apache.flink.table.api.TableException$.apply(exceptions.scala:53)
> at 
> org.apache.flink.table.api.TableEnvironment$.validateType(TableEnvironment.scala:936)
> at 
> org.apache.flink.table.api.TableEnvironment.getFieldInfo(TableEnvironment.scala:616)
> at 
> org.apache.flink.table.api.BatchTableEnvironment.registerDataSetInternal(BatchTableEnvironment.scala:248)
> at 
> org.apache.flink.table.api.java.BatchTableEnvironment.registerDataSet(BatchTableEnvironment.scala:129)
> at 
> org.apache.flink.table.examples.java.WordCountSQL.main(WordCountSQL.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
> ... 13 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6967) Fully separate batch and storm examples

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6967.
---
Resolution: Fixed

1.4: 9f77073a56cccf06399dae24de1c38dc27dafcb6

> Fully separate batch and storm examples
> ---
>
> Key: FLINK-6967
> URL: https://issues.apache.org/jira/browse/FLINK-6967
> Project: Flink
>  Issue Type: Improvement
>  Components: Examples, Storm Compatibility
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.4.0
>
>
> Like the streaming examples (see FLINK-6863) the storm examples have a 
> dependency on the batch examples, exclusively for the WordCount example data.
> I propose to duplicate the test data again for the storm examples.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6788) Remove unused GenericFlatTypePostPass/AbstractSchema class

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6788.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.4: af0ced984ba9f13302f1d4c4d4b064bd580cc29c

> Remove unused GenericFlatTypePostPass/AbstractSchema class
> --
>
> Key: FLINK-6788
> URL: https://issues.apache.org/jira/browse/FLINK-6788
> Project: Flink
>  Issue Type: Improvement
>  Components: Optimizer
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Miao Wang
>Priority: Trivial
> Fix For: 1.4.0
>
>
> The {{AbstractSchema}} and {{GenericFlatTypePostPass}} classes in 
> {{org.apache.flink.optimizer.postpass}} are unused and could maybe be removed.
> [~fhueske] your thoughts?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6488) Mark deprecated for 'start-local.sh' and 'stop-local' scripts

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6488.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.4: 3557ff8636e899d56ba23534944669aeca2b5df4

> Mark deprecated for 'start-local.sh' and 'stop-local' scripts
> -
>
> Key: FLINK-6488
> URL: https://issues.apache.org/jira/browse/FLINK-6488
> Project: Flink
>  Issue Type: Sub-task
>  Components: Startup Shell Scripts
>Reporter: Stephan Ewen
>Assignee: mingleizhang
> Fix For: 1.4.0
>
>
> The {{start-cluster.sh}} scripts work locally now, without needing SSH setup.
> We can remove {{start-local.sh}} without any loss of functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6769) Replace usage of deprecated FileSystem#create(Path, boolean)

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6769.
---
Resolution: Fixed

1.4: d7bf7eebed29f35c797bd3e21bafd0eb071f9f34

> Replace usage of deprecated FileSystem#create(Path, boolean)
> 
>
> Key: FLINK-6769
> URL: https://issues.apache.org/jira/browse/FLINK-6769
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6863) Fully separate streaming examples

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6863.
---
Resolution: Fixed

1.4: 387dc4b992fc0ea3b728ee343c3a052bc746e3a3

> Fully separate streaming examples
> ---
>
> Key: FLINK-6863
> URL: https://issues.apache.org/jira/browse/FLINK-6863
> Project: Flink
>  Issue Type: Improvement
>  Components: Examples
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> The streaming examples module currently depends on the batch examples, 
> specifically they re-use {{WordCountData}} for {{PojoExample}}, 
> {{SideOutputExample}}, {{WordCount}} and {{WindowWordCount}}.
> I propose simply copying the example data to make the module more 
> self-contained.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6774) build-helper-maven-plugin version not set

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062242#comment-16062242
 ] 

ASF GitHub Bot commented on FLINK-6774:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4017


> build-helper-maven-plugin version not set
> -
>
> Key: FLINK-6774
> URL: https://issues.apache.org/jira/browse/FLINK-6774
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Minor
>
> some modules forgot to specify the version of their 
> {{build-helper-maven-plugin}} which causes the following warning in a maven 
> build:
> {code}
> [WARNING] 'build.plugins.plugin.version' for 
> org.codehaus.mojo:build-helper-maven-plugin is missing. @ 
> org.apache.flink:flink-connector-kafka-base_${scala.binary.version}:[unknown-version],
>  
> /home/nico/Projects/flink/flink-connectors/flink-connector-kafka-base/pom.xml,
>  line 216, column 12
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6488) Mark deprecated for 'start-local.sh' and 'stop-local' scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062239#comment-16062239
 ] 

ASF GitHub Bot commented on FLINK-6488:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4074


> Mark deprecated for 'start-local.sh' and 'stop-local' scripts
> -
>
> Key: FLINK-6488
> URL: https://issues.apache.org/jira/browse/FLINK-6488
> Project: Flink
>  Issue Type: Sub-task
>  Components: Startup Shell Scripts
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>
> The {{start-cluster.sh}} scripts work locally now, without needing SSH setup.
> We can remove {{start-local.sh}} without any loss of functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4074: [FLINK-6488] [scripts] Mark deprecated 'start-loca...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4074


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6769) Replace usage of deprecated FileSystem#create(Path, boolean)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062227#comment-16062227
 ] 

ASF GitHub Bot commented on FLINK-6769:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4116


> Replace usage of deprecated FileSystem#create(Path, boolean)
> 
>
> Key: FLINK-6769
> URL: https://issues.apache.org/jira/browse/FLINK-6769
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6782) Update savepoint documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062231#comment-16062231
 ] 

ASF GitHub Bot commented on FLINK-6782:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4024


> Update savepoint documentation
> --
>
> Key: FLINK-6782
> URL: https://issues.apache.org/jira/browse/FLINK-6782
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.3.0
>
>
> Savepoint documentation is a bit outdated regarding full data being stored in 
> the savepoint path, not just a metadata file



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6742) Improve error message when savepoint migration fails due to task removal

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062229#comment-16062229
 ] 

ASF GitHub Bot commented on FLINK-6742:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4083


> Improve error message when savepoint migration fails due to task removal
> 
>
> Key: FLINK-6742
> URL: https://issues.apache.org/jira/browse/FLINK-6742
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Gyula Fora
>Assignee: Chesnay Schepler
>Priority: Minor
>  Labels: flink-rel-1.3.1-blockers
>
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointV2.convertToOperatorStateSavepointV2(SavepointV2.java:171)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:75)
>   at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1090)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4159: [FLINK-6967] Remove batch-examples dependency from...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4159


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6682) Improve error message in case parallelism exceeds maxParallelism

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062235#comment-16062235
 ] 

ASF GitHub Bot commented on FLINK-6682:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4125


> Improve error message in case parallelism exceeds maxParallelism
> 
>
> Key: FLINK-6682
> URL: https://issues.apache.org/jira/browse/FLINK-6682
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Chesnay Schepler
>Assignee: mingleizhang
>
> When restoring a job with a parallelism that exceeds the maxParallelism we're 
> not providing a useful error message, as all you get is an 
> IllegalArgumentException:
> {code}
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
> at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:343)
> at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:396)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:467)
> ... 22 more
> Caused by: java.lang.IllegalArgumentException
> at 
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.createKeyGroupPartitions(StateAssignmentOperation.java:449)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignAttemptState(StateAssignmentOperation.java:117)
> at 
> org.apache.flink.runtime.checkpoint.StateAssignmentOperation.assignStates(StateAssignmentOperation.java:102)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedState(CheckpointCoordinator.java:1038)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1101)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1386)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6956) Table examples broken

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062241#comment-16062241
 ] 

ASF GitHub Bot commented on FLINK-6956:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4148


> Table examples broken
> -
>
> Key: FLINK-6956
> URL: https://issues.apache.org/jira/browse/FLINK-6956
> Project: Flink
>  Issue Type: Bug
>  Components: Examples, Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> When running the examples you get this exception:
> {code}
> Caused by: org.apache.flink.table.api.TableException: Class 'class 
> org.apache.flink.table.examples.java.WordCountSQL$WC' described in type 
> information 
> 'GenericType' must be 
> static and globally accessible.
> at 
> org.apache.flink.table.api.TableException$.apply(exceptions.scala:53)
> at 
> org.apache.flink.table.api.TableEnvironment$.validateType(TableEnvironment.scala:936)
> at 
> org.apache.flink.table.api.TableEnvironment.getFieldInfo(TableEnvironment.scala:616)
> at 
> org.apache.flink.table.api.BatchTableEnvironment.registerDataSetInternal(BatchTableEnvironment.scala:248)
> at 
> org.apache.flink.table.api.java.BatchTableEnvironment.registerDataSet(BatchTableEnvironment.scala:129)
> at 
> org.apache.flink.table.examples.java.WordCountSQL.main(WordCountSQL.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
> ... 13 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4024: [FLINK-6782][docs] update snapshot documentation t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4024


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4017: [FLINK-6774][build] set missing build-helper-maven...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4017


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---