[jira] [Commented] (FLINK-6328) Savepoints must not be counted as retained checkpoints

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020682#comment-16020682
 ] 

ASF GitHub Bot commented on FLINK-6328:
---

Github user tony810430 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3965#discussion_r117902054
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
 ---
@@ -864,22 +864,28 @@ private void 
completePendingCheckpoint(PendingCheckpoint pendingCheckpoint) thro
// the pending checkpoint must be discarded after the 
finalization

Preconditions.checkState(pendingCheckpoint.isDiscarded() && completedCheckpoint 
!= null);
 
-   try {
-   
completedCheckpointStore.addCheckpoint(completedCheckpoint);
-   } catch (Exception exception) {
-   // we failed to store the completed checkpoint. 
Let's clean up
-   executor.execute(new Runnable() {
-   @Override
-   public void run() {
-   try {
-   
completedCheckpoint.discardOnFailedStoring();
-   } catch (Throwable t) {
-   LOG.warn("Could not 
properly discard completed checkpoint {}.", 
completedCheckpoint.getCheckpointID(), t);
+   // TODO: add savepoints to completed checkpoint store 
once FLINK-4815 has been completed
+   if 
(!CheckpointProperties.isSavepoint(completedCheckpoint.getProperties())) {
--- End diff --

Why not use `completedCheckpoint.getProperties().isSavepoint()` to check 
whether it is a savepoint or not?
This method had already been implemented in `CheckpoinProperties`.


> Savepoints must not be counted as retained checkpoints
> --
>
> Key: FLINK-6328
> URL: https://issues.apache.org/jira/browse/FLINK-6328
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.3.0, 1.4.0
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.3.0, 1.2.2
>
>
> The Checkpoint Store retains the *n* latest checkpoints.
> Savepoints are counted as well, meaning that for settings with 1 retained 
> checkpoint, there are sometimes no retained checkpoints at all, only a 
> savepoint.
> That is dangerous, because savepoints must be assumed to disappear at any 
> point in time - their lifecycle is out of control of the 
> CheckpointCoordinator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3965: [FLINK-6328] [chkPts] Don't add savepoints to Comp...

[tony810430]

Github user tony810430 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3965#discussion_r117902054
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
 ---
@@ -864,22 +864,28 @@ private void 
completePendingCheckpoint(PendingCheckpoint pendingCheckpoint) thro
// the pending checkpoint must be discarded after the 
finalization

Preconditions.checkState(pendingCheckpoint.isDiscarded() && completedCheckpoint 
!= null);
 
-   try {
-   
completedCheckpointStore.addCheckpoint(completedCheckpoint);
-   } catch (Exception exception) {
-   // we failed to store the completed checkpoint. 
Let's clean up
-   executor.execute(new Runnable() {
-   @Override
-   public void run() {
-   try {
-   
completedCheckpoint.discardOnFailedStoring();
-   } catch (Throwable t) {
-   LOG.warn("Could not 
properly discard completed checkpoint {}.", 
completedCheckpoint.getCheckpointID(), t);
+   // TODO: add savepoints to completed checkpoint store 
once FLINK-4815 has been completed
+   if 
(!CheckpointProperties.isSavepoint(completedCheckpoint.getProperties())) {
--- End diff --

Why not use `completedCheckpoint.getProperties().isSavepoint()` to check 
whether it is a savepoint or not?
This method had already been implemented in `CheckpoinProperties`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-6494) Migrate ResourceManager configuration options

[Fang Yong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fang Yong reassigned FLINK-6494:


Assignee: Fang Yong

> Migrate ResourceManager configuration options
> -
>
> Key: FLINK-6494
> URL: https://issues.apache.org/jira/browse/FLINK-6494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, ResourceManager
>Reporter: Chesnay Schepler
>Assignee: Fang Yong
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6494) Migrate ResourceManager configuration options

[Fang Yong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020675#comment-16020675
 ] 

Fang Yong commented on FLINK-6494:
--

Should configurations of yarn and mesos be moved to ResourceManagerOptions? 
[~Zentol]

> Migrate ResourceManager configuration options
> -
>
> Key: FLINK-6494
> URL: https://issues.apache.org/jira/browse/FLINK-6494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, ResourceManager
>Reporter: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6421) Unchecked reflection calls in PojoSerializer

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved FLINK-6421.
---
Resolution: Not A Problem

> Unchecked reflection calls in PojoSerializer
> 
>
> Key: FLINK-6421
> URL: https://issues.apache.org/jira/browse/FLINK-6421
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Reporter: Ted Yu
>Priority: Minor
>
> Here is one example:
> {code}
>   String subclassName = source.readUTF();
>   try {
> actualSubclass = Class.forName(subclassName, true, cl);
> {code}
> subclassName may carry tainted value, allowing an attacker to bypass security 
> checks, obtain unauthorized data, or execute arbitrary code



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6422) Unreachable code in FileInputFormat#createInputSplits

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-6422:
--
Description: 
Here is related code:

{code}
if (minNumSplits < 1) {
  throw new IllegalArgumentException("Number of input splits has to be at 
least 1.");
}
...
final long maxSplitSize = (minNumSplits < 1) ? Long.MAX_VALUE : 
(totalLength / minNumSplits +
  (totalLength % minNumSplits == 0 ? 0 : 1));
{code}
minNumSplits wouldn't be less than 1 getting to the assignment of maxSplitSize.

  was:
Here is related code:
{code}
if (minNumSplits < 1) {
  throw new IllegalArgumentException("Number of input splits has to be at 
least 1.");
}
...
final long maxSplitSize = (minNumSplits < 1) ? Long.MAX_VALUE : 
(totalLength / minNumSplits +
  (totalLength % minNumSplits == 0 ? 0 : 1));
{code}
minNumSplits wouldn't be less than 1 getting to the assignment of maxSplitSize.


> Unreachable code in FileInputFormat#createInputSplits
> -
>
> Key: FLINK-6422
> URL: https://issues.apache.org/jira/browse/FLINK-6422
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Priority: Minor
>
> Here is related code:
> {code}
> if (minNumSplits < 1) {
>   throw new IllegalArgumentException("Number of input splits has to be at 
> least 1.");
> }
> ...
> final long maxSplitSize = (minNumSplits < 1) ? Long.MAX_VALUE : 
> (totalLength / minNumSplits +
>   (totalLength % minNumSplits == 0 ? 0 : 1));
> {code}
> minNumSplits wouldn't be less than 1 getting to the assignment of 
> maxSplitSize.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6467) Potential lack of synchronization w.r.t. newSstFiles in RocksDBKeyedStateBackend#releaseResources()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020613#comment-16020613
 ] 

Ted Yu commented on FLINK-6467:
---

Alright.

> Potential lack of synchronization w.r.t. newSstFiles in 
> RocksDBKeyedStateBackend#releaseResources()
> ---
>
> Key: FLINK-6467
> URL: https://issues.apache.org/jira/browse/FLINK-6467
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
>   if (canceled) {
> List statesToDiscard = new ArrayList<>();
> statesToDiscard.add(metaStateHandle);
> statesToDiscard.addAll(miscFiles.values());
> statesToDiscard.addAll(newSstFiles.values());
> {code}
> It seems access to newSstFiles should be protected by 
> stateBackend.asyncSnapshotLock



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6493) Ineffective null check in RegisteredOperatorBackendStateMetaInfo#equals()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-6493:
--
Description: 
{code}
&& ((partitionStateSerializer == null && ((Snapshot) 
obj).getPartitionStateSerializer() == null)
  || partitionStateSerializer.equals(((Snapshot) 
obj).getPartitionStateSerializer()))
&& ((partitionStateSerializerConfigSnapshot == null && ((Snapshot) 
obj).getPartitionStateSerializerConfigSnapshot() == null)
  || partitionStateSerializerConfigSnapshot.equals(((Snapshot) 
obj).getPartitionStateSerializerConfigSnapshot()));
{code}

The null check for partitionStateSerializer / 
partitionStateSerializerConfigSnapshot is in combination with another clause.

This may lead to NPE in the partitionStateSerializer.equals() call.

  was:
{code}
&& ((partitionStateSerializer == null && ((Snapshot) 
obj).getPartitionStateSerializer() == null)
  || partitionStateSerializer.equals(((Snapshot) 
obj).getPartitionStateSerializer()))
&& ((partitionStateSerializerConfigSnapshot == null && ((Snapshot) 
obj).getPartitionStateSerializerConfigSnapshot() == null)
  || partitionStateSerializerConfigSnapshot.equals(((Snapshot) 
obj).getPartitionStateSerializerConfigSnapshot()));
{code}
The null check for partitionStateSerializer / 
partitionStateSerializerConfigSnapshot is in combination with another clause.

This may lead to NPE in the partitionStateSerializer.equals() call.


> Ineffective null check in RegisteredOperatorBackendStateMetaInfo#equals()
> -
>
> Key: FLINK-6493
> URL: https://issues.apache.org/jira/browse/FLINK-6493
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> && ((partitionStateSerializer == null && ((Snapshot) 
> obj).getPartitionStateSerializer() == null)
>   || partitionStateSerializer.equals(((Snapshot) 
> obj).getPartitionStateSerializer()))
> && ((partitionStateSerializerConfigSnapshot == null && ((Snapshot) 
> obj).getPartitionStateSerializerConfigSnapshot() == null)
>   || partitionStateSerializerConfigSnapshot.equals(((Snapshot) 
> obj).getPartitionStateSerializerConfigSnapshot()));
> {code}
> The null check for partitionStateSerializer / 
> partitionStateSerializerConfigSnapshot is in combination with another clause.
> This may lead to NPE in the partitionStateSerializer.equals() call.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6669) [Build] Scala style check errror on Windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020598#comment-16020598
 ] 

ASF GitHub Bot commented on FLINK-6669:
---

GitHub user lingjinjiang opened a pull request:

https://github.com/apache/flink/pull/3967

[FLINK-6669] set inputEncoding to UTF-8 in scalastyle-maven-plugin

When build the source code on Windows, a scala style check error happend.
It may be caused by the Windows default encoding. When set the 
inputEncoding to UTF-8 in scalastyle-maven-plugin, the error don't happen.

[https://issues.apache.org/jira/browse/FLINK-6669]

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lingjinjiang/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3967.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3967


commit 505dfc5ae8140bc1be02fee610fd0f4de3e59a4a
Author: lingjinjiang 
Date:   2017-05-23T02:51:46Z

[FLINK-6669] set inputEncoding to UTF-8 in scalastyle-maven-plugin




> [Build] Scala style check errror on Windows
> ---
>
> Key: FLINK-6669
> URL: https://issues.apache.org/jira/browse/FLINK-6669
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
> Environment: Windows
>Reporter: Jinjiang Ling
>Assignee: Jinjiang Ling
>Priority: Minor
> Attachments: FLINK-6669.patch
>
>
> When build the source code on Windows, a scala style check error happend.
> Here is the error messages.
> [INFO]
> [INFO] --- scalastyle-maven-plugin:0.8.0:check (default) @ flink-scala_2.10 
> ---
> error 
> file=E:\github\flink\flink-scala\src\main\scala\org\apache\flink\api\scala\utils\package.scala
>  message=Input length = 2
> Saving to outputFile=E:\github\flink\flink-scala\target\scalastyle-output.xml
> Processed 78 file(s)
> Found 1 errors
> Found 0 warnings
> Found 0 infos
> Finished in 1189 ms
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] force-shading .. SUCCESS [ 37.206 
> s]
> [INFO] flink .. SUCCESS [03:27 
> min]
> [INFO] flink-annotations .. SUCCESS [  3.020 
> s]
> [INFO] flink-shaded-hadoop  SUCCESS [  0.928 
> s]
> [INFO] flink-shaded-hadoop2 ... SUCCESS [ 15.314 
> s]
> [INFO] flink-shaded-hadoop2-uber .. SUCCESS [ 13.085 
> s]
> [INFO] flink-shaded-curator ... SUCCESS [  0.234 
> s]
> [INFO] flink-shaded-curator-recipes ... SUCCESS [  3.336 
> s]
> [INFO] flink-shaded-curator-test .. SUCCESS [  2.948 
> s]
> [INFO] flink-metrics .. SUCCESS [  0.286 
> s]
> [INFO] flink-metrics-core . SUCCESS [  9.065 
> s]
> [INFO] flink-test-utils-parent  SUCCESS [  0.327 
> s]
> [INFO] flink-test-utils-junit . SUCCESS [  1.452 
> s]
> [INFO] flink-core . SUCCESS [ 54.277 
> s]
> \[INFO\] flink-java . SUCCESS [ 
> 25.244 s]
> [INFO] flink-runtime .. SUCCESS [03:08 
> min]
> [INFO] flink-optimizer  SUCCESS [ 14.540 
> s]
> [INFO] flink-clients .. SUCCESS [ 14.457 
> s]
> [INFO] flink-streaming-java ... SUCCESS [ 58.130 
> s]
> [INFO] flink-test-utils ... SUCCESS [ 19.906 
> s]
> [INFO] flink-scala  FAILURE [ 56.634 
> s]
> [INFO] flink-runtime-web .. SKIPPED
> I think this is caused by the Windows default encoding. When I set the 
> inputEncoding to UTF-8 in scalastyle-maven-plugin, the error don't happen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3967: [FLINK-6669] set inputEncoding to UTF-8 in scalast...

[lingjinjiang]

GitHub user lingjinjiang opened a pull request:

https://github.com/apache/flink/pull/3967

[FLINK-6669] set inputEncoding to UTF-8 in scalastyle-maven-plugin

When build the source code on Windows, a scala style check error happend.
It may be caused by the Windows default encoding. When set the 
inputEncoding to UTF-8 in scalastyle-maven-plugin, the error don't happen.

[https://issues.apache.org/jira/browse/FLINK-6669]

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lingjinjiang/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3967.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3967


commit 505dfc5ae8140bc1be02fee610fd0f4de3e59a4a
Author: lingjinjiang 
Date:   2017-05-23T02:51:46Z

[FLINK-6669] set inputEncoding to UTF-8 in scalastyle-maven-plugin




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-6669) [Build] Scala style check errror on Windows

[Jinjiang Ling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinjiang Ling updated FLINK-6669:
-
Attachment: FLINK-6669.patch

> [Build] Scala style check errror on Windows
> ---
>
> Key: FLINK-6669
> URL: https://issues.apache.org/jira/browse/FLINK-6669
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
> Environment: Windows
>Reporter: Jinjiang Ling
>Assignee: Jinjiang Ling
>Priority: Minor
> Attachments: FLINK-6669.patch
>
>
> When build the source code on Windows, a scala style check error happend.
> Here is the error messages.
> [INFO]
> [INFO] --- scalastyle-maven-plugin:0.8.0:check (default) @ flink-scala_2.10 
> ---
> error 
> file=E:\github\flink\flink-scala\src\main\scala\org\apache\flink\api\scala\utils\package.scala
>  message=Input length = 2
> Saving to outputFile=E:\github\flink\flink-scala\target\scalastyle-output.xml
> Processed 78 file(s)
> Found 1 errors
> Found 0 warnings
> Found 0 infos
> Finished in 1189 ms
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] force-shading .. SUCCESS [ 37.206 
> s]
> [INFO] flink .. SUCCESS [03:27 
> min]
> [INFO] flink-annotations .. SUCCESS [  3.020 
> s]
> [INFO] flink-shaded-hadoop  SUCCESS [  0.928 
> s]
> [INFO] flink-shaded-hadoop2 ... SUCCESS [ 15.314 
> s]
> [INFO] flink-shaded-hadoop2-uber .. SUCCESS [ 13.085 
> s]
> [INFO] flink-shaded-curator ... SUCCESS [  0.234 
> s]
> [INFO] flink-shaded-curator-recipes ... SUCCESS [  3.336 
> s]
> [INFO] flink-shaded-curator-test .. SUCCESS [  2.948 
> s]
> [INFO] flink-metrics .. SUCCESS [  0.286 
> s]
> [INFO] flink-metrics-core . SUCCESS [  9.065 
> s]
> [INFO] flink-test-utils-parent  SUCCESS [  0.327 
> s]
> [INFO] flink-test-utils-junit . SUCCESS [  1.452 
> s]
> [INFO] flink-core . SUCCESS [ 54.277 
> s]
> \[INFO\] flink-java . SUCCESS [ 
> 25.244 s]
> [INFO] flink-runtime .. SUCCESS [03:08 
> min]
> [INFO] flink-optimizer  SUCCESS [ 14.540 
> s]
> [INFO] flink-clients .. SUCCESS [ 14.457 
> s]
> [INFO] flink-streaming-java ... SUCCESS [ 58.130 
> s]
> [INFO] flink-test-utils ... SUCCESS [ 19.906 
> s]
> [INFO] flink-scala  FAILURE [ 56.634 
> s]
> [INFO] flink-runtime-web .. SKIPPED
> I think this is caused by the Windows default encoding. When I set the 
> inputEncoding to UTF-8 in scalastyle-maven-plugin, the error don't happen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6669) [Build] Scala style check errror on Windows

[Jinjiang Ling (JIRA)]

Jinjiang Ling created FLINK-6669:


 Summary: [Build] Scala style check errror on Windows
 Key: FLINK-6669
 URL: https://issues.apache.org/jira/browse/FLINK-6669
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.3.0, 1.4.0
 Environment: Windows
Reporter: Jinjiang Ling
Assignee: Jinjiang Ling
Priority: Minor


When build the source code on Windows, a scala style check error happend.
Here is the error messages.

[INFO]
[INFO] --- scalastyle-maven-plugin:0.8.0:check (default) @ flink-scala_2.10 ---
error 
file=E:\github\flink\flink-scala\src\main\scala\org\apache\flink\api\scala\utils\package.scala
 message=Input length = 2
Saving to outputFile=E:\github\flink\flink-scala\target\scalastyle-output.xml
Processed 78 file(s)
Found 1 errors
Found 0 warnings
Found 0 infos
Finished in 1189 ms
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] force-shading .. SUCCESS [ 37.206 s]
[INFO] flink .. SUCCESS [03:27 min]
[INFO] flink-annotations .. SUCCESS [  3.020 s]
[INFO] flink-shaded-hadoop  SUCCESS [  0.928 s]
[INFO] flink-shaded-hadoop2 ... SUCCESS [ 15.314 s]
[INFO] flink-shaded-hadoop2-uber .. SUCCESS [ 13.085 s]
[INFO] flink-shaded-curator ... SUCCESS [  0.234 s]
[INFO] flink-shaded-curator-recipes ... SUCCESS [  3.336 s]
[INFO] flink-shaded-curator-test .. SUCCESS [  2.948 s]
[INFO] flink-metrics .. SUCCESS [  0.286 s]
[INFO] flink-metrics-core . SUCCESS [  9.065 s]
[INFO] flink-test-utils-parent  SUCCESS [  0.327 s]
[INFO] flink-test-utils-junit . SUCCESS [  1.452 s]
[INFO] flink-core . SUCCESS [ 54.277 s]
\[INFO\] flink-java . SUCCESS [ 25.244 
s]
[INFO] flink-runtime .. SUCCESS [03:08 min]
[INFO] flink-optimizer  SUCCESS [ 14.540 s]
[INFO] flink-clients .. SUCCESS [ 14.457 s]
[INFO] flink-streaming-java ... SUCCESS [ 58.130 s]
[INFO] flink-test-utils ... SUCCESS [ 19.906 s]
[INFO] flink-scala  FAILURE [ 56.634 s]
[INFO] flink-runtime-web .. SKIPPED

I think this is caused by the Windows default encoding. When I set the 
inputEncoding to UTF-8 in scalastyle-maven-plugin, the error don't happen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020414#comment-16020414
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zohar-mizrahi commented on the issue:

https://github.com/apache/flink/pull/3838
  
In the last change, I've rebased locally on top of origin/master, so I did 
`git push -f` to the master branch in my fork.


> Python API for streaming applications
> -
>
> Key: FLINK-5886
> URL: https://issues.apache.org/jira/browse/FLINK-5886
> Project: Flink
>  Issue Type: New Feature
>  Components: Python API
>Reporter: Zohar Mizrahi
>Assignee: Zohar Mizrahi
>
> A work in progress to provide python interface for Flink streaming APIs. The 
> core technology is based on jython and thus imposes two limitations: a. user 
> defined functions cannot use python extensions. b. the python version is 2.x
> The branch is based on Flink release 1.2.0, as can be found here:
> https://github.com/zohar-pm/flink/tree/python-streaming
> In order to test it, someone can use IntelliJ IDE. Assuming IntelliJ was 
> setup properly (see: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/ide_setup.html),
>  one can run/debug {{org.apache.flink.python.api.PythonStreamBinderTest}}, 
> which in return will execute all the tests under 
> {{/Users/zohar/dev/pm-flink/flink-libraries/flink-python/src/test/python/org/apache/flink/python/api/streaming}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3838: [FLINK-5886] Python API for streaming applications

[zohar-mizrahi]

Github user zohar-mizrahi commented on the issue:

https://github.com/apache/flink/pull/3838
  
In the last change, I've rebased locally on top of origin/master, so I did 
`git push -f` to the master branch in my fork.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020364#comment-16020364
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117864592
  
--- Diff: flink-libraries/flink-streaming-python/pom.xml ---
@@ -0,0 +1,99 @@
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+   4.0.0
+
+   
+   org.apache.flink
+   flink-libraries
+   1.3-SNAPSHOT
+   ..
+   
+
+   flink-streaming-python_2.10
+   flink-streaming-python
+   jar
+
+   
+   
+   
+   org.apache.maven.plugins
+   maven-jar-plugin
+   
+   
+   
jar-with-dependencies
+   
+   
+   
+   
true
+   
org.apache.flink.streaming.python.api.PythonStreamBinder
+   
+   
+   
+   
+   
+   
+
+   
+
+   
+
+   
+   org.apache.flink
+   flink-core
+   ${project.version}
+   
+   
+   org.apache.flink
+   flink-java
+   ${project.version}
--- End diff --

I'm not sure what you mean. Please explain.


> Python API for streaming applications
> -
>
> Key: FLINK-5886
> URL: https://issues.apache.org/jira/browse/FLINK-5886
> Project: Flink
>  Issue Type: New Feature
>  Components: Python API
>Reporter: Zohar Mizrahi
>Assignee: Zohar Mizrahi
>
> A work in progress to provide python interface for Flink streaming APIs. The 
> core technology is based on jython and thus imposes two limitations: a. user 
> defined functions cannot use python extensions. b. the python version is 2.x
> The branch is based on Flink release 1.2.0, as can be found here:
> https://github.com/zohar-pm/flink/tree/python-streaming
> In order to test it, someone can use IntelliJ IDE. Assuming IntelliJ was 
> setup properly (see: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/ide_setup.html),
>  one can run/debug {{org.apache.flink.python.api.PythonStreamBinderTest}}, 
> which in return will execute all the tests under 
> {{/Users/zohar/dev/pm-flink/flink-libraries/flink-python/src/test/python/org/apache/flink/python/api/streaming}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3838: [FLINK-5886] Python API for streaming applications

[zohar-mizrahi]

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117864592
  
--- Diff: flink-libraries/flink-streaming-python/pom.xml ---
@@ -0,0 +1,99 @@
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+   4.0.0
+
+   
+   org.apache.flink
+   flink-libraries
+   1.3-SNAPSHOT
+   ..
+   
+
+   flink-streaming-python_2.10
+   flink-streaming-python
+   jar
+
+   
+   
+   
+   org.apache.maven.plugins
+   maven-jar-plugin
+   
+   
+   
jar-with-dependencies
+   
+   
+   
+   
true
+   
org.apache.flink.streaming.python.api.PythonStreamBinder
+   
+   
+   
+   
+   
+   
+
+   
+
+   
+
+   
+   org.apache.flink
+   flink-core
+   ${project.version}
+   
+   
+   org.apache.flink
+   flink-java
+   ${project.version}
--- End diff --

I'm not sure what you mean. Please explain.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6668) Add flink history server to DCOS

[Eron Wright (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020261#comment-16020261
 ] 

Eron Wright  commented on FLINK-6668:
-

[~skonto] I see this as related to the dispatcher work (FLINK-4897).   The 
dispatcher is envisioned as an always-on service, providing a REST API for job 
graph submission (from a Flink client/driver program) and mgmt, a web UI and 
history server.   

I haven't studied the work done in FLINK-1579 but assume it provided the core 
for this.   

> Add flink history server to DCOS
> 
>
> Key: FLINK-6668
> URL: https://issues.apache.org/jira/browse/FLINK-6668
> Project: Flink
>  Issue Type: New Feature
>  Components: Mesos
>Reporter: Stavros Kontopoulos
>
> We need to have history server within dc/os env as with the spark case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6668) Add flink history server to DCOS

[Stavros Kontopoulos (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020233#comment-16020233
 ] 

Stavros Kontopoulos commented on FLINK-6668:


[~eronwright] what do you think? I created a separate issue for this. Is this 
viable to implement, I am willing to try it.

> Add flink history server to DCOS
> 
>
> Key: FLINK-6668
> URL: https://issues.apache.org/jira/browse/FLINK-6668
> Project: Flink
>  Issue Type: New Feature
>  Components: Mesos
>Reporter: Stavros Kontopoulos
>
> We need to have history server within dc/os env as with the spark case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6668) Add flink history server to DCOS

[Stavros Kontopoulos (JIRA)]

Stavros Kontopoulos created FLINK-6668:
--

 Summary: Add flink history server to DCOS
 Key: FLINK-6668
 URL: https://issues.apache.org/jira/browse/FLINK-6668
 Project: Flink
  Issue Type: New Feature
  Components: Mesos
Reporter: Stavros Kontopoulos


We need to have history server within dc/os env as with the spark case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6660) expand the streaming connectors overview page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020147#comment-16020147
 ] 

ASF GitHub Bot commented on FLINK-6660:
---

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/3964#discussion_r117838617
  
--- Diff: docs/dev/connectors/index.md ---
@@ -25,22 +25,54 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-Connectors provide code for interfacing with various third-party systems.
+* toc
+{:toc}
 
-Currently these systems are supported: (Please select the respective 
documentation page from the navigation on the left.)
+## Predefined Sources and Sinks
 
- * [Apache Kafka](https://kafka.apache.org/) (sink/source)
- * [Elasticsearch](https://elastic.co/) (sink)
- * [Hadoop FileSystem](http://hadoop.apache.org) (sink)
- * [RabbitMQ](http://www.rabbitmq.com/) (sink/source)
- * [Amazon Kinesis Streams](http://aws.amazon.com/kinesis/streams/) 
(sink/source)
- * [Twitter Streaming API](https://dev.twitter.com/docs/streaming-apis) 
(source)
- * [Apache NiFi](https://nifi.apache.org) (sink/source)
- * [Apache Cassandra](https://cassandra.apache.org/) (sink)
+A few basic data sources and sinks are built into Flink and are always 
available.
+The [predefined data sources]({{ site.baseurll 
}}/dev/datastream_api.html#data-sources) include reading from files, 
directories, and sockets, and
--- End diff --

Should there be a comma between sockets and and?


> expand the streaming connectors overview page 
> --
>
> Key: FLINK-6660
> URL: https://issues.apache.org/jira/browse/FLINK-6660
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Streaming Connectors
>Affects Versions: 1.3.0, 1.4.0
>Reporter: David Anderson
>Assignee: David Anderson
>
> The overview page for streaming connectors is too lean -- it should provide 
> more context and also guide the reader toward related topics.
> Note that FLINK-6038 will add links to the Bahir connectors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3964: [FLINK-6660][docs] expand the connectors overview ...

[greghogan]

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/3964#discussion_r117838617
  
--- Diff: docs/dev/connectors/index.md ---
@@ -25,22 +25,54 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-Connectors provide code for interfacing with various third-party systems.
+* toc
+{:toc}
 
-Currently these systems are supported: (Please select the respective 
documentation page from the navigation on the left.)
+## Predefined Sources and Sinks
 
- * [Apache Kafka](https://kafka.apache.org/) (sink/source)
- * [Elasticsearch](https://elastic.co/) (sink)
- * [Hadoop FileSystem](http://hadoop.apache.org) (sink)
- * [RabbitMQ](http://www.rabbitmq.com/) (sink/source)
- * [Amazon Kinesis Streams](http://aws.amazon.com/kinesis/streams/) 
(sink/source)
- * [Twitter Streaming API](https://dev.twitter.com/docs/streaming-apis) 
(source)
- * [Apache NiFi](https://nifi.apache.org) (sink/source)
- * [Apache Cassandra](https://cassandra.apache.org/) (sink)
+A few basic data sources and sinks are built into Flink and are always 
available.
+The [predefined data sources]({{ site.baseurll 
}}/dev/datastream_api.html#data-sources) include reading from files, 
directories, and sockets, and
--- End diff --

Should there be a comma between sockets and and?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6603) Enable checkstyle on test sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020111#comment-16020111
 ] 

ASF GitHub Bot commented on FLINK-6603:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3941
  
@greghogan +1, feel free to merge it :)


> Enable checkstyle on test sources
> -
>
> Key: FLINK-6603
> URL: https://issues.apache.org/jira/browse/FLINK-6603
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.4.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.4.0
>
>
> With the addition of strict checkstyle to select modules (currently limited 
> to {{flink-streaming-java}}) we can enable the checkstyle flag 
> {{includeTestSourceDirectory}} to perform the same unused imports, 
> whitespace, and other checks on test sources.
> Should first resolve the import grouping as discussed in FLINK-6107. Also, 
> several tests exceed the 2500 line limit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3941: [FLINK-6603] [streaming] Enable checkstyle on test source...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3941
  
@greghogan +1, feel free to merge it :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6659) RocksDBMergeIteratorTest, SavepointITCase leave temporary directories behind

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020110#comment-16020110
 ] 

Chesnay Schepler commented on FLINK-6659:
-

+ to remove the method.

> RocksDBMergeIteratorTest, SavepointITCase leave temporary directories behind
> 
>
> Key: FLINK-6659
> URL: https://issues.apache.org/jira/browse/FLINK-6659
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.2.2
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> {{RocksDBMergeIteratorTest}} uses a newly created temporary directory (via 
> {{CommonTestUtils.createTempDirectory()}}) for its RocksDB instance but does 
> not delete is when finished. We should better replace this pattern with a 
> proper {{@Rule}}-based approach



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6659) RocksDBMergeIteratorTest, SavepointITCase leave temporary directories behind

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020109#comment-16020109
 ] 

ASF GitHub Bot commented on FLINK-6659:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3962
  
+1.


> RocksDBMergeIteratorTest, SavepointITCase leave temporary directories behind
> 
>
> Key: FLINK-6659
> URL: https://issues.apache.org/jira/browse/FLINK-6659
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.2.2
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> {{RocksDBMergeIteratorTest}} uses a newly created temporary directory (via 
> {{CommonTestUtils.createTempDirectory()}}) for its RocksDB instance but does 
> not delete is when finished. We should better replace this pattern with a 
> proper {{@Rule}}-based approach



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3962: [FLINK-6659] fix (some) unit tests leaving temporary data...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3962
  
+1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6446) Various improvements to the Web Frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020070#comment-16020070
 ] 

ASF GitHub Bot commented on FLINK-6446:
---

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/3946


> Various improvements to the Web Frontend
> 
>
> Key: FLINK-6446
> URL: https://issues.apache.org/jira/browse/FLINK-6446
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Stephan Ewen
>
> This is the umbrella issue for various improvements to the web frontend,



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3946: [FLINK-6446] Fix some small issues in the web UI

[zentol]

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/3946


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-6495) Migrate Akka configuration options

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6495.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

1.4: 302c6741f862f13c6ea3d5490a31fadc20e976c8

> Migrate Akka configuration options
> --
>
> Key: FLINK-6495
> URL: https://issues.apache.org/jira/browse/FLINK-6495
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Reporter: Chesnay Schepler
>Assignee: Fang Yong
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-6448) Web UI TaskManager view: Rename 'Free Memory' to 'JVM Heap'

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6448.
---
   Resolution: Fixed
Fix Version/s: 1.4.0
   1.3.0

1.3: f4f5cf13ea0821658d800cca8be6c02729c23dd9
1.4: b3e1642f33e9a6e3df523aae5c4b04205f1151d8

> Web UI TaskManager view: Rename 'Free Memory' to 'JVM Heap'
> ---
>
> Key: FLINK-6448
> URL: https://issues.apache.org/jira/browse/FLINK-6448
> Project: Flink
>  Issue Type: Sub-task
>  Components: Webfrontend
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
>  Labels: easyfix, starter
> Fix For: 1.3.0, 1.4.0
>
>
> In the TaskManager view, the laben 'Free Memory' is wrong / misleading and 
> should be 'JVM Heap Size' instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-6451) Web UI: Rename 'Metrics' view to 'Task Metrics'

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6451.
---
   Resolution: Fixed
Fix Version/s: 1.4.0
   1.3.0

1.3: 7a9030fe94b8c5577422195ae2c3fc29cf4bf680
1.4: 8b89e2fdb855d78cea8b08c060f884a4660187fe

> Web UI: Rename 'Metrics' view to 'Task Metrics'
> ---
>
> Key: FLINK-6451
> URL: https://issues.apache.org/jira/browse/FLINK-6451
> Project: Flink
>  Issue Type: Sub-task
>  Components: Webfrontend
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
>  Labels: easyfix, starter
> Fix For: 1.3.0, 1.4.0
>
>
> In the UI, under the {{Overview}} of a specific job, the tab {{Metrics}} 
> shows metrics for tasks only, and not all available metrics.
> We should rename that to {{Task Metrics}}. That also differentiates the view 
> clearly from the job-level metrics view proposed in FLINK-6449



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-6610) WebServer could not be created,when set the "jobmanager.web.submit.enable" to false

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6610.
---
Resolution: Fixed

1.3: f72b8737912f5546aa97c3cfcde86d9d1aebc60e
1.4: 60fa0d4da38eb0e724304903fe42471264ff7e2d

> WebServer could not be created,when set the "jobmanager.web.submit.enable" to 
> false
> ---
>
> Key: FLINK-6610
> URL: https://issues.apache.org/jira/browse/FLINK-6610
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.3.0, 1.4.0
>Reporter: zhihao chen
>Assignee: Chesnay Schepler
>
> WebServer could not be created,when set the "jobmanager.web.submit.enable" to 
> false  
> because the WebFrontendBootstrap will check uploadDir not allow be null 
> this.uploadDir = Preconditions.checkNotNull(directory);
> {code}
> 2017-05-17 15:15:46,938 ERROR 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils   - WebServer 
> could not be created
> java.lang.NullPointerException
>   at 
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
>   at 
> org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:73)
>   at 
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:359)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:135)
>   at 
> org.apache.flink.runtime.clusterframework.BootstrapTools.createWebMonitorIfConfigured(BootstrapTools.java:242)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.runApplicationMaster(YarnApplicationMasterRunner.java:352)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:195)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:192)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.run(YarnApplicationMasterRunner.java:192)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.main(YarnApplicationMasterRunner.java:116)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-6450) Web UI Subtasks view for TaskManagers has a misleading name

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6450.
---
   Resolution: Fixed
Fix Version/s: 1.4.0
   1.3.0

1.3: 4df9b5a7fc15fb00b598bf2d60fd085b93db32bf
1.4: ec21ce6726be0c7bca50f744da75ec20d1e1fc46

> Web UI Subtasks view for TaskManagers has a misleading name
> ---
>
> Key: FLINK-6450
> URL: https://issues.apache.org/jira/browse/FLINK-6450
> Project: Flink
>  Issue Type: Sub-task
>  Components: Webfrontend
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
>  Labels: easyfix, starter
> Fix For: 1.3.0, 1.4.0
>
>
> The register for the subtasks grouped by TaskManager is simply called 
> {{TaskManager}}, which is confusing users. I suggest to rename it to 
> {{Subtasks by TaskManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6610) WebServer could not be created,when set the "jobmanager.web.submit.enable" to false

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-6610:

Affects Version/s: 1.4.0

> WebServer could not be created,when set the "jobmanager.web.submit.enable" to 
> false
> ---
>
> Key: FLINK-6610
> URL: https://issues.apache.org/jira/browse/FLINK-6610
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.3.0, 1.4.0
>Reporter: zhihao chen
>Assignee: Chesnay Schepler
>
> WebServer could not be created,when set the "jobmanager.web.submit.enable" to 
> false  
> because the WebFrontendBootstrap will check uploadDir not allow be null 
> this.uploadDir = Preconditions.checkNotNull(directory);
> {code}
> 2017-05-17 15:15:46,938 ERROR 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils   - WebServer 
> could not be created
> java.lang.NullPointerException
>   at 
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
>   at 
> org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:73)
>   at 
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:359)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:135)
>   at 
> org.apache.flink.runtime.clusterframework.BootstrapTools.createWebMonitorIfConfigured(BootstrapTools.java:242)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.runApplicationMaster(YarnApplicationMasterRunner.java:352)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:195)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:192)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.run(YarnApplicationMasterRunner.java:192)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.main(YarnApplicationMasterRunner.java:116)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6610) WebServer could not be created,when set the "jobmanager.web.submit.enable" to false

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020059#comment-16020059
 ] 

ASF GitHub Bot commented on FLINK-6610:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3947


> WebServer could not be created,when set the "jobmanager.web.submit.enable" to 
> false
> ---
>
> Key: FLINK-6610
> URL: https://issues.apache.org/jira/browse/FLINK-6610
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.3.0
>Reporter: zhihao chen
>Assignee: Chesnay Schepler
>
> WebServer could not be created,when set the "jobmanager.web.submit.enable" to 
> false  
> because the WebFrontendBootstrap will check uploadDir not allow be null 
> this.uploadDir = Preconditions.checkNotNull(directory);
> {code}
> 2017-05-17 15:15:46,938 ERROR 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils   - WebServer 
> could not be created
> java.lang.NullPointerException
>   at 
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
>   at 
> org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:73)
>   at 
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:359)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:135)
>   at 
> org.apache.flink.runtime.clusterframework.BootstrapTools.createWebMonitorIfConfigured(BootstrapTools.java:242)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.runApplicationMaster(YarnApplicationMasterRunner.java:352)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:195)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:192)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.run(YarnApplicationMasterRunner.java:192)
>   at 
> org.apache.flink.yarn.YarnApplicationMasterRunner.main(YarnApplicationMasterRunner.java:116)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6495) Migrate Akka configuration options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020060#comment-16020060
 ] 

ASF GitHub Bot commented on FLINK-6495:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3935


> Migrate Akka configuration options
> --
>
> Key: FLINK-6495
> URL: https://issues.apache.org/jira/browse/FLINK-6495
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Reporter: Chesnay Schepler
>Assignee: Fang Yong
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3935: [FLINK-6495] Migrate Akka configuration options

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3935


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3947: [FLINK-6610][web] Allow uploadDir to be null in We...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3947


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-3610) Scala DataStream#addSink returns Java DataStreamSink

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-3610:

Issue Type: Sub-task  (was: Bug)
Parent: FLINK-3957

> Scala DataStream#addSink returns Java DataStreamSink
> 
>
> Key: FLINK-3610
> URL: https://issues.apache.org/jira/browse/FLINK-3610
> Project: Flink
>  Issue Type: Sub-task
>  Components: DataStream API, Scala API
>Affects Versions: 1.0.0
>Reporter: Márton Balassi
>Assignee: Márton Balassi
> Fix For: 2.0.0
>
>
> I have just come across a shortcoming of the streaming Scala API: it 
> completely lacks the Scala implementation of the DataStreamSink and instead 
> the Java version is used. [1]
> I would regard this as a bug that needs a fix for 1.0.1. Unfortunately this 
> is also api-breaking.
> [1] 
> https://github.com/apache/flink/blob/master/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/DataStream.scala#L928-L929



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (FLINK-3448) WebRuntimeMonitor: Move initialization of handlers to start() method

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-3448:
---

Assignee: Chesnay Schepler  (was: Maximilian Michels)

> WebRuntimeMonitor: Move initialization of handlers to start() method
> 
>
> Key: FLINK-3448
> URL: https://issues.apache.org/jira/browse/FLINK-3448
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.0.0
>
>
> HTTP handlers are initialized upon instance creation of 
> {{WebRuntimeMonitor}}. However, this is useless because the initialization is 
> not completed until the start(jobManager) method is called. In the meantime, 
> user gets messages about ongoing leader elections from the webfrontend.
> I would suggest to move the starting code in the constructor to the 
> start(jobManager) method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6320) Flakey JobManagerHAJobGraphRecoveryITCase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020007#comment-16020007
 ] 

ASF GitHub Bot commented on FLINK-6320:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3966
  
+1 to merge.


> Flakey JobManagerHAJobGraphRecoveryITCase
> -
>
> Key: FLINK-6320
> URL: https://issues.apache.org/jira/browse/FLINK-6320
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>  Labels: test-stability
>
> it looks as if there is a race condition in the cleanup of 
> {{JobManagerHAJobGraphRecoveryITCase}}.
> {code}
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 50.271 sec 
> <<< FAILURE! - in 
> org.apache.flink.test.recovery.JobManagerHAJobGraphRecoveryITCase
> testJobPersistencyWhenJobManagerShutdown(org.apache.flink.test.recovery.JobManagerHAJobGraphRecoveryITCase)
>   Time elapsed: 0.129 sec  <<< ERROR!
> java.io.FileNotFoundException: File does not exist: 
> /tmp/9b63934b-789d-428c-aa9e-47d5d8fa1e32/recovery/submittedJobGraphf763d61fba47
>   at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275)
>   at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
>   at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
>   at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
>   at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
>   at 
> org.apache.flink.test.recovery.JobManagerHAJobGraphRecoveryITCase.cleanUp(JobManagerHAJobGraphRecoveryITCase.java:112)
> {code}
> Full log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/223124016/log.txt
> Maybe a rule-based temporary directory is a better solution:
> {code:java}
>   @Rule
>   public TemporaryFolder tempFolder = new TemporaryFolder();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3966: [FLINK-6320] fix unit test failing sometimes when deletin...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3966
  
+1 to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6537) Umbrella issue for fixes to incremental snapshots

[Gyula Fora (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1601#comment-1601
 ] 

Gyula Fora commented on FLINK-6537:
---

You can look at all the logs I have posted. For example: 
https://gist.github.com/gyfora/2bb5569fb703bbd7e47ba60352f90086#file-gistfile1-txt-L197

This exact error happens at every single savepoint attempt.

> Umbrella issue for fixes to incremental snapshots
> -
>
> Key: FLINK-6537
> URL: https://issues.apache.org/jira/browse/FLINK-6537
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
> Fix For: 1.3.0
>
>
> This issue tracks ongoing fixes in the incremental checkpointing feature for 
> the 1.3 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6667) Pass a callback type to the RestartStrategy, rather than the full ExecutionGraph

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-6667:
---

 Summary: Pass a callback type to the RestartStrategy, rather than 
the full ExecutionGraph
 Key: FLINK-6667
 URL: https://issues.apache.org/jira/browse/FLINK-6667
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
Affects Versions: 1.3.0
Reporter: Stephan Ewen


To let the {{ResourceStrategy}} work across multiple {{FailoverStrategy}} 
implementations, it needs to be passed a "callback" to call to trigger the 
restart of tasks/regions/etc.

Such a "callback" would be a nice abstraction to use for global restarts as 
well, to not expose the full execution graph.

Ideally, the callback is one-shot, so it cannot accidentally be used to call 
restart() multiple times.

This would also make the testing of RestartStrategies much easier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6666) RestartStrategy should differentiate between types of recovery (global / local / resource missing)

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-:
---

 Summary: RestartStrategy should differentiate between types of 
recovery (global / local / resource missing)
 Key: FLINK-
 URL: https://issues.apache.org/jira/browse/FLINK-
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
Affects Versions: 1.3.0
Reporter: Stephan Ewen


Currently, the {{RestrartStrategy}} has a single method that is called when a 
failure requires an ExecutionGraph restart.

With the new addition of incremental recovery, it is desirable to distinguish 
between the type of failover that happens.

I would suggest to extend the {{RestartStrategy}} to support three 
cases/methods:

  - {{restartGlobal()}} for a full restart recovery
  - {{restartLocal()}} for a recovery coordinated by the {{FailoverStrategy}}
  - {{restartOnMissingResources()}} if the failure cause was missing slots

The last case is interesting, in my opinion, because it is commonly desirable 
that regular failover has no delay, but failover on missing resources has a 
short delay (1s or so) to avoid very fast cycles of restart attempts (in 
standalone mode, there can easily be 100,000 restarts after a second, when no 
resources are available and no delay happens during restarts).




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6665) Pass a ScheduledExecutorService to the RestartStrategy

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-6665:
---

 Summary: Pass a ScheduledExecutorService to the RestartStrategy
 Key: FLINK-6665
 URL: https://issues.apache.org/jira/browse/FLINK-6665
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
Reporter: Stephan Ewen
 Fix For: 1.3.0


Currently, the {{RestartStrategy}} is called when the {{ExecutionGraph}} should 
be restarted.

To facilitate delays before restarting, the strategy simply sleeps, blocking 
the thread that runs the ExecutionGraph's recovery method.

I suggest to pass {{ScheduledExecutorService}}) to the {{RestartStrategy}} and 
let it schedule the restart call that way, avoiding any sleeps.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6664) Extend Restart Strategy for Fine-grained Recovery and missing Resources

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-6664:
---

 Summary: Extend Restart Strategy for Fine-grained Recovery and 
missing Resources
 Key: FLINK-6664
 URL: https://issues.apache.org/jira/browse/FLINK-6664
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Coordination
Affects Versions: 1.3.0
Reporter: Stephan Ewen


This is the umbrella issue to extend the {{RestartStrategy}} to handle 
different kinds of failures / recoveries better:

  - Differentiation between local- and global failover
  - Better abstraction over the execution graph
  - Delaying restarts without blocking threads



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-4112) Replace SuppressRestartException with disabling RestartStrategy

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019956#comment-16019956
 ] 

Stephan Ewen commented on FLINK-4112:
-

Is this still relevant? Just asking, because the Exception is now used by 
certain tests as well...

> Replace SuppressRestartException with disabling RestartStrategy
> ---
>
> Key: FLINK-4112
> URL: https://issues.apache.org/jira/browse/FLINK-4112
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Priority: Minor
>
> Currently, Flink uses {{SuppressRestartExceptions}} to suppress job restarts. 
> This mechanism can be used by different components to control the lifecycle 
> of a job. Since the control flow of exceptions can be quite difficult 
> retrace, I propose to replace this mechanism by an explicit 
> {{RestartStrategy}} disable method. So instead of throwing a 
> {{SuppressRestartException}} we should disable the {{RestartStrategy}} to 
> avoid further job restarts.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6654) missing maven dependency on "flink-shaded-hadoop2-uber" in flink-dist

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019936#comment-16019936
 ] 

ASF GitHub Bot commented on FLINK-6654:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3960
  
Thanks. The exclusion should not be necessary, since the 
`flink-shaded-hadoop2-uber` dependency shades everything into one jar and has 
no further dependencies.


> missing maven dependency on "flink-shaded-hadoop2-uber" in flink-dist
> -
>
> Key: FLINK-6654
> URL: https://issues.apache.org/jira/browse/FLINK-6654
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.3.0
>
>
> Since applying FLINK-6514, flink-dist includes 
> {{flink-shaded-hadoop2-uber-*.jar}} but without giving this dependency in its 
> {{pom.xml}}. This may lead to concurrency issues during builds but also fails 
> building the flink-dist module only (with dependencies) as in
> {code}
> mvn clean install -pl flink-dist -am
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3960: [FLINK-6654][build] let 'flink-dist' properly depend on '...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3960
  
Thanks. The exclusion should not be necessary, since the 
`flink-shaded-hadoop2-uber` dependency shades everything into one jar and has 
no further dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019916#comment-16019916
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117807878
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/filecache/FileCache.java 
---
@@ -233,6 +234,20 @@ boolean holdsStillReference(String name, JobID jobId) {
//  Utilities
// 

 
+   /**
+* Remove a given path recursively if exists. The path can be a 
filepath or directory.
+*
+* @param path  The root path to remove.
+* @throws IOException
+* @throws URISyntaxException
+*/
+   public static void clearPath(String path) throws IOException, 
URISyntaxException {
--- End diff --

My bad! I don't use it anymore.


> Python API for streaming applications
> -
>
> Key: FLINK-5886
> URL: https://issues.apache.org/jira/browse/FLINK-5886
> Project: Flink
>  Issue Type: New Feature
>  Components: Python API
>Reporter: Zohar Mizrahi
>Assignee: Zohar Mizrahi
>
> A work in progress to provide python interface for Flink streaming APIs. The 
> core technology is based on jython and thus imposes two limitations: a. user 
> defined functions cannot use python extensions. b. the python version is 2.x
> The branch is based on Flink release 1.2.0, as can be found here:
> https://github.com/zohar-pm/flink/tree/python-streaming
> In order to test it, someone can use IntelliJ IDE. Assuming IntelliJ was 
> setup properly (see: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/ide_setup.html),
>  one can run/debug {{org.apache.flink.python.api.PythonStreamBinderTest}}, 
> which in return will execute all the tests under 
> {{/Users/zohar/dev/pm-flink/flink-libraries/flink-python/src/test/python/org/apache/flink/python/api/streaming}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3838: [FLINK-5886] Python API for streaming applications

[zohar-mizrahi]

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117807878
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/filecache/FileCache.java 
---
@@ -233,6 +234,20 @@ boolean holdsStillReference(String name, JobID jobId) {
//  Utilities
// 

 
+   /**
+* Remove a given path recursively if exists. The path can be a 
filepath or directory.
+*
+* @param path  The root path to remove.
+* @throws IOException
+* @throws URISyntaxException
+*/
+   public static void clearPath(String path) throws IOException, 
URISyntaxException {
--- End diff --

My bad! I don't use it anymore.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6662) ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job

[Jared Stehler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019914#comment-16019914
 ] 

Jared Stehler commented on FLINK-6662:
--

Yup, was just booting a new flink on 1.3 over an existing 1.2 setup, so the 
system was attempting recovery on startup. I can clear out the zk references 
and redeploy.


> ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job
> --
>
> Key: FLINK-6662
> URL: https://issues.apache.org/jira/browse/FLINK-6662
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Mesos, State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Jared Stehler
>
> Running flink mesos on 1.3-release branch, I'm seeing the following error on 
> appmaster startup:
> {noformat}
> 2017-05-22 15:32:45.946 [flink-akka.actor.default-dispatcher-17] WARN  
> o.a.flink.mesos.runtime.clusterframework.MesosJobManager  - Failed to recover 
> job 088027410f1a628e7dfc59dc23df3ded.
> java.lang.Exception: Failed to retrieve the submitted job graph from state 
> handle.
> at 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:186)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JobManager.scala:536)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply$mcV$sp(JobManager.scala:533)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.flink.runtime.jobgraph.tasks.JobSnapshottingSettings
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at 
> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:64)
> at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
> at 
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at 
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:305)
> 

[jira] [Commented] (FLINK-6662) ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019911#comment-16019911
 ] 

Stephan Ewen commented on FLINK-6662:
-

[~trohrm...@apache.org] What do you think about this issue?

> ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job
> --
>
> Key: FLINK-6662
> URL: https://issues.apache.org/jira/browse/FLINK-6662
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Mesos, State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Jared Stehler
>
> Running flink mesos on 1.3-release branch, I'm seeing the following error on 
> appmaster startup:
> {noformat}
> 2017-05-22 15:32:45.946 [flink-akka.actor.default-dispatcher-17] WARN  
> o.a.flink.mesos.runtime.clusterframework.MesosJobManager  - Failed to recover 
> job 088027410f1a628e7dfc59dc23df3ded.
> java.lang.Exception: Failed to retrieve the submitted job graph from state 
> handle.
> at 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:186)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JobManager.scala:536)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply$mcV$sp(JobManager.scala:533)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.flink.runtime.jobgraph.tasks.JobSnapshottingSettings
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at 
> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:64)
> at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
> at 
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at 
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:305)
> at 
> 

[jira] [Commented] (FLINK-6662) ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019909#comment-16019909
 ] 

Stephan Ewen commented on FLINK-6662:
-

Are you trying to let a Flink 1.3 recover a pending Job from an earlier Flink 
version via a high-availability setup?

That pattern is not supported - Framework version upgrades should go through 
savepoints, and not through ZooKeeper.
A fair point is, though, that this should be reported as an explicit exception, 
not simply a {{ClassNotFoundException}}.

> ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job
> --
>
> Key: FLINK-6662
> URL: https://issues.apache.org/jira/browse/FLINK-6662
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Mesos, State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Jared Stehler
>
> Running flink mesos on 1.3-release branch, I'm seeing the following error on 
> appmaster startup:
> {noformat}
> 2017-05-22 15:32:45.946 [flink-akka.actor.default-dispatcher-17] WARN  
> o.a.flink.mesos.runtime.clusterframework.MesosJobManager  - Failed to recover 
> job 088027410f1a628e7dfc59dc23df3ded.
> java.lang.Exception: Failed to retrieve the submitted job graph from state 
> handle.
> at 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:186)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JobManager.scala:536)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply$mcV$sp(JobManager.scala:533)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.flink.runtime.jobgraph.tasks.JobSnapshottingSettings
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at 
> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:64)
> at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
> at 
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
>   

[jira] [Commented] (FLINK-6659) RocksDBMergeIteratorTest, SavepointITCase leave temporary directories behind

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019905#comment-16019905
 ] 

Stephan Ewen commented on FLINK-6659:
-

Should we just drop the {{CommonTestUtils.createTempDirectory()}} method, to 
not encourage use of that pattern?

> RocksDBMergeIteratorTest, SavepointITCase leave temporary directories behind
> 
>
> Key: FLINK-6659
> URL: https://issues.apache.org/jira/browse/FLINK-6659
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.2.2
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> {{RocksDBMergeIteratorTest}} uses a newly created temporary directory (via 
> {{CommonTestUtils.createTempDirectory()}}) for its RocksDB instance but does 
> not delete is when finished. We should better replace this pattern with a 
> proper {{@Rule}}-based approach



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019906#comment-16019906
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117806756
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/environment/PythonStreamExecutionEnvironment.java
 ---
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.streaming.python.api.environment;
+
+import org.apache.flink.annotation.Public;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.filecache.FileCache;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.python.api.datastream.PythonDataStream;
+import 
org.apache.flink.streaming.python.api.functions.PythonGeneratorFunction;
+import 
org.apache.flink.streaming.python.api.functions.PythonIteratorFunction;
+import org.apache.flink.streaming.python.api.functions.UtilityFunctions;
+import 
org.apache.flink.streaming.python.util.serialization.PyObjectSerializer;
+import org.python.core.PyObject;
+import org.python.core.PyString;
+import org.python.core.PyInteger;
+import org.python.core.PyLong;
+import org.python.core.PyUnicode;
+import org.python.core.PyTuple;
+import org.python.core.PyObjectDerived;
+import org.python.core.PyInstance;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Paths;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Random;
+
+
+/**
+ * A thin wrapper layer over {@link StreamExecutionEnvironment}.
+ *
+ * The PythonStreamExecutionEnvironment is the context in which a 
streaming program is executed.
+ * 
+ *
+ * The environment provides methods to control the job execution (such 
as setting the parallelism
+ * or the fault tolerance/checkpointing parameters) and to interact with 
the outside world
+ * (data access).
+ */
+@Public
+public class PythonStreamExecutionEnvironment {
+   private final StreamExecutionEnvironment env;
+   private static final Logger LOG = 
LoggerFactory.getLogger(PythonStreamExecutionEnvironment.class);
+
+   /**
+* A thin wrapper layer over {@link 
StreamExecutionEnvironment#getExecutionEnvironment()}. In addition it takes
+* care for required Jython serializers registration.
+*
+* @return The python execution environment of the context in which the 
program is
+* executed.
+*/
+   public static PythonStreamExecutionEnvironment 
get_execution_environment() {
+   return new PythonStreamExecutionEnvironment();
+   }
+
+   /**
+* Creates a {@link LocalStreamEnvironment}. The local execution 
environment
+* will run the program in a multi-threaded fashion in the same JVM as 
the
+* environment was created in. The default parallelism of the local
+* environment is the number of hardware contexts (CPU cores / threads),
+* unless it was specified differently by {@link #setParallelism(int)}.
+*
+* @param configuration
+

[GitHub] flink pull request #3838: [FLINK-5886] Python API for streaming applications

[zohar-mizrahi]

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117806756
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/environment/PythonStreamExecutionEnvironment.java
 ---
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.streaming.python.api.environment;
+
+import org.apache.flink.annotation.Public;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.filecache.FileCache;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.python.api.datastream.PythonDataStream;
+import 
org.apache.flink.streaming.python.api.functions.PythonGeneratorFunction;
+import 
org.apache.flink.streaming.python.api.functions.PythonIteratorFunction;
+import org.apache.flink.streaming.python.api.functions.UtilityFunctions;
+import 
org.apache.flink.streaming.python.util.serialization.PyObjectSerializer;
+import org.python.core.PyObject;
+import org.python.core.PyString;
+import org.python.core.PyInteger;
+import org.python.core.PyLong;
+import org.python.core.PyUnicode;
+import org.python.core.PyTuple;
+import org.python.core.PyObjectDerived;
+import org.python.core.PyInstance;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Paths;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Random;
+
+
+/**
+ * A thin wrapper layer over {@link StreamExecutionEnvironment}.
+ *
+ * The PythonStreamExecutionEnvironment is the context in which a 
streaming program is executed.
+ * 
+ *
+ * The environment provides methods to control the job execution (such 
as setting the parallelism
+ * or the fault tolerance/checkpointing parameters) and to interact with 
the outside world
+ * (data access).
+ */
+@Public
+public class PythonStreamExecutionEnvironment {
+   private final StreamExecutionEnvironment env;
+   private static final Logger LOG = 
LoggerFactory.getLogger(PythonStreamExecutionEnvironment.class);
+
+   /**
+* A thin wrapper layer over {@link 
StreamExecutionEnvironment#getExecutionEnvironment()}. In addition it takes
+* care for required Jython serializers registration.
+*
+* @return The python execution environment of the context in which the 
program is
+* executed.
+*/
+   public static PythonStreamExecutionEnvironment 
get_execution_environment() {
+   return new PythonStreamExecutionEnvironment();
+   }
+
+   /**
+* Creates a {@link LocalStreamEnvironment}. The local execution 
environment
+* will run the program in a multi-threaded fashion in the same JVM as 
the
+* environment was created in. The default parallelism of the local
+* environment is the number of hardware contexts (CPU cores / threads),
+* unless it was specified differently by {@link #setParallelism(int)}.
+*
+* @param configuration
+*  Pass a custom configuration into the cluster
+* @return A local execution environment with the specified parallelism.
+*/
+   public static PythonStreamExecutionEnvironment 

[jira] [Commented] (FLINK-6656) Migrate CEP PriorityQueue to MapState

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019898#comment-16019898
 ] 

ASF GitHub Bot commented on FLINK-6656:
---

Github user dawidwys commented on the issue:

https://github.com/apache/flink/pull/3961
  
Thanks for clarification!

Z pozdrowieniami! / Cheers!

Dawid Wysakowicz

*Data/Software Engineer*

Skype: dawid_wys | Twitter: @OneMoreCoder



2017-05-22 18:01 GMT+02:00 Kostas Kloudas :

> Hi @dawidwys  ! Your understanding is
> correct in the sense that it does not leverage to the fullest the fact of
> keys being returned in order. The reason is that this is valid only in
> RocksDB state backend.
>
> But when it comes to insertion to the PriorityQueue, this can come in
> handy as inserting will be fast. In addition, as you said, we only
> deserialize the values we need, when we need them, instead of everything
> all the time.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



> Migrate CEP PriorityQueue to MapState
> -
>
> Key: FLINK-6656
> URL: https://issues.apache.org/jira/browse/FLINK-6656
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3961: [FLINK-6656] [cep] Change element PriorityQueue to MapSta...

[dawidwys]

Github user dawidwys commented on the issue:

https://github.com/apache/flink/pull/3961
  
Thanks for clarification!

Z pozdrowieniami! / Cheers!

Dawid Wysakowicz

*Data/Software Engineer*

Skype: dawid_wys | Twitter: @OneMoreCoder



2017-05-22 18:01 GMT+02:00 Kostas Kloudas :

> Hi @dawidwys  ! Your understanding is
> correct in the sense that it does not leverage to the fullest the fact of
> keys being returned in order. The reason is that this is valid only in
> RocksDB state backend.
>
> But when it comes to insertion to the PriorityQueue, this can come in
> handy as inserting will be fast. In addition, as you said, we only
> deserialize the values we need, when we need them, instead of everything
> all the time.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019874#comment-16019874
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117802052
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/datastream/PythonObjectInputStream2.java
 ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.python.api.datastream;
+
+import org.python.util.PythonObjectInputStream;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectStreamClass;
+
+/**
+ * A helper class to overcome the inability to set the serialVersionUID in 
a python user-defined
+ * function (UDF).
+ * The fact the this field is not set, results in a dynamic calculation 
of this serialVersionUID,
+ * using SHA, to make sure it is a unique number. This unique number is a 
64-bit hash of the
+ * class name, interface class names, methods, and fields. If a Python 
class inherits from a
+ * Java class, as in the case of Python UDFs, then a proxy wrapper class 
is created. Its name is
+ * constructed using the following pattern:
+ * {@code 
org.python.proxies.$$}. The {@code 
}
+ * part is increased by one in runtime, for every job submission. It 
results in different serial
+ * version UID for each run for the same Python class. Therefore, it is 
required to silently
+ * suppress the serial version UID mismatch check.
+ */
+public class PythonObjectInputStream2 extends PythonObjectInputStream {
--- End diff --

Ah didn't see that; figured it was a leftover from the development.


> Python API for streaming applications
> -
>
> Key: FLINK-5886
> URL: https://issues.apache.org/jira/browse/FLINK-5886
> Project: Flink
>  Issue Type: New Feature
>  Components: Python API
>Reporter: Zohar Mizrahi
>Assignee: Zohar Mizrahi
>
> A work in progress to provide python interface for Flink streaming APIs. The 
> core technology is based on jython and thus imposes two limitations: a. user 
> defined functions cannot use python extensions. b. the python version is 2.x
> The branch is based on Flink release 1.2.0, as can be found here:
> https://github.com/zohar-pm/flink/tree/python-streaming
> In order to test it, someone can use IntelliJ IDE. Assuming IntelliJ was 
> setup properly (see: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/ide_setup.html),
>  one can run/debug {{org.apache.flink.python.api.PythonStreamBinderTest}}, 
> which in return will execute all the tests under 
> {{/Users/zohar/dev/pm-flink/flink-libraries/flink-python/src/test/python/org/apache/flink/python/api/streaming}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3838: [FLINK-5886] Python API for streaming applications

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117802052
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/datastream/PythonObjectInputStream2.java
 ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.python.api.datastream;
+
+import org.python.util.PythonObjectInputStream;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectStreamClass;
+
+/**
+ * A helper class to overcome the inability to set the serialVersionUID in 
a python user-defined
+ * function (UDF).
+ * The fact the this field is not set, results in a dynamic calculation 
of this serialVersionUID,
+ * using SHA, to make sure it is a unique number. This unique number is a 
64-bit hash of the
+ * class name, interface class names, methods, and fields. If a Python 
class inherits from a
+ * Java class, as in the case of Python UDFs, then a proxy wrapper class 
is created. Its name is
+ * constructed using the following pattern:
+ * {@code 
org.python.proxies.$$}. The {@code 
}
+ * part is increased by one in runtime, for every job submission. It 
results in different serial
+ * version UID for each run for the same Python class. Therefore, it is 
required to silently
+ * suppress the serial version UID mismatch check.
+ */
+public class PythonObjectInputStream2 extends PythonObjectInputStream {
--- End diff --

Ah didn't see that; figured it was a leftover from the development.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-6663) NPE on web startup when disabling web submissions

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-6663.
---
Resolution: Duplicate

> NPE on web startup when disabling web submissions
> -
>
> Key: FLINK-6663
> URL: https://issues.apache.org/jira/browse/FLINK-6663
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.3.0
>Reporter: Jared Stehler
>
> The web frontend fails to start when I disable web submissions, with a null 
> pointer for upload dir:
> {noformat}
> 2017-05-22 16:56:42.818 [main] ERROR 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils  - WebServer could not be 
> created
> java.lang.NullPointerException: null
>   at 
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
>   at 
> org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:65)
>   at 
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:385)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:140)
>   at 
> org.apache.flink.runtime.clusterframework.BootstrapTools.startWebMonitorIfConfigured(BootstrapTools.java:195)
>   at 
> org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.runPrivileged(MesosApplicationMasterRunner.java:325)
>   at 
> org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:181)
>   at 
> org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:178)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at 
> org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.run(MesosApplicationMasterRunner.java:178)
>   at 
> org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.main(MesosApplicationMasterRunner.java:139)
> {noformat}
> This seems to be due to the code explicitly setting it to null?
> {code:java}
>   final boolean webSubmitAllow = cfg.isProgramSubmitEnabled();
>   if (webSubmitAllow) {
>   // create storage for uploads
>   this.uploadDir = getUploadDir(config);
>   // the upload directory should either 1. exist and 
> writable or 2. can be created and writable
>   if (!(uploadDir.exists() && uploadDir.canWrite()) && 
> !(uploadDir.mkdir() && uploadDir.canWrite())) {
>   throw new IOException(
>   String.format("Jar upload directory %s 
> cannot be created or is not writable.",
>   uploadDir.getAbsolutePath()));
>   }
>   LOG.info("Using directory {} for web frontend JAR file 
> uploads", uploadDir);
>   }
>   else {
>   this.uploadDir = null;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6662) ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job

[Jared Stehler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jared Stehler updated FLINK-6662:
-
Description: 
Running flink mesos on 1.3-release branch, I'm seeing the following error on 
appmaster startup:
{noformat}
2017-05-22 15:32:45.946 [flink-akka.actor.default-dispatcher-17] WARN  
o.a.flink.mesos.runtime.clusterframework.MesosJobManager  - Failed to recover 
job 088027410f1a628e7dfc59dc23df3ded.
java.lang.Exception: Failed to retrieve the submitted job graph from state 
handle.
at 
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:186)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JobManager.scala:536)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply$mcV$sp(JobManager.scala:533)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.runtime.jobgraph.tasks.JobSnapshottingSettings
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at 
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:64)
at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at 
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:305)
at 
org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
at 
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:184)
... 15 common frames omitted
{noformat}


  was:
Running flink mesos on 1.3-release branch, I'm seeing the following error on 
appmaster startup:

2017-05-22 15:32:45.946 [flink-akka.actor.default-dispatcher-17] WARN  
o.a.flink.mesos.runtime.clusterframework.MesosJobManager  - Failed to recover 
job 088027410f1a628e7dfc59dc23df3ded.
java.lang.Exception: Failed to retrieve the submitted job graph from state 
handle.
at 

[jira] [Updated] (FLINK-6663) NPE on web startup when disabling web submissions

[Jared Stehler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jared Stehler updated FLINK-6663:
-
Description: 
The web frontend fails to start when I disable web submissions, with a null 
pointer for upload dir:
{noformat}
2017-05-22 16:56:42.818 [main] ERROR 
org.apache.flink.runtime.webmonitor.WebMonitorUtils  - WebServer could not be 
created
java.lang.NullPointerException: null
at 
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
at 
org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:65)
at 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:385)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:140)
at 
org.apache.flink.runtime.clusterframework.BootstrapTools.startWebMonitorIfConfigured(BootstrapTools.java:195)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.runPrivileged(MesosApplicationMasterRunner.java:325)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:181)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:178)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.run(MesosApplicationMasterRunner.java:178)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.main(MesosApplicationMasterRunner.java:139)
{noformat}

This seems to be due to the code explicitly setting it to null?
{code:java}
final boolean webSubmitAllow = cfg.isProgramSubmitEnabled();
if (webSubmitAllow) {
// create storage for uploads
this.uploadDir = getUploadDir(config);
// the upload directory should either 1. exist and 
writable or 2. can be created and writable
if (!(uploadDir.exists() && uploadDir.canWrite()) && 
!(uploadDir.mkdir() && uploadDir.canWrite())) {
throw new IOException(
String.format("Jar upload directory %s 
cannot be created or is not writable.",
uploadDir.getAbsolutePath()));
}
LOG.info("Using directory {} for web frontend JAR file 
uploads", uploadDir);
}
else {
this.uploadDir = null;
}
{code}

  was:
The web frontend fails to start when I disable web submissions, with a null 
pointer for upload dir:

2017-05-22 16:56:42.818 [main] ERROR 
org.apache.flink.runtime.webmonitor.WebMonitorUtils  - WebServer could not be 
created
java.lang.NullPointerException: null
at 
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
at 
org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:65)
at 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:385)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:140)
at 
org.apache.flink.runtime.clusterframework.BootstrapTools.startWebMonitorIfConfigured(BootstrapTools.java:195)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.runPrivileged(MesosApplicationMasterRunner.java:325)
at 

[jira] [Updated] (FLINK-6663) NPE on web startup when disabling web submissions

[Jared Stehler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jared Stehler updated FLINK-6663:
-
Description: 
The web frontend fails to start when I disable web submissions, with a null 
pointer for upload dir:

2017-05-22 16:56:42.818 [main] ERROR 
org.apache.flink.runtime.webmonitor.WebMonitorUtils  - WebServer could not be 
created
java.lang.NullPointerException: null
at 
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
at 
org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:65)
at 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:385)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:140)
at 
org.apache.flink.runtime.clusterframework.BootstrapTools.startWebMonitorIfConfigured(BootstrapTools.java:195)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.runPrivileged(MesosApplicationMasterRunner.java:325)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:181)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:178)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.run(MesosApplicationMasterRunner.java:178)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.main(MesosApplicationMasterRunner.java:139)

This seems to be due to the code explicitly setting it to null?
{code:java}
final boolean webSubmitAllow = cfg.isProgramSubmitEnabled();
if (webSubmitAllow) {
// create storage for uploads
this.uploadDir = getUploadDir(config);
// the upload directory should either 1. exist and 
writable or 2. can be created and writable
if (!(uploadDir.exists() && uploadDir.canWrite()) && 
!(uploadDir.mkdir() && uploadDir.canWrite())) {
throw new IOException(
String.format("Jar upload directory %s 
cannot be created or is not writable.",
uploadDir.getAbsolutePath()));
}
LOG.info("Using directory {} for web frontend JAR file 
uploads", uploadDir);
}
else {
this.uploadDir = null;
}
{code}

  was:
The web frontend fails to start when I disable web submissions, with a null 
pointer for upload dir:

2017-05-22 16:56:42.818 [main] ERROR 
org.apache.flink.runtime.webmonitor.WebMonitorUtils  - WebServer could not be 
created
java.lang.NullPointerException: null
at 
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
at 
org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:65)
at 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:385)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:140)
at 
org.apache.flink.runtime.clusterframework.BootstrapTools.startWebMonitorIfConfigured(BootstrapTools.java:195)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.runPrivileged(MesosApplicationMasterRunner.java:325)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:181)
at 

[jira] [Created] (FLINK-6663) NPE on web startup when disabling web submissions

[Jared Stehler (JIRA)]

Jared Stehler created FLINK-6663:


 Summary: NPE on web startup when disabling web submissions
 Key: FLINK-6663
 URL: https://issues.apache.org/jira/browse/FLINK-6663
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Affects Versions: 1.3.0
Reporter: Jared Stehler


The web frontend fails to start when I disable web submissions, with a null 
pointer for upload dir:

2017-05-22 16:56:42.818 [main] ERROR 
org.apache.flink.runtime.webmonitor.WebMonitorUtils  - WebServer could not be 
created
java.lang.NullPointerException: null
at 
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
at 
org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrap.(WebFrontendBootstrap.java:65)
at 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:385)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:140)
at 
org.apache.flink.runtime.clusterframework.BootstrapTools.startWebMonitorIfConfigured(BootstrapTools.java:195)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.runPrivileged(MesosApplicationMasterRunner.java:325)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:181)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner$1.call(MesosApplicationMasterRunner.java:178)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.run(MesosApplicationMasterRunner.java:178)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner.main(MesosApplicationMasterRunner.java:139)

This seems to be due to the code explicitly setting it to null?

final boolean webSubmitAllow = cfg.isProgramSubmitEnabled();
if (webSubmitAllow) {
// create storage for uploads
this.uploadDir = getUploadDir(config);
// the upload directory should either 1. exist and 
writable or 2. can be created and writable
if (!(uploadDir.exists() && uploadDir.canWrite()) && 
!(uploadDir.mkdir() && uploadDir.canWrite())) {
throw new IOException(
String.format("Jar upload directory %s 
cannot be created or is not writable.",
uploadDir.getAbsolutePath()));
}
LOG.info("Using directory {} for web frontend JAR file 
uploads", uploadDir);
}
else {
this.uploadDir = null;
}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6632) Fix parameter case sensitive error for test passing/rejecting filter API

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-6632.
-
   Resolution: Fixed
Fix Version/s: 1.4.0
   1.3.0

Fixed in 1.4: 9a9e193a1edf335c89804dc3643fc02681570821
Fixed in 1.3: 95fce15483fe6b12adc465a0b709362b8ea7fd0c

> Fix parameter case sensitive error for test passing/rejecting filter API
> 
>
> Key: FLINK-6632
> URL: https://issues.apache.org/jira/browse/FLINK-6632
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
> Fix For: 1.3.0, 1.4.0
>
>
> TableAPI {{testAllPassingFilter}}:
> {code} 
>val t = util.addTable[(Int, Long, String)]('int, 'long, 'string)
> val resScala = t.filter(Literal(true)).select('int as 'myInt, 'string)
> val resJava = t.filter("TrUe").select("int as myInt, string")
> {code}
> We got error:
> {code}
> org.apache.flink.table.api.ValidationException: Cannot resolve [TrUe] given 
> input [int, long, string].
> {code}
> The error is caused by :
> {code}
> lazy val boolLiteral: PackratParser[Expression] = ("true" | "false") ^^ {
> str => Literal(str.toBoolean)
>   }
> {code}
> I want improve the method as follow:
> {code}
>  lazy val boolLiteral: PackratParser[Expression] =
> ("(t|T)(r|R)(u|U)(e|E)".r | "(f|F)(a|A)(l|L)(s|S)(e|E)".r) ^^ { str => 
> Literal(str.toBoolean)}
> {code}
> Is there any drawback to this improvement? Welcome anyone feedback ?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3944: [FLINK-6632][table]Improved the method BoolLiteral...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3944


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6632) Fix parameter case sensitive error for test passing/rejecting filter API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019821#comment-16019821
 ] 

ASF GitHub Bot commented on FLINK-6632:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3944


> Fix parameter case sensitive error for test passing/rejecting filter API
> 
>
> Key: FLINK-6632
> URL: https://issues.apache.org/jira/browse/FLINK-6632
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> TableAPI {{testAllPassingFilter}}:
> {code} 
>val t = util.addTable[(Int, Long, String)]('int, 'long, 'string)
> val resScala = t.filter(Literal(true)).select('int as 'myInt, 'string)
> val resJava = t.filter("TrUe").select("int as myInt, string")
> {code}
> We got error:
> {code}
> org.apache.flink.table.api.ValidationException: Cannot resolve [TrUe] given 
> input [int, long, string].
> {code}
> The error is caused by :
> {code}
> lazy val boolLiteral: PackratParser[Expression] = ("true" | "false") ^^ {
> str => Literal(str.toBoolean)
>   }
> {code}
> I want improve the method as follow:
> {code}
>  lazy val boolLiteral: PackratParser[Expression] =
> ("(t|T)(r|R)(u|U)(e|E)".r | "(f|F)(a|A)(l|L)(s|S)(e|E)".r) ^^ { str => 
> Literal(str.toBoolean)}
> {code}
> Is there any drawback to this improvement? Welcome anyone feedback ?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019811#comment-16019811
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117793843
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/environment/PythonStreamExecutionEnvironment.java
 ---
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.streaming.python.api.environment;
+
+import org.apache.flink.annotation.Public;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.filecache.FileCache;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.python.api.datastream.PythonDataStream;
+import 
org.apache.flink.streaming.python.api.functions.PythonGeneratorFunction;
+import 
org.apache.flink.streaming.python.api.functions.PythonIteratorFunction;
+import org.apache.flink.streaming.python.api.functions.UtilityFunctions;
+import 
org.apache.flink.streaming.python.util.serialization.PyObjectSerializer;
+import org.python.core.PyObject;
+import org.python.core.PyString;
+import org.python.core.PyInteger;
+import org.python.core.PyLong;
+import org.python.core.PyUnicode;
+import org.python.core.PyTuple;
+import org.python.core.PyObjectDerived;
+import org.python.core.PyInstance;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Paths;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Random;
+
+
+/**
+ * A thin wrapper layer over {@link StreamExecutionEnvironment}.
+ *
+ * The PythonStreamExecutionEnvironment is the context in which a 
streaming program is executed.
+ * 
+ *
+ * The environment provides methods to control the job execution (such 
as setting the parallelism
+ * or the fault tolerance/checkpointing parameters) and to interact with 
the outside world
+ * (data access).
+ */
+@Public
+public class PythonStreamExecutionEnvironment {
+   private final StreamExecutionEnvironment env;
+   private static final Logger LOG = 
LoggerFactory.getLogger(PythonStreamExecutionEnvironment.class);
+
+   /**
+* A thin wrapper layer over {@link 
StreamExecutionEnvironment#getExecutionEnvironment()}. In addition it takes
+* care for required Jython serializers registration.
+*
+* @return The python execution environment of the context in which the 
program is
+* executed.
+*/
+   public static PythonStreamExecutionEnvironment 
get_execution_environment() {
+   return new PythonStreamExecutionEnvironment();
+   }
+
+   /**
+* Creates a {@link LocalStreamEnvironment}. The local execution 
environment
+* will run the program in a multi-threaded fashion in the same JVM as 
the
+* environment was created in. The default parallelism of the local
+* environment is the number of hardware contexts (CPU cores / threads),
+* unless it was specified differently by {@link #setParallelism(int)}.
+*
+* @param configuration
+

[GitHub] flink pull request #3838: [FLINK-5886] Python API for streaming applications

[zohar-mizrahi]

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117793843
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/environment/PythonStreamExecutionEnvironment.java
 ---
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.streaming.python.api.environment;
+
+import org.apache.flink.annotation.Public;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.filecache.FileCache;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.python.api.datastream.PythonDataStream;
+import 
org.apache.flink.streaming.python.api.functions.PythonGeneratorFunction;
+import 
org.apache.flink.streaming.python.api.functions.PythonIteratorFunction;
+import org.apache.flink.streaming.python.api.functions.UtilityFunctions;
+import 
org.apache.flink.streaming.python.util.serialization.PyObjectSerializer;
+import org.python.core.PyObject;
+import org.python.core.PyString;
+import org.python.core.PyInteger;
+import org.python.core.PyLong;
+import org.python.core.PyUnicode;
+import org.python.core.PyTuple;
+import org.python.core.PyObjectDerived;
+import org.python.core.PyInstance;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Paths;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Random;
+
+
+/**
+ * A thin wrapper layer over {@link StreamExecutionEnvironment}.
+ *
+ * The PythonStreamExecutionEnvironment is the context in which a 
streaming program is executed.
+ * 
+ *
+ * The environment provides methods to control the job execution (such 
as setting the parallelism
+ * or the fault tolerance/checkpointing parameters) and to interact with 
the outside world
+ * (data access).
+ */
+@Public
+public class PythonStreamExecutionEnvironment {
+   private final StreamExecutionEnvironment env;
+   private static final Logger LOG = 
LoggerFactory.getLogger(PythonStreamExecutionEnvironment.class);
+
+   /**
+* A thin wrapper layer over {@link 
StreamExecutionEnvironment#getExecutionEnvironment()}. In addition it takes
+* care for required Jython serializers registration.
+*
+* @return The python execution environment of the context in which the 
program is
+* executed.
+*/
+   public static PythonStreamExecutionEnvironment 
get_execution_environment() {
+   return new PythonStreamExecutionEnvironment();
+   }
+
+   /**
+* Creates a {@link LocalStreamEnvironment}. The local execution 
environment
+* will run the program in a multi-threaded fashion in the same JVM as 
the
+* environment was created in. The default parallelism of the local
+* environment is the number of hardware contexts (CPU cores / threads),
+* unless it was specified differently by {@link #setParallelism(int)}.
+*
+* @param configuration
+*  Pass a custom configuration into the cluster
+* @return A local execution environment with the specified parallelism.
+*/
+   public static PythonStreamExecutionEnvironment 

[jira] [Commented] (FLINK-6632) Fix parameter case sensitive error for test passing/rejecting filter API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019806#comment-16019806
 ] 

ASF GitHub Bot commented on FLINK-6632:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/3944
  
Thanks @sunjincheng121. Yes, I think a keyword is a nicer approach. I will 
add it and merge your tests.


> Fix parameter case sensitive error for test passing/rejecting filter API
> 
>
> Key: FLINK-6632
> URL: https://issues.apache.org/jira/browse/FLINK-6632
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> TableAPI {{testAllPassingFilter}}:
> {code} 
>val t = util.addTable[(Int, Long, String)]('int, 'long, 'string)
> val resScala = t.filter(Literal(true)).select('int as 'myInt, 'string)
> val resJava = t.filter("TrUe").select("int as myInt, string")
> {code}
> We got error:
> {code}
> org.apache.flink.table.api.ValidationException: Cannot resolve [TrUe] given 
> input [int, long, string].
> {code}
> The error is caused by :
> {code}
> lazy val boolLiteral: PackratParser[Expression] = ("true" | "false") ^^ {
> str => Literal(str.toBoolean)
>   }
> {code}
> I want improve the method as follow:
> {code}
>  lazy val boolLiteral: PackratParser[Expression] =
> ("(t|T)(r|R)(u|U)(e|E)".r | "(f|F)(a|A)(l|L)(s|S)(e|E)".r) ^^ { str => 
> Literal(str.toBoolean)}
> {code}
> Is there any drawback to this improvement? Welcome anyone feedback ?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3944: [FLINK-6632][table]Improved the method BoolLiteral of Exp...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/3944
  
Thanks @sunjincheng121. Yes, I think a keyword is a nicer approach. I will 
add it and merge your tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019776#comment-16019776
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117788892
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/datastream/PythonObjectInputStream2.java
 ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.python.api.datastream;
+
+import org.python.util.PythonObjectInputStream;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectStreamClass;
+
+/**
+ * A helper class to overcome the inability to set the serialVersionUID in 
a python user-defined
+ * function (UDF).
+ * The fact the this field is not set, results in a dynamic calculation 
of this serialVersionUID,
+ * using SHA, to make sure it is a unique number. This unique number is a 
64-bit hash of the
+ * class name, interface class names, methods, and fields. If a Python 
class inherits from a
+ * Java class, as in the case of Python UDFs, then a proxy wrapper class 
is created. Its name is
+ * constructed using the following pattern:
+ * {@code 
org.python.proxies.$$}. The {@code 
}
+ * part is increased by one in runtime, for every job submission. It 
results in different serial
+ * version UID for each run for the same Python class. Therefore, it is 
required to silently
+ * suppress the serial version UID mismatch check.
+ */
+public class PythonObjectInputStream2 extends PythonObjectInputStream {
--- End diff --

How would you suggest to call it? It extends the `PythonObjectInputStream`.


> Python API for streaming applications
> -
>
> Key: FLINK-5886
> URL: https://issues.apache.org/jira/browse/FLINK-5886
> Project: Flink
>  Issue Type: New Feature
>  Components: Python API
>Reporter: Zohar Mizrahi
>Assignee: Zohar Mizrahi
>
> A work in progress to provide python interface for Flink streaming APIs. The 
> core technology is based on jython and thus imposes two limitations: a. user 
> defined functions cannot use python extensions. b. the python version is 2.x
> The branch is based on Flink release 1.2.0, as can be found here:
> https://github.com/zohar-pm/flink/tree/python-streaming
> In order to test it, someone can use IntelliJ IDE. Assuming IntelliJ was 
> setup properly (see: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/ide_setup.html),
>  one can run/debug {{org.apache.flink.python.api.PythonStreamBinderTest}}, 
> which in return will execute all the tests under 
> {{/Users/zohar/dev/pm-flink/flink-libraries/flink-python/src/test/python/org/apache/flink/python/api/streaming}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3838: [FLINK-5886] Python API for streaming applications

[zohar-mizrahi]

Github user zohar-mizrahi commented on a diff in the pull request:

https://github.com/apache/flink/pull/3838#discussion_r117788892
  
--- Diff: 
flink-libraries/flink-streaming-python/src/main/java/org/apache/flink/streaming/python/api/datastream/PythonObjectInputStream2.java
 ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.python.api.datastream;
+
+import org.python.util.PythonObjectInputStream;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectStreamClass;
+
+/**
+ * A helper class to overcome the inability to set the serialVersionUID in 
a python user-defined
+ * function (UDF).
+ * The fact the this field is not set, results in a dynamic calculation 
of this serialVersionUID,
+ * using SHA, to make sure it is a unique number. This unique number is a 
64-bit hash of the
+ * class name, interface class names, methods, and fields. If a Python 
class inherits from a
+ * Java class, as in the case of Python UDFs, then a proxy wrapper class 
is created. Its name is
+ * constructed using the following pattern:
+ * {@code 
org.python.proxies.$$}. The {@code 
}
+ * part is increased by one in runtime, for every job submission. It 
results in different serial
+ * version UID for each run for the same Python class. Therefore, it is 
required to silently
+ * suppress the serial version UID mismatch check.
+ */
+public class PythonObjectInputStream2 extends PythonObjectInputStream {
--- End diff --

How would you suggest to call it? It extends the `PythonObjectInputStream`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6320) Flakey JobManagerHAJobGraphRecoveryITCase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019760#comment-16019760
 ] 

ASF GitHub Bot commented on FLINK-6320:
---

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/3966

[FLINK-6320] fix unit test failing sometimes when deleting a temp dir

use a JUnit rule with a `TemporaryFolder` instead which should fix this 
(alternatively, catching the exception may have fixed this symptom but using a 
temporary folder should be even more save in case a real concurrency issue was 
present)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-6320

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3966.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3966


commit 5b440c4084076c8086d888c4d5439c2e5f65bdf6
Author: Nico Kruber 
Date:   2017-05-22T14:31:08Z

[FLINK-6320] fix unit test failing sometimes when deleting a temp directory




> Flakey JobManagerHAJobGraphRecoveryITCase
> -
>
> Key: FLINK-6320
> URL: https://issues.apache.org/jira/browse/FLINK-6320
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>  Labels: test-stability
>
> it looks as if there is a race condition in the cleanup of 
> {{JobManagerHAJobGraphRecoveryITCase}}.
> {code}
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 50.271 sec 
> <<< FAILURE! - in 
> org.apache.flink.test.recovery.JobManagerHAJobGraphRecoveryITCase
> testJobPersistencyWhenJobManagerShutdown(org.apache.flink.test.recovery.JobManagerHAJobGraphRecoveryITCase)
>   Time elapsed: 0.129 sec  <<< ERROR!
> java.io.FileNotFoundException: File does not exist: 
> /tmp/9b63934b-789d-428c-aa9e-47d5d8fa1e32/recovery/submittedJobGraphf763d61fba47
>   at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275)
>   at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
>   at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
>   at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
>   at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
>   at 
> org.apache.flink.test.recovery.JobManagerHAJobGraphRecoveryITCase.cleanUp(JobManagerHAJobGraphRecoveryITCase.java:112)
> {code}
> Full log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/223124016/log.txt
> Maybe a rule-based temporary directory is a better solution:
> {code:java}
>   @Rule
>   public TemporaryFolder tempFolder = new TemporaryFolder();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3966: [FLINK-6320] fix unit test failing sometimes when ...

[NicoK]

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/3966

[FLINK-6320] fix unit test failing sometimes when deleting a temp dir

use a JUnit rule with a `TemporaryFolder` instead which should fix this 
(alternatively, catching the exception may have fixed this symptom but using a 
temporary folder should be even more save in case a real concurrency issue was 
present)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-6320

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3966.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3966


commit 5b440c4084076c8086d888c4d5439c2e5f65bdf6
Author: Nico Kruber 
Date:   2017-05-22T14:31:08Z

[FLINK-6320] fix unit test failing sometimes when deleting a temp directory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-6537) Umbrella issue for fixes to incremental snapshots

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019717#comment-16019717
 ] 

Stefan Richter edited comment on FLINK-6537 at 5/22/17 4:19 PM:


Which exception is that exactly? And how does it happen all the time? There is 
no JIRA for it, because I have not encountered the problem. Can this be 
reproduced somehow? Is only the first savepoint failing or all? What exceptions 
still remain after the latest fixes?

Edit: In a quick run on GCloud, I could not reproduce any problems with 
savepoints and RocksDB.


was (Author: srichter):
Which exception is that exactly? And how does it happen all the time? There is 
no JIRA for it, because I have not encountered the problem. Can this be 
reproduced somehow? Is only the first savepoint failing or all? What exceptions 
still remain after the latest fixes?

> Umbrella issue for fixes to incremental snapshots
> -
>
> Key: FLINK-6537
> URL: https://issues.apache.org/jira/browse/FLINK-6537
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
> Fix For: 1.3.0
>
>
> This issue tracks ongoing fixes in the incremental checkpointing feature for 
> the 1.3 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6662) ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job

[Jared Stehler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jared Stehler updated FLINK-6662:
-
Description: 
Running flink mesos on 1.3-release branch, I'm seeing the following error on 
appmaster startup:

2017-05-22 15:32:45.946 [flink-akka.actor.default-dispatcher-17] WARN  
o.a.flink.mesos.runtime.clusterframework.MesosJobManager  - Failed to recover 
job 088027410f1a628e7dfc59dc23df3ded.
java.lang.Exception: Failed to retrieve the submitted job graph from state 
handle.
at 
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:186)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JobManager.scala:536)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1$$anonfun$apply$mcV$sp$1.apply(JobManager.scala:533)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply$mcV$sp(JobManager.scala:533)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$1.apply(JobManager.scala:529)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.runtime.jobgraph.tasks.JobSnapshottingSettings
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at 
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:64)
at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at 
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:305)
at 
org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
at 
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:184)
... 15 common frames omitted


  was:
Running flink mesos on 1.3-release branch, I'm seeing the following error on 
appmaster startup:

sLast login: Sun May 21 19:03:05 on ttys005
sg 10.80.54.119%



 ~/dev/scratch/flink   release-1.3 ●  ssg 10.80.54.119   
 

[jira] [Created] (FLINK-6662) ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job

[Jared Stehler (JIRA)]

Jared Stehler created FLINK-6662:


 Summary: ClassNotFoundException: 
o.a.f.r.j.t.JobSnapshottingSettings recovering job
 Key: FLINK-6662
 URL: https://issues.apache.org/jira/browse/FLINK-6662
 Project: Flink
  Issue Type: Bug
  Components: JobManager, Mesos, State Backends, Checkpointing
Reporter: Jared Stehler


Running flink mesos on 1.3-release branch, I'm seeing the following error on 
appmaster startup:

sLast login: Sun May 21 19:03:05 on ttys005
sg 10.80.54.119%



 ~/dev/scratch/flink   release-1.3 ●  ssg 10.80.54.119   

  ✓  
11436  12:00:16 
zsh: command not found: ssg
 ~/dev/scratch/flink   release-1.3 ●  ssh 10.80.54.119   

  127 ↵  
11437  12:00:16 
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-117-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Mon May 22 15:00:39 UTC 2017

  System load:  0.0   Processes:  159
  Usage of /:   68.1% of 7.74GB   Users logged in:0
  Memory usage: 20%   IP address for eth0:10.80.54.119
  Swap usage:   0%IP address for docker0: 172.17.0.1

  Graph this data and manage this system at:
https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud

31 packages can be updated.
27 updates are security updates.

New release '16.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


Last login: Sun May 21 18:44:31 2017 from 
ip-10-80-48-143.us-west-2.compute.internal
ubuntu@ip-10-80-54-119:~$ cd /mnt/mesos/
docker/   logs/ lost+found/   
singularity-executor/ work/ 
ubuntu@ip-10-80-54-119:~$ cd /mnt/mesos/work/
meta/provisioner/ slaves/  
ubuntu@ip-10-80-54-119:~$ cd 
/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/
docker/ frameworks/ 
ubuntu@ip-10-80-54-119:~$ cd 
/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/
e87689b9-a83a-449b-9e6b-3e339ead141a-0004/ 
e87689b9-a83a-449b-9e6b-3e339ead141a-0006/ 
e87689b9-a83a-449b-9e6b-3e339ead141a-0008/ Singularity/ 
  
e87689b9-a83a-449b-9e6b-3e339ead141a-0005/ 
e87689b9-a83a-449b-9e6b-3e339ead141a-0007/ 
e87689b9-a83a-449b-9e6b-3e339ead141a-0009/ 
ubuntu@ip-10-80-54-119:~$ cd 
/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors/
1vn1/ 3un1/ 4un1/ 6tn1/ 6un1/ asn1/ cvn1/ hun1/ iin1/ isn1/ jsn1/ jtn1/ ltn1/ 
ntn1/ osn1/ smn1/ tsn1/ won1/ 
ubuntu@ip-10-80-54-119:~$ cd 
/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors/
ubuntu@ip-10-80-54-119:/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors$
 ls 1vn1/
runs
ubuntu@ip-10-80-54-119:/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors$
 ls 1vn1/runs/
e24faf7e-9553-4c07-8c6a-e85acdfe88af  latest
ubuntu@ip-10-80-54-119:/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors$
 find . -name "*1495467133685*"
find: 
`./asn1/runs/9b6627de-d545-4e34-87e6-f639d94afe47/pdf-service-1495313993-1495314044332-1-10.80.54.119-us_west_2c/logs':
 Permission denied
find: 
`./asn1/runs/9b6627de-d545-4e34-87e6-f639d94afe47/pdf-service-1495313993-1495314044332-1-10.80.54.119-us_west_2c/tmp':
 Permission denied
find: 
`./iin1/runs/b0bcea6d-4a88-4587-bf34-619aa627338d/deployinator-1494544026-1494956610579-1-10.80.54.119-us_west_2c/logs':
 Permission denied
find: 
`./iin1/runs/b0bcea6d-4a88-4587-bf34-619aa627338d/deployinator-1494544026-1494956610579-1-10.80.54.119-us_west_2c/tmp':
 Permission denied
find: 
`./ntn1/runs/6483530f-0e0a-4b27-ad89-dd0fede1e3c0/prometheus-1495389842-1495389843120-1-10.80.54.119-us_west_2c/storage':
 Permission denied
./cvn1/runs/11f8a647-1cad-4881-b30c-9f68a4aa1cc3/flink-mesos-1495467129-1495467133685-1-10.80.54.119-us_west_2c
ubuntu@ip-10-80-54-119:/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors$
 cd 
cvn1/runs/11f8a647-1cad-4881-b30c-9f68a4aa1cc3/flink-mesos-1495467129-1495467133685-1-10.80.54.119-us_west_2c

[jira] [Updated] (FLINK-6662) ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job

[Jared Stehler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jared Stehler updated FLINK-6662:
-
Affects Version/s: 1.3.0

> ClassNotFoundException: o.a.f.r.j.t.JobSnapshottingSettings recovering job
> --
>
> Key: FLINK-6662
> URL: https://issues.apache.org/jira/browse/FLINK-6662
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Mesos, State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Jared Stehler
>
> Running flink mesos on 1.3-release branch, I'm seeing the following error on 
> appmaster startup:
> sLast login: Sun May 21 19:03:05 on ttys005
> sg 10.80.54.119%  
>   
>   
>   
>  ~/dev/scratch/flink   release-1.3 ●  ssg 10.80.54.119 
>   
>   ✓ 
>  11436  12:00:16 
> zsh: command not found: ssg
>  ~/dev/scratch/flink   release-1.3 ●  ssh 10.80.54.119 
>   
>   127 ↵ 
>  11437  12:00:16 
> Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-117-generic x86_64)
>  * Documentation:  https://help.ubuntu.com/
>   System information as of Mon May 22 15:00:39 UTC 2017
>   System load:  0.0   Processes:  159
>   Usage of /:   68.1% of 7.74GB   Users logged in:0
>   Memory usage: 20%   IP address for eth0:10.80.54.119
>   Swap usage:   0%IP address for docker0: 172.17.0.1
>   Graph this data and manage this system at:
> https://landscape.canonical.com/
>   Get cloud support with Ubuntu Advantage Cloud Guest:
> http://www.ubuntu.com/business/services/cloud
> 31 packages can be updated.
> 27 updates are security updates.
> New release '16.04.2 LTS' available.
> Run 'do-release-upgrade' to upgrade to it.
> Last login: Sun May 21 18:44:31 2017 from 
> ip-10-80-48-143.us-west-2.compute.internal
> ubuntu@ip-10-80-54-119:~$ cd /mnt/mesos/
> docker/   logs/ lost+found/   
> singularity-executor/ work/ 
> ubuntu@ip-10-80-54-119:~$ cd /mnt/mesos/work/
> meta/provisioner/ slaves/  
> ubuntu@ip-10-80-54-119:~$ cd 
> /mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/
> docker/ frameworks/ 
> ubuntu@ip-10-80-54-119:~$ cd 
> /mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/
> e87689b9-a83a-449b-9e6b-3e339ead141a-0004/ 
> e87689b9-a83a-449b-9e6b-3e339ead141a-0006/ 
> e87689b9-a83a-449b-9e6b-3e339ead141a-0008/ Singularity/   
> 
> e87689b9-a83a-449b-9e6b-3e339ead141a-0005/ 
> e87689b9-a83a-449b-9e6b-3e339ead141a-0007/ 
> e87689b9-a83a-449b-9e6b-3e339ead141a-0009/ 
> ubuntu@ip-10-80-54-119:~$ cd 
> /mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors/
> 1vn1/ 3un1/ 4un1/ 6tn1/ 6un1/ asn1/ cvn1/ hun1/ iin1/ isn1/ jsn1/ jtn1/ ltn1/ 
> ntn1/ osn1/ smn1/ tsn1/ won1/ 
> ubuntu@ip-10-80-54-119:~$ cd 
> /mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors/
> ubuntu@ip-10-80-54-119:/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors$
>  ls 1vn1/
> runs
> ubuntu@ip-10-80-54-119:/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors$
>  ls 1vn1/runs/
> e24faf7e-9553-4c07-8c6a-e85acdfe88af  latest
> ubuntu@ip-10-80-54-119:/mnt/mesos/work/slaves/e87689b9-a83a-449b-9e6b-3e339ead141a-S13/frameworks/Singularity/executors$
>  find . -name "*1495467133685*"
> find: 
> `./asn1/runs/9b6627de-d545-4e34-87e6-f639d94afe47/pdf-service-1495313993-1495314044332-1-10.80.54.119-us_west_2c/logs':
>  Permission denied
> find: 
> `./asn1/runs/9b6627de-d545-4e34-87e6-f639d94afe47/pdf-service-1495313993-1495314044332-1-10.80.54.119-us_west_2c/tmp':
>  Permission denied
> find: 
> `./iin1/runs/b0bcea6d-4a88-4587-bf34-619aa627338d/deployinator-1494544026-1494956610579-1-10.80.54.119-us_west_2c/logs':
>  Permission denied
> find: 
> `./iin1/runs/b0bcea6d-4a88-4587-bf34-619aa627338d/deployinator-1494544026-1494956610579-1-10.80.54.119-us_west_2c/tmp':
>  Permission denied
> find: 
> `./ntn1/runs/6483530f-0e0a-4b27-ad89-dd0fede1e3c0/prometheus-1495389842-1495389843120-1-10.80.54.119-us_west_2c/storage':
>  Permission denied
> 

[GitHub] flink issue #3961: [FLINK-6656] [cep] Change element PriorityQueue to MapSta...

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/3961
  
Hi @dawidwys ! Your understanding is correct in the sense that it does not 
leverage to the fullest the fact of keys being returned in order. The reason is 
that this is valid only in RocksDB state backend. 

But when it comes to insertion to the PriorityQueue, this can come in handy 
as inserting will be fast. In addition, as you said, we only deserialize the 
values we need, when we need them, instead of everything all the time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6656) Migrate CEP PriorityQueue to MapState

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019737#comment-16019737
 ] 

ASF GitHub Bot commented on FLINK-6656:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/3961
  
Hi @dawidwys ! Your understanding is correct in the sense that it does not 
leverage to the fullest the fact of keys being returned in order. The reason is 
that this is valid only in RocksDB state backend. 

But when it comes to insertion to the PriorityQueue, this can come in handy 
as inserting will be fast. In addition, as you said, we only deserialize the 
values we need, when we need them, instead of everything all the time.


> Migrate CEP PriorityQueue to MapState
> -
>
> Key: FLINK-6656
> URL: https://issues.apache.org/jira/browse/FLINK-6656
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6482) Add nested serializers into configuration snapshots of composite serializers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019735#comment-16019735
 ] 

ASF GitHub Bot commented on FLINK-6482:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3937
  
Thanks a lot for the quick review and merge!!


> Add nested serializers into configuration snapshots of composite serializers
> 
>
> Key: FLINK-6482
> URL: https://issues.apache.org/jira/browse/FLINK-6482
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing, Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.3.0
>
>
> Currently, the composite serializers' configuration snapshots only wrap the 
> config snapshots of nested serializers.
> We should also consider adding serialization of the nested serializers into 
> the config snapshot, so that in the case where only some nested serializer 
> cannot be loaded (class missing / implementation changed), we can also 
> provide a path for serializer upgrades.
> This applies for all composite serializers that have nested serializers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3937: [FLINK-6482] [core] Add nested serializers to config snap...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3937
  
Thanks a lot for the quick review and merge!!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6038) Add deep links to Apache Bahir Flink streaming connector documentations

[David Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019728#comment-16019728
 ] 

David Anderson commented on FLINK-6038:
---

I can take this ticket if you like.
I hope FLINK-6660 can be merged first. 


> Add deep links to Apache Bahir Flink streaming connector documentations
> ---
>
> Key: FLINK-6038
> URL: https://issues.apache.org/jira/browse/FLINK-6038
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>
> Recently, the Bahir documentation for Flink streaming connectors in Bahir was 
> added to Bahir's website: BAHIR-90.
> We should add deep links to the individual Bahir connector dos under 
> {{/dev/connectors/overview}}, instead of just shallow links to the source 
> {{README.md}} s in the community ecosystem page.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6656) Migrate CEP PriorityQueue to MapState

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019721#comment-16019721
 ] 

ASF GitHub Bot commented on FLINK-6656:
---

Github user dawidwys commented on the issue:

https://github.com/apache/flink/pull/3961
  
Just a question from my side to understand this change better. Does it 
really leverage the fact of keys being returned in order? To evaluate 
`getSortedTimestamps` you need to iterate whole keySet anyway.

I see though it benefits from laziness of value deserialization and 
partitioning of the incoming events (by timestamp). 

Is my understanding correct?


> Migrate CEP PriorityQueue to MapState
> -
>
> Key: FLINK-6656
> URL: https://issues.apache.org/jira/browse/FLINK-6656
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3961: [FLINK-6656] [cep] Change element PriorityQueue to MapSta...

[dawidwys]

Github user dawidwys commented on the issue:

https://github.com/apache/flink/pull/3961
  
Just a question from my side to understand this change better. Does it 
really leverage the fact of keys being returned in order? To evaluate 
`getSortedTimestamps` you need to iterate whole keySet anyway.

I see though it benefits from laziness of value deserialization and 
partitioning of the incoming events (by timestamp). 

Is my understanding correct?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6328) Savepoints must not be counted as retained checkpoints

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019720#comment-16019720
 ] 

ASF GitHub Bot commented on FLINK-6328:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/3965

[FLINK-6328] [chkPts] Don't add savepoints to CompletedCheckpointStore

The lifecycle of savepoints is not managed by the CheckpointCoordinator and 
fully
in the hand of the user. Therefore, the CheckpointCoordinator cannot rely 
on them
when trying to recover from failures. E.g. a user moving a savepoint 
shortly before
a failure could completely break Flink's recovery mechanism because Flink 
cannot
skip failed checkpoints when recovering.

Therefore, until Flink is able to skip failed checkpoints when recovering, 
we should
not add savepoints to the CompletedCheckpointStore which is used to 
retrieve checkpoint
for recovery. The distinction of a savepoint is done on the basis of the
CheckpointProperties (CheckpointProperties.STANDARD_SAVEPOINT).

cc @rmetzger 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixSavepointHandling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3965.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3965


commit 9c069ad80d66f03a0f90c8ba1a780cbba111896e
Author: Till Rohrmann 
Date:   2017-05-22T15:41:14Z

[FLINK-6328] [chkPts] Don't add savepoints to CompletedCheckpointStore

The lifecycle of savepoints is not managed by the CheckpointCoordinator and 
fully
in the hand of the user. Therefore, the CheckpointCoordinator cannot rely 
on them
when trying to recover from failures. E.g. a user moving a savepoint 
shortly before
a failure could completely break Flink's recovery mechanism because Flink 
cannot
skip failed checkpoints when recovering.

Therefore, until Flink is able to skip failed checkpoints when recovering, 
we should
not add savepoints to the CompletedCheckpointStore which is used to 
retrieve checkpoint
for recovery. The distinction of a savepoint is done on the basis of the
CheckpointProperties (CheckpointProperties.STANDARD_SAVEPOINT).




> Savepoints must not be counted as retained checkpoints
> --
>
> Key: FLINK-6328
> URL: https://issues.apache.org/jira/browse/FLINK-6328
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.3.0, 1.4.0
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.3.0, 1.2.2
>
>
> The Checkpoint Store retains the *n* latest checkpoints.
> Savepoints are counted as well, meaning that for settings with 1 retained 
> checkpoint, there are sometimes no retained checkpoints at all, only a 
> savepoint.
> That is dangerous, because savepoints must be assumed to disappear at any 
> point in time - their lifecycle is out of control of the 
> CheckpointCoordinator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3965: [FLINK-6328] [chkPts] Don't add savepoints to Comp...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/3965

[FLINK-6328] [chkPts] Don't add savepoints to CompletedCheckpointStore

The lifecycle of savepoints is not managed by the CheckpointCoordinator and 
fully
in the hand of the user. Therefore, the CheckpointCoordinator cannot rely 
on them
when trying to recover from failures. E.g. a user moving a savepoint 
shortly before
a failure could completely break Flink's recovery mechanism because Flink 
cannot
skip failed checkpoints when recovering.

Therefore, until Flink is able to skip failed checkpoints when recovering, 
we should
not add savepoints to the CompletedCheckpointStore which is used to 
retrieve checkpoint
for recovery. The distinction of a savepoint is done on the basis of the
CheckpointProperties (CheckpointProperties.STANDARD_SAVEPOINT).

cc @rmetzger 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixSavepointHandling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3965.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3965


commit 9c069ad80d66f03a0f90c8ba1a780cbba111896e
Author: Till Rohrmann 
Date:   2017-05-22T15:41:14Z

[FLINK-6328] [chkPts] Don't add savepoints to CompletedCheckpointStore

The lifecycle of savepoints is not managed by the CheckpointCoordinator and 
fully
in the hand of the user. Therefore, the CheckpointCoordinator cannot rely 
on them
when trying to recover from failures. E.g. a user moving a savepoint 
shortly before
a failure could completely break Flink's recovery mechanism because Flink 
cannot
skip failed checkpoints when recovering.

Therefore, until Flink is able to skip failed checkpoints when recovering, 
we should
not add savepoints to the CompletedCheckpointStore which is used to 
retrieve checkpoint
for recovery. The distinction of a savepoint is done on the basis of the
CheckpointProperties (CheckpointProperties.STANDARD_SAVEPOINT).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-6537) Umbrella issue for fixes to incremental snapshots

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019717#comment-16019717
 ] 

Stefan Richter edited comment on FLINK-6537 at 5/22/17 3:42 PM:


Which exception is that exactly? And how does it happen all the time? There is 
no JIRA for it, because I have not encountered the problem. Can this be 
reproduced somehow? Is only the first savepoint failing or all? What exceptions 
still remain after the latest fixes?


was (Author: srichter):
Which exception is that exactly? And how does it happen all the time? There is 
no JIRA for it, because I have not encountered the problem. Can this be 
reproduced somehow?

> Umbrella issue for fixes to incremental snapshots
> -
>
> Key: FLINK-6537
> URL: https://issues.apache.org/jira/browse/FLINK-6537
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
> Fix For: 1.3.0
>
>
> This issue tracks ongoing fixes in the incremental checkpointing feature for 
> the 1.3 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6660) expand the streaming connectors overview page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019718#comment-16019718
 ] 

ASF GitHub Bot commented on FLINK-6660:
---

GitHub user alpinegizmo opened a pull request:

https://github.com/apache/flink/pull/3964

[FLINK-6660][docs] expand the connectors overview page

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alpinegizmo/flink 6660-connectors-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3964.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3964


commit d02b1da7ca570e1fc369595d91d7df1b2230918d
Author: David Anderson 
Date:   2017-05-22T15:39:34Z

[FLINK-6660][docs] expand the connectors overview page




> expand the streaming connectors overview page 
> --
>
> Key: FLINK-6660
> URL: https://issues.apache.org/jira/browse/FLINK-6660
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Streaming Connectors
>Affects Versions: 1.3.0, 1.4.0
>Reporter: David Anderson
>Assignee: David Anderson
>
> The overview page for streaming connectors is too lean -- it should provide 
> more context and also guide the reader toward related topics.
> Note that FLINK-6038 will add links to the Bahir connectors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3964: [FLINK-6660][docs] expand the connectors overview ...

[alpinegizmo]

GitHub user alpinegizmo opened a pull request:

https://github.com/apache/flink/pull/3964

[FLINK-6660][docs] expand the connectors overview page

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alpinegizmo/flink 6660-connectors-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3964.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3964


commit d02b1da7ca570e1fc369595d91d7df1b2230918d
Author: David Anderson 
Date:   2017-05-22T15:39:34Z

[FLINK-6660][docs] expand the connectors overview page




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6537) Umbrella issue for fixes to incremental snapshots

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019717#comment-16019717
 ] 

Stefan Richter commented on FLINK-6537:
---

Which exception is that exactly? And how does it happen all the time? There is 
no JIRA for it, because I have not encountered the problem. Can this be 
reproduced somehow?

> Umbrella issue for fixes to incremental snapshots
> -
>
> Key: FLINK-6537
> URL: https://issues.apache.org/jira/browse/FLINK-6537
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
> Fix For: 1.3.0
>
>
> This issue tracks ongoing fixes in the incremental checkpointing feature for 
> the 1.3 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6661) Merge "Subtasks" and "TaskManagers" view

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6661:
---

 Summary: Merge "Subtasks" and "TaskManagers" view
 Key: FLINK-6661
 URL: https://issues.apache.org/jira/browse/FLINK-6661
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.4.0
Reporter: Chesnay Schepler


The {{Subtasks}} and {{TaskManagers/ Subtasks by TaskManager}} view are very 
similar in what they do, so much in fact that they are identical if the 
taskmanagers have a single task slot.

I propose to merge them, and add a checkbox to aggregate the subtasks by 
taskmanagers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6482) Add nested serializers into configuration snapshots of composite serializers

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai resolved FLINK-6482.

Resolution: Fixed

Fixed for master via a7bc5de9b17dde793c6da0e7cb74af117148.
Fixed for 1.3.0 via 09cc3f7c577741d7271f73c6e916c98f7b79a820.

> Add nested serializers into configuration snapshots of composite serializers
> 
>
> Key: FLINK-6482
> URL: https://issues.apache.org/jira/browse/FLINK-6482
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing, Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.3.0
>
>
> Currently, the composite serializers' configuration snapshots only wrap the 
> config snapshots of nested serializers.
> We should also consider adding serialization of the nested serializers into 
> the config snapshot, so that in the case where only some nested serializer 
> cannot be loaded (class missing / implementation changed), we can also 
> provide a path for serializer upgrades.
> This applies for all composite serializers that have nested serializers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6603) Enable checkstyle on test sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019709#comment-16019709
 ] 

ASF GitHub Bot commented on FLINK-6603:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/3941
  
@zentol added `EmptyLineSeparator` to the strict checkstyle.


> Enable checkstyle on test sources
> -
>
> Key: FLINK-6603
> URL: https://issues.apache.org/jira/browse/FLINK-6603
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.4.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.4.0
>
>
> With the addition of strict checkstyle to select modules (currently limited 
> to {{flink-streaming-java}}) we can enable the checkstyle flag 
> {{includeTestSourceDirectory}} to perform the same unused imports, 
> whitespace, and other checks on test sources.
> Should first resolve the import grouping as discussed in FLINK-6107. Also, 
> several tests exceed the 2500 line limit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6446) Various improvements to the Web Frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019708#comment-16019708
 ] 

ASF GitHub Bot commented on FLINK-6446:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3946#discussion_r117775543
  
--- Diff: flink-runtime-web/web-dashboard/app/partials/jobs/job.plan.jade 
---
@@ -28,10 +28,10 @@ split
 a(ui-sref=".subtasks({nodeid: nodeid})") Subtasks
 
   li(ui-sref-active='active')
-a(ui-sref=".taskmanagers({nodeid: nodeid})") TaskManagers
+a(ui-sref=".taskmanagers({nodeid: nodeid})") Subtasks by 
TaskManager
--- End diff --

OK. I'll merge the PR for both versions and create a follow-up JIRA.


> Various improvements to the Web Frontend
> 
>
> Key: FLINK-6446
> URL: https://issues.apache.org/jira/browse/FLINK-6446
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Stephan Ewen
>
> This is the umbrella issue for various improvements to the web frontend,



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3941: [FLINK-6603] [streaming] Enable checkstyle on test source...

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/3941
  
@zentol added `EmptyLineSeparator` to the strict checkstyle.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---