[jira] [Comment Edited] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Michael Fong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204394#comment-16204394
 ] 

Michael Fong edited comment on FLINK-7839 at 10/14/17 12:46 AM:


>From above comment, I propose we could put a note in the document addressing 
>the following:
1. for users with maven 3.0+
1.1 Get rid of '-DarchetypeCatalog' 
1.2 Optionally define snapshot repository in settings.xml (only for dev, i 
assume) ; otherwise, maven should locate the latest release repository by 
default .
2. for users with older maven
1.1 Optionally provide '-DarchetypeCatalog' as maven would attempt to retrieve 
meta from snapshot repo over release repo.


was (Author: mcfongtw):
>From above comment, I propose we could put a note in document indicate 
1. Get rid of '-DarchetypeCatalog' for users with maven 3.0+
2. Optionally define snapshot repository in settings.xml (only for dev, i 
assume) ; otherwise, maven should locate the latest release repository by 
default .

> Creating Quickstart project for SNAPSHOT version fails
> --
>
> Key: FLINK-7839
> URL: https://issues.apache.org/jira/browse/FLINK-7839
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.4.0
>Reporter: Gary Yao
>Assignee: Michael Fong
>Priority: Blocker
>  Labels: documentation
>
> The documentation on creating quickstart projects is broken for SNAPSHOT 
> releases. For example, the documentation suggests to use the following 
> command to generate a Flink 1.4-SNAPSHOT project using maven archetypes:
> {code}
> mvn archetype:generate \
>   -DarchetypeGroupId=org.apache.flink  \
>   -DarchetypeArtifactId=flink-quickstart-java  \
>   
> -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/
>  \
>   -DarchetypeVersion=1.4-SNAPSHOT
> {code}
> The command fails with the error:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) 
> on project flink-training-exercises: archetypeCatalog 
> 'https://repository.apache.org/content/repositories/snapshots/' is not 
> supported anymore. Please read the plugin documentation for details. -> [Help 
> 1]
> {code}
> This also affects the quickstart script.
> Since version 3.0.0, the archetype plugin does not allow to specify 
> repositories as command line arguments. See 
> http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Michael Fong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204394#comment-16204394
 ] 

Michael Fong commented on FLINK-7839:
-

>From above comment, I propose we could put a note in document indicate 
1. Get rid of '-DarchetypeCatalog' for users with maven 3.0+
2. Optionally define snapshot repository in settings.xml (only for dev, i 
assume) ; otherwise, maven should locate the latest release repository by 
default .

> Creating Quickstart project for SNAPSHOT version fails
> --
>
> Key: FLINK-7839
> URL: https://issues.apache.org/jira/browse/FLINK-7839
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.4.0
>Reporter: Gary Yao
>Assignee: Michael Fong
>Priority: Blocker
>  Labels: documentation
>
> The documentation on creating quickstart projects is broken for SNAPSHOT 
> releases. For example, the documentation suggests to use the following 
> command to generate a Flink 1.4-SNAPSHOT project using maven archetypes:
> {code}
> mvn archetype:generate \
>   -DarchetypeGroupId=org.apache.flink  \
>   -DarchetypeArtifactId=flink-quickstart-java  \
>   
> -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/
>  \
>   -DarchetypeVersion=1.4-SNAPSHOT
> {code}
> The command fails with the error:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) 
> on project flink-training-exercises: archetypeCatalog 
> 'https://repository.apache.org/content/repositories/snapshots/' is not 
> supported anymore. Please read the plugin documentation for details. -> [Help 
> 1]
> {code}
> This also affects the quickstart script.
> Since version 3.0.0, the archetype plugin does not allow to specify 
> repositories as command line arguments. See 
> http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Michael Fong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204390#comment-16204390
 ] 

Michael Fong edited comment on FLINK-7839 at 10/14/17 12:41 AM:


I can help update the document since I also encountered the same error for 
v1.4.0.  I could not see the same commands for earlier versions, thus I assume 
this was added in 1.4.0. 

{color:#205081}However, the url leads to a SNAPSHOT repository rather than a 
RELEASE repository. I am curious about which one should be used once 1.4.0 is 
released.{color}

For users with older maven, this command line should work just fine; however, 
for those with Maven 3.0+, you need to get rid of '-DarchetypeCatalog' argument 
to make it work. 

{code:java}
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom 
---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from 
[org.apache.flink:flink-quickstart-java:1.3.2] found in catalog remote
{code}


As for the missing 'archetypeCatalog' argument, you could optionally define a 
repository entry in settings.xml as official suggested 
(http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html).
 You will download latest snapshot meta while generating the quick-start 
project by executing the `mvn archetype:generate` For example, 

{code:bash}
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom 
---
[INFO] Generating project in Interactive mode
Archetype repository not defined. Using the one from 
[org.apache.flink:flink-quickstart-java:1.1-SNAPSHOT -> 
https://repository.apache.org/content/repositories/snapshots] found in catalog 
remote
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
 (1.2 kB at 579 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
 (1.2 kB at 580 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.pom
 (3.7 kB at 2.6 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
 (609 B at 415 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
 (609 B at 414 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/flink-quickstart-1.4-20171009.044305-294.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/flink-quickstart-1.4-20171009.044305-294.pom
 (3.1 kB at 2.2 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
 (605 B at 444 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
 (605 B at 429 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/flink-parent-1.4-20171013.033404-301.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/flink-parent-1.4-20171013.033404-301.pom
 (50 kB at 27 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.jar
Downloaded: 

[jira] [Comment Edited] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Michael Fong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204390#comment-16204390
 ] 

Michael Fong edited comment on FLINK-7839 at 10/14/17 12:34 AM:


I can help update the document since I also encountered the same error for 
v1.4.0.  I could not see the same commands for earlier versions, thus I assume 
this was added in 1.4.0. 

{color:#205081}However, the url leads to a SNAPSHOT repository rather than a 
RELEASE repository. I am curious about which one should be used once 1.4.0 is 
released.{color}

For users with older maven, this command line should work just fine; however, 
for those with Maven 3.0+, you need to get rid of '-DarchetypeCatalog' argument 
to make it work. 

As for the missing 'archetypeCatalog' argument, you could optionally define a 
repository entry in settings.xml as official suggested 
(http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html).
 You will download latest snapshot meta while generating the quick-start 
project by executing the `mvn archetype:generate` For example, 

{code:bash}
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom 
---
[INFO] Generating project in Interactive mode
Archetype repository not defined. Using the one from 
[org.apache.flink:flink-quickstart-java:1.1-SNAPSHOT -> 
https://repository.apache.org/content/repositories/snapshots] found in catalog 
remote
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
 (1.2 kB at 579 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
 (1.2 kB at 580 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.pom
 (3.7 kB at 2.6 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
 (609 B at 415 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
 (609 B at 414 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/flink-quickstart-1.4-20171009.044305-294.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/flink-quickstart-1.4-20171009.044305-294.pom
 (3.1 kB at 2.2 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
 (605 B at 444 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
 (605 B at 429 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/flink-parent-1.4-20171013.033404-301.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/flink-parent-1.4-20171013.033404-301.pom
 (50 kB at 27 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.jar
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.jar
 (16 kB at 9.6 kB/s)
{code}




was (Author: mcfongtw):
I can help update the document since I also encountered the same error for 
v1.4.0.  I could not see the same commands for earlier versions, thus I assume 
this was added in 1.4.0. 

[jira] [Commented] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Michael Fong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204390#comment-16204390
 ] 

Michael Fong commented on FLINK-7839:
-

I can help update the document since I also encountered the same error for 
v1.4.0.  I could not see the same commands for earlier versions, thus I assume 
this was added in 1.4.0. 

{color:#f6c342}However, the url leads to a SNAPSHOT repository rather than a 
RELEASE repository. I am curious about which one should be used once 1.4.0 is 
released.{color}

For users with older maven, this command line should work just fine; however, 
for those with Maven 3.0+, you need to get rid of '-DarchetypeCatalog' argument 
to make it work. 

As for the missing 'archetypeCatalog' argument, you could optionally define a 
repository entry in settings.xml as official suggested 
(http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html).
 You will download latest snapshot meta while generating the quick-start 
project by executing the `mvn archetype:generate` For example, 

{code:bash}
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom 
---
[INFO] Generating project in Interactive mode
{color:#f6c342}[INFO] Archetype repository not defined. Using the one from 
[org.apache.flink:flink-quickstart-java:1.1-SNAPSHOT -> 
https://repository.apache.org/content/repositories/snapshots] found in catalog 
remote{color}
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
 (1.2 kB at 579 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/maven-metadata.xml
 (1.2 kB at 580 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.pom
 (3.7 kB at 2.6 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
 (609 B at 415 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/maven-metadata.xml
 (609 B at 414 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/flink-quickstart-1.4-20171009.044305-294.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart/1.4-SNAPSHOT/flink-quickstart-1.4-20171009.044305-294.pom
 (3.1 kB at 2.2 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
 (605 B at 444 B/s)
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/maven-metadata.xml
 (605 B at 429 B/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/flink-parent-1.4-20171013.033404-301.pom
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-parent/1.4-SNAPSHOT/flink-parent-1.4-20171013.033404-301.pom
 (50 kB at 27 kB/s)
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.jar
Downloaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-quickstart-java/1.4-SNAPSHOT/flink-quickstart-java-1.4-20171009.044311-294.jar
 (16 kB at 9.6 kB/s)
{code}



> Creating Quickstart project for SNAPSHOT version fails
> --
>
> Key: FLINK-7839
> URL: https://issues.apache.org/jira/browse/FLINK-7839
>   

[jira] [Assigned] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Michael Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Fong reassigned FLINK-7839:
---

Assignee: Michael Fong

> Creating Quickstart project for SNAPSHOT version fails
> --
>
> Key: FLINK-7839
> URL: https://issues.apache.org/jira/browse/FLINK-7839
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.4.0
>Reporter: Gary Yao
>Assignee: Michael Fong
>Priority: Blocker
>  Labels: documentation
>
> The documentation on creating quickstart projects is broken for SNAPSHOT 
> releases. For example, the documentation suggests to use the following 
> command to generate a Flink 1.4-SNAPSHOT project using maven archetypes:
> {code}
> mvn archetype:generate \
>   -DarchetypeGroupId=org.apache.flink  \
>   -DarchetypeArtifactId=flink-quickstart-java  \
>   
> -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/
>  \
>   -DarchetypeVersion=1.4-SNAPSHOT
> {code}
> The command fails with the error:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) 
> on project flink-training-exercises: archetypeCatalog 
> 'https://repository.apache.org/content/repositories/snapshots/' is not 
> supported anymore. Please read the plugin documentation for details. -> [Help 
> 1]
> {code}
> This also affects the quickstart script.
> Since version 3.0.0, the archetype plugin does not allow to specify 
> repositories as command line arguments. See 
> http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3688: [FLINK-6273] [core] don't convert hostname to lowe...

[barcahead]

Github user barcahead closed the pull request at:

https://github.com/apache/flink/pull/3688


---

[jira] [Commented] (FLINK-6273) Client can't connect to jobmanager whose hostname contains capital letters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204289#comment-16204289
 ] 

ASF GitHub Bot commented on FLINK-6273:
---

Github user barcahead closed the pull request at:

https://github.com/apache/flink/pull/3688


> Client can't connect to jobmanager whose hostname contains capital letters
> --
>
> Key: FLINK-6273
> URL: https://issues.apache.org/jira/browse/FLINK-6273
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Yelei Feng
>Assignee: Yelei Feng
>Priority: Blocker
> Fix For: 1.4.0
>
>
> In non-HA mode, if we set jobmanager.rpc.address to a hostname with some 
> capital letters, flink client can't connect to jobmananger.
> ERROR | [flink-akka.actor.default-dispatcher-4] | dropping message [class 
> akka.actor.ActorSelectionMessage] for non-local recipient 
> [Actor[akka.tcp://flink@szv1000258958:32586/]] arriving at 
> [akka.tcp://flink@szv1000258958:32586] inbound addresses are 
> [akka.tcp://flink@SZV1000258958:32586] | akka.remote.EndpointWriter 
> (Slf4jLogger.scala:65)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4816: [hotfix][docs] CEP docs review to remove weasel wo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4816


---

[jira] [Assigned] (FLINK-7799) Improve performance of windowed joins

[Xingcan Cui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xingcan Cui reassigned FLINK-7799:
--

Assignee: Xingcan Cui

> Improve performance of windowed joins
> -
>
> Key: FLINK-7799
> URL: https://issues.apache.org/jira/browse/FLINK-7799
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Xingcan Cui
>Priority: Critical
>
> The performance of windowed joins can be improved by changing the state 
> access patterns.
> Right now, rows are inserted into a MapState with their timestamp as key. 
> Since we use a time resolution of 1ms, this means that the full key space of 
> the state must be iterated and many map entries must be accessed when joining 
> or evicting rows. 
> A better strategy would be to block the time into larger intervals and 
> register the rows in their respective interval. Another benefit would be that 
> we can directly access the state entries because we know exactly which 
> timestamps to look up. Hence, we can limit the state access to the relevant 
> section during joining and state eviction. 
> The good size for intervals needs to be identified and might depend on the 
> size of the window.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7798) Add support for windowed joins to Table API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204015#comment-16204015
 ] 

ASF GitHub Bot commented on FLINK-7798:
---

GitHub user xccui opened a pull request:

https://github.com/apache/flink/pull/4825

[FLINK-7798] [table] Add support for stream windowed inner joins to Table 
API

## What is the purpose of the change

The PR enables the time-windowed inner join for 
`StreamExecutionEnvironment` in Table API.

## Brief change log

  - Remove the inner join check for `StreamTableEnvironment` in 
`operators.scala`.
  - Add some tests for stream table inner join translation.
  - Update related docs.


## Verifying this change

This change adds tests to `JoinITCase.scala`, which can be run directly.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (docs)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xccui/flink FLINK-7798

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4825.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4825


commit 7a424ac402e5690e510f1a4286c40936230fb012
Author: Xingcan Cui 
Date:   2017-10-13T18:25:47Z

[FLINK-7798] [table] Add support for stream time-windowed inner join to 
Table API




> Add support for windowed joins to Table API
> ---
>
> Key: FLINK-7798
> URL: https://issues.apache.org/jira/browse/FLINK-7798
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Xingcan Cui
>Priority: Blocker
> Fix For: 1.4.0
>
>
> Currently, windowed joins on streaming tables are only supported through SQL.
> The Table API should support these joins as well. For that, we have to adjust 
> the Table API validation and translate the API into the respective logical 
> plan. Since most of the code should already be there for the batch Table API 
> joins, this should be fairly straightforward.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4825: [FLINK-7798] [table] Add support for stream window...

[xccui]

GitHub user xccui opened a pull request:

https://github.com/apache/flink/pull/4825

[FLINK-7798] [table] Add support for stream windowed inner joins to Table 
API

## What is the purpose of the change

The PR enables the time-windowed inner join for 
`StreamExecutionEnvironment` in Table API.

## Brief change log

  - Remove the inner join check for `StreamTableEnvironment` in 
`operators.scala`.
  - Add some tests for stream table inner join translation.
  - Update related docs.


## Verifying this change

This change adds tests to `JoinITCase.scala`, which can be run directly.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (docs)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xccui/flink FLINK-7798

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4825.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4825


commit 7a424ac402e5690e510f1a4286c40936230fb012
Author: Xingcan Cui 
Date:   2017-10-13T18:25:47Z

[FLINK-7798] [table] Add support for stream time-windowed inner join to 
Table API




---

[jira] [Commented] (FLINK-6225) Support Row Stream for CassandraSink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203933#comment-16203933
 ] 

ASF GitHub Bot commented on FLINK-6225:
---

Github user PangZhi commented on the issue:

https://github.com/apache/flink/pull/3748
  
@zentol @fhueske Can you take another look.


> Support Row Stream for CassandraSink
> 
>
> Key: FLINK-6225
> URL: https://issues.apache.org/jira/browse/FLINK-6225
> Project: Flink
>  Issue Type: New Feature
>  Components: Cassandra Connector
>Affects Versions: 1.3.0
>Reporter: Jing Fan
>Assignee: Haohui Mai
> Fix For: 1.4.0
>
>
> Currently in CassandraSink, specifying query is not supported for row-stream. 
> The solution should be similar to CassandraTupleSink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

[PangZhi]

Github user PangZhi commented on the issue:

https://github.com/apache/flink/pull/3748
  
@zentol @fhueske Can you take another look.


---

[GitHub] flink issue #4559: [FLINK-7468][network] Implement sender backlog logic for ...

[zhijiangW]

Github user zhijiangW commented on the issue:

https://github.com/apache/flink/pull/4559
  
@pnowojski , this PR is ready for review. :)


---

[jira] [Commented] (FLINK-7468) Implement sender backlog logic for credit-based

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203825#comment-16203825
 ] 

ASF GitHub Bot commented on FLINK-7468:
---

Github user zhijiangW commented on the issue:

https://github.com/apache/flink/pull/4559
  
@pnowojski , this PR is ready for review. :)


> Implement sender backlog logic for credit-based
> ---
>
> Key: FLINK-7468
> URL: https://issues.apache.org/jira/browse/FLINK-7468
> Project: Flink
>  Issue Type: Sub-task
>  Components: Network
>Reporter: zhijiang
>Assignee: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control.
> Receivers should know how many buffers are available on the sender side (the 
> backlog). The receivers use this information to decide how to distribute 
> floating buffers.
> The {{ResultSubpartition}} maintains the backlog which only indicates the 
> number of buffers in this subpartition, not including the number of events. 
> The backlog is increased for adding buffer to this subpartition, and 
> decreased for polling buffer from it.
> The backlog is attached in {{BufferResponse}} by sender as an absolute value 
> after the buffer being transferred.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7456) Implement Netty sender incoming pipeline for credit-based

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203810#comment-16203810
 ] 

ASF GitHub Bot commented on FLINK-7456:
---

Github user zhijiangW commented on the issue:

https://github.com/apache/flink/pull/4552
  
@pnowojski , this PR is ready for review.

It covers almost all the logics of credit-based on sender side. 
In addition, I replace the current `PartitionRequestClientHandler` with 
`CreditBasedClientHandler` and remove previous temporary codes for making this 
feature work on both sides.

It leaves a small work to do in this PR related with 
`SpilledSubpartitionView#nextBufferIsEvent` because the existing process in 
spilled sub-partition can not get next buffer directly. But the current default 
value for  `nextBufferIsEvent`` will not affect the core process, only results 
in wasting a unnecessary credit, then I will try to solve it in a lightweight 
way later. 


> Implement Netty sender incoming pipeline for credit-based
> -
>
> Key: FLINK-7456
> URL: https://issues.apache.org/jira/browse/FLINK-7456
> Project: Flink
>  Issue Type: Sub-task
>  Components: Network
>Reporter: zhijiang
>Assignee: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control.
> On sender side, each subpartition view maintains an atomic integer 
> {{currentCredit}} from receiver. Once receiving the messages of 
> {{PartitionRequest}} and {{AddCredit}}, the {{currentCredit}} is added by 
> deltas.
> Each view also maintains an atomic boolean field to mark it as registered 
> available for transfer to make sure it is enqueued in handler only once. If 
> the {{currentCredit}} increases from zero and there are available buffers in 
> the subpartition, the corresponding view will be enqueued for transferring 
> data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4552: [FLINK-7456][network] Implement Netty sender incoming pip...

[zhijiangW]

Github user zhijiangW commented on the issue:

https://github.com/apache/flink/pull/4552
  
@pnowojski , this PR is ready for review.

It covers almost all the logics of credit-based on sender side. 
In addition, I replace the current `PartitionRequestClientHandler` with 
`CreditBasedClientHandler` and remove previous temporary codes for making this 
feature work on both sides.

It leaves a small work to do in this PR related with 
`SpilledSubpartitionView#nextBufferIsEvent` because the existing process in 
spilled sub-partition can not get next buffer directly. But the current default 
value for  `nextBufferIsEvent`` will not affect the core process, only results 
in wasting a unnecessary credit, then I will try to solve it in a lightweight 
way later. 


---

[jira] [Updated] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao updated FLINK-7839:

Description: 
The documentation on creating quickstart projects is broken for SNAPSHOT 
releases. For example, the documentation suggests to use the following command 
to generate a Flink 1.4-SNAPSHOT project using maven archetypes:
{code}
mvn archetype:generate \
  -DarchetypeGroupId=org.apache.flink  \
  -DarchetypeArtifactId=flink-quickstart-java\
  
-DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/
 \
  -DarchetypeVersion=1.4-SNAPSHOT
{code}
The command fails with the error:
{code}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) on 
project flink-training-exercises: archetypeCatalog 
'https://repository.apache.org/content/repositories/snapshots/' is not 
supported anymore. Please read the plugin documentation for details. -> [Help 1]
{code}

This also affects the quickstart script.

Since version 3.0.0, the archetype plugin does not allow to specify 
repositories as command line arguments. See 
http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html

  was:
The documentation on creating quickstart projects is broken for SNAPSHOT 
releases. For example, the documentation suggests to use the following command 
to generate a Flink 1.4-SNAPSHOT project using maven archetypes:
{code}
mvn archetype:generate   \
  -DarchetypeGroupId=org.apache.flink  \
  -DarchetypeArtifactId=flink-quickstart-java  \
  
-DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/
 \
  -DarchetypeVersion=1.4-SNAPSHOT
{code}
The command fails with the error:
{code}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) on 
project flink-training-exercises: archetypeCatalog 
'https://repository.apache.org/content/repositories/snapshots/' is not 
supported anymore. Please read the plugin documentation for details. -> [Help 1]
{code}

This also affects the quickstart script.

Since version 3.0.0, the archetype plugin does not allow to specify 
repositories as command line arguments. See 
http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html


> Creating Quickstart project for SNAPSHOT version fails
> --
>
> Key: FLINK-7839
> URL: https://issues.apache.org/jira/browse/FLINK-7839
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.4.0
>Reporter: Gary Yao
>Priority: Blocker
>  Labels: documentation
>
> The documentation on creating quickstart projects is broken for SNAPSHOT 
> releases. For example, the documentation suggests to use the following 
> command to generate a Flink 1.4-SNAPSHOT project using maven archetypes:
> {code}
> mvn archetype:generate \
>   -DarchetypeGroupId=org.apache.flink  \
>   -DarchetypeArtifactId=flink-quickstart-java\
>   
> -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/
>  \
>   -DarchetypeVersion=1.4-SNAPSHOT
> {code}
> The command fails with the error:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) 
> on project flink-training-exercises: archetypeCatalog 
> 'https://repository.apache.org/content/repositories/snapshots/' is not 
> supported anymore. Please read the plugin documentation for details. -> [Help 
> 1]
> {code}
> This also affects the quickstart script.
> Since version 3.0.0, the archetype plugin does not allow to specify 
> repositories as command line arguments. See 
> http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7839) Creating Quickstart project for SNAPSHOT version fails

[Gary Yao (JIRA)]

Gary Yao created FLINK-7839:
---

 Summary: Creating Quickstart project for SNAPSHOT version fails
 Key: FLINK-7839
 URL: https://issues.apache.org/jira/browse/FLINK-7839
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.4.0
Reporter: Gary Yao
Priority: Blocker


The documentation on creating quickstart projects is broken for SNAPSHOT 
releases. For example, the documentation suggests to use the following command 
to generate a Flink 1.4-SNAPSHOT project using maven archetypes:
{code}
mvn archetype:generate   \
  -DarchetypeGroupId=org.apache.flink  \
  -DarchetypeArtifactId=flink-quickstart-java  \
  
-DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/
 \
  -DarchetypeVersion=1.4-SNAPSHOT
{code}
The command fails with the error:
{code}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) on 
project flink-training-exercises: archetypeCatalog 
'https://repository.apache.org/content/repositories/snapshots/' is not 
supported anymore. Please read the plugin documentation for details. -> [Help 1]
{code}

This also affects the quickstart script.

Since version 3.0.0, the archetype plugin does not allow to specify 
repositories as command line arguments. See 
http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-5808) Missing verification for setParallelism and setMaxParallelism

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-5808:
---

Assignee: (was: Aljoscha Krettek)

> Missing verification for setParallelism and setMaxParallelism
> -
>
> Key: FLINK-5808
> URL: https://issues.apache.org/jira/browse/FLINK-5808
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.2.0
>Reporter: Aljoscha Krettek
>
> When {{setParallelism()}} is called we don't verify that it is <= than max 
> parallelism. Likewise, when {{setMaxParallelism()}} is called we don't check 
> that the new value doesn't clash with a previously set parallelism.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-6113) Implement split/select with Side Outputs

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-6113:
---

Assignee: (was: Aljoscha Krettek)

> Implement split/select with Side Outputs
> 
>
> Key: FLINK-6113
> URL: https://issues.apache.org/jira/browse/FLINK-6113
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.3.0
>Reporter: Chen Qin
>Priority: Minor
>
> With completion of FLINK-4460(side outputs), this is one of follow up item 
> towards deprecate string tag based split/select with OutputTag based 
> split/select.
> In Flink 2.0, we might consider eventually deprecate split/select 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-5363) Fire timers when window state is currently empty

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-5363:
---

Assignee: (was: Aljoscha Krettek)

> Fire timers when window state is currently empty
> 
>
> Key: FLINK-5363
> URL: https://issues.apache.org/jira/browse/FLINK-5363
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Aljoscha Krettek
>
> Currently, when a {{Trigger}} sets a timer and that timer fires in the future 
> at a point when there is currently no data in the window state, then that 
> timer is being ignored.
> This is a problem for some users because they manually set cleanup timers and 
> they need to be called because the trigger needs to cleanup some state. (For 
> normal time windows this is not a problem, but for special cases built on top 
> of {{GlobalWindows}} the current behaviour leads to problems.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-3582) Add Iterator over State for All Keys in Partitioned State

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-3582.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

Implemented in FLINK-7683

> Add Iterator over State for All Keys in Partitioned State
> -
>
> Key: FLINK-3582
> URL: https://issues.apache.org/jira/browse/FLINK-3582
> Project: Flink
>  Issue Type: Sub-task
>  Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>
> Having a non-keyed trigger requires that we have a way to iterate over the 
> state for all keys, so that we can emit window results.
> This should only be for internal use, but maybe users also want to iterate 
> over the state for all keys. 
> As a corollary, we then also need a way to drop state for all keys at the 
> same time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-2706) Add support for streaming RollingFileSink to truncate / append on UNIX file systems

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-2706.
---
Resolution: Won't Fix

This has been superseded by the work to make the {{BucketingSink}} our 
{{FileSystem}} abstraction: FLINK-5789

> Add support for streaming RollingFileSink to truncate / append on UNIX file 
> systems
> ---
>
> Key: FLINK-2706
> URL: https://issues.apache.org/jira/browse/FLINK-2706
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector, Streaming Connectors
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Aljoscha Krettek
>
> Efficient exactly-once behavior needs the filesystem to support appending and 
> truncating files.
> Since the UNIX file system API allows to append files and truncate files, we 
> can support perfect exactly-once behavior efficiently on all file systems 
> that expose a UNIX / POSIX-style interface (local FS, NFS, MapR FS).
> Without this support, only Hadoop 2.7+ versions support proper exactly once 
> behavior.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-2747) TypeExtractor does not correctly analyze Scala Immutables (AnyVal)

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-2747:
---

Assignee: (was: Aljoscha Krettek)

> TypeExtractor does not correctly analyze Scala Immutables (AnyVal)
> --
>
> Key: FLINK-2747
> URL: https://issues.apache.org/jira/browse/FLINK-2747
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> This example program only works correctly if Kryo is force-enabled.
> {code}
> object Test {
>   class Id(val underlying: Int) extends AnyVal
>   class X(var id: Id) {
> def this() { this(new Id(0)) }
>   }
>   class MySource extends SourceFunction[X] {
> def run(ctx: SourceFunction.SourceContext[X]) {
>   ctx.collect(new X(new Id(1)))
> }
> def cancel() {}
>   }
>   def main(args: Array[String]) {
> val env = StreamExecutionContext.getExecutionContext
> env.addSource(new MySource).print
> env.execute("Test")
>   }
> }
> {code}
> The program fails with this:
> {code}
> Caused by: java.lang.RuntimeException: Cannot instantiate class.
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.createInstance(PojoSerializer.java:227)
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:421)
>   at 
> org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.deserialize(StreamRecordSerializer.java:110)
>   at 
> org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.deserialize(StreamRecordSerializer.java:41)
>   at 
> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
>   at 
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:125)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:136)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:55)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:198)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:579)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7830) Problematic interaction of CEP and asynchronous snapshots

[Kostas Kloudas (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203760#comment-16203760
 ] 

Kostas Kloudas commented on FLINK-7830:
---

I closed the issues here because, as I wrote in the JIRAs themselves, we merged 
fixes for:

https://issues.apache.org/jira/browse/FLINK-7835
https://issues.apache.org/jira/browse/FLINK-7484

Hopefully, these fix the problems with CEP and RocksDB. If not, please re-open 
the issues.

> Problematic interaction of CEP and asynchronous snapshots
> -
>
> Key: FLINK-7830
> URL: https://issues.apache.org/jira/browse/FLINK-7830
> Project: Flink
>  Issue Type: Bug
>  Components: CEP, State Backends, Checkpointing
>Reporter: Aljoscha Krettek
> Fix For: 1.4.0
>
>
> Just so we collect all the (possibly duplicate) issue reports.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-2600) Failing ElasticsearchSinkITCase.testNodeClient test case

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-2600.
---
Resolution: Cannot Reproduce

Hasn't occurred in about a year.

> Failing ElasticsearchSinkITCase.testNodeClient test case
> 
>
> Key: FLINK-2600
> URL: https://issues.apache.org/jira/browse/FLINK-2600
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Till Rohrmann
>Assignee: Aljoscha Krettek
>  Labels: test-stability
>
> I observed that the {{ElasticsearchSinkITCase.testNodeClient}} test case 
> fails on Travis. The stack trace is
> {code}
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:414)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.testingUtils.TestingJobManager$$anonfun$handleTestingMessage$1.applyOrElse(TestingJobManager.scala:285)
>   at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
>   at 
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>   at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:104)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>   at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.RuntimeException: An error occured in ElasticsearchSink.
>   at 
> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink.close(ElasticsearchSink.java:307)
>   at 
> org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:40)
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.close(AbstractUdfStreamOperator.java:75)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:243)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:185)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:581)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: IndexMissingException[[my-index] 
> missing]
>   at 
> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink$1.afterBulk(ElasticsearchSink.java:240)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:316)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:299)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:281)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:260)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:246)
>   at 
> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink.invoke(ElasticsearchSink.java:286)
>   at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:37)
>   at 
> 

[jira] [Closed] (FLINK-2506) HBase connection closing down (table distributed over more than 1 region server - Flink Cluster-Mode)

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-2506.
---
Resolution: Won't Fix

Closing after too much inactivity and code changed in between.

> HBase connection closing down (table distributed over more than 1 region 
> server - Flink Cluster-Mode)
> -
>
> Key: FLINK-2506
> URL: https://issues.apache.org/jira/browse/FLINK-2506
> Project: Flink
>  Issue Type: Bug
>  Components: Examples
>Affects Versions: 0.10.0
>Reporter: Lydia Ickler
>Assignee: Aljoscha Krettek
> Attachments: hbase-example.tar.gz
>
>
> If I fill a default table (create 'test-table', 'someCf') with the 
> HBaseWriteExample.java program from the HBase addon library then a table 
> without start/end key is created. 
> The data reading works great with the HBaseReadExample.java.
> Nevertheless, if I manually create a "test-table" that is distributed over 
> more than one region server (create 'test-table2', 'someCf',{NUMREGIONS => 
> 3,SPLITALGO => 'HexStringSplit'}) the run is canceled with the following 
> error message: 
> {noformat}
> grips2
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
> attempts=35, exceptions:
> Fri Aug 07 11:18:29 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:18:38 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:18:48 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:18:58 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:08 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:18 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:28 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:38 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:48 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:58 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:20:18 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:20:38 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:20:58 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:21:19 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:21:39 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:21:59 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:22:19 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:22:39 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:22:59 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:23:19 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:23:39 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:24:00 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:24:20 CEST 2015, 
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, 
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:24:40 CEST 2015, 
> 

[jira] [Assigned] (FLINK-987) Extend TypeSerializers and -Comparators to work directly on Memory Segments

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-987:
--

Assignee: (was: Aljoscha Krettek)

> Extend TypeSerializers and -Comparators to work directly on Memory Segments
> ---
>
> Key: FLINK-987
> URL: https://issues.apache.org/jira/browse/FLINK-987
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Affects Versions: 0.6-incubating
>Reporter: Stephan Ewen
> Fix For: 1.0.0
>
>
> As per discussion with [~till.rohrmann], [~uce], [~aljoscha], we suggest to 
> change the way that the TypeSerialzers/Comparators and 
> DataInputViews/DataOutputViews work.
> The goal is to allow more flexibility in the construction on the binary 
> representation of data types, and to allow partial deserialization of 
> individual fields. Both is currently prohibited by the fact that the 
> abstraction of the memory (into which the data goes) is a stream abstraction 
> ({{DataInputView}}, {{DataOutputView}}).
> An idea is to offer a random-access buffer like view for construction and 
> random-access deserialization, as well as various methods to copy elements in 
> a binary fashion between such buffers and streams.
> A possible set of methods for the {{TypeSerializer}} could be:
> {code}
> long serialize(T record, TargetBuffer buffer);
>   
> T deserialize(T reuse, SourceBuffer source);
>   
> void ensureBufferSufficientlyFilled(SourceBuffer source);
>   
>  X deserializeField(X reuse, int logicalPos, SourceBuffer buffer);
>   
> int getOffsetForField(int logicalPos, int offset, SourceBuffer buffer);
>   
> void copy(DataInputView in, TargetBuffer buffer);
>   
> void copy(SourceBuffer buffer,, DataOutputView out);
>   
> void copy(DataInputView source, DataOutputView target);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-4072) EventTimeWindowCheckpointingITCase.testSlidingTimeWindow fails on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-4072:
---

Assignee: (was: Aljoscha Krettek)

> EventTimeWindowCheckpointingITCase.testSlidingTimeWindow fails on Travis
> 
>
> Key: FLINK-4072
> URL: https://issues.apache.org/jira/browse/FLINK-4072
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
>
> The test case {{EventTimeWindowCheckpointingITCase.testSlidingTimeWindow}} 
> failed on Travis.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/137498497/log.txt



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7830) Problematic interaction of CEP and asynchronous snapshots

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-7830.
---
Resolution: Fixed

> Problematic interaction of CEP and asynchronous snapshots
> -
>
> Key: FLINK-7830
> URL: https://issues.apache.org/jira/browse/FLINK-7830
> Project: Flink
>  Issue Type: Bug
>  Components: CEP, State Backends, Checkpointing
>Reporter: Aljoscha Krettek
> Fix For: 1.4.0
>
>
> Just so we collect all the (possibly duplicate) issue reports.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7756) RocksDB state backend Checkpointing (Async and Incremental) is not working with CEP.

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-7756.
-
Resolution: Fixed

Hi, 

We just merged 2 in the master fixes for the following issues. 

https://issues.apache.org/jira/browse/FLINK-7835
https://issues.apache.org/jira/browse/FLINK-7484

I believe that these also fix the issue in this JIRA, this is why I am closing 
it.
Please try the master, and if the problem persists, feel free to re-open it.

> RocksDB state backend Checkpointing (Async and Incremental)  is not working 
> with CEP.
> -
>
> Key: FLINK-7756
> URL: https://issues.apache.org/jira/browse/FLINK-7756
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP, State Backends, Checkpointing, Streaming
>Affects Versions: 1.3.2
> Environment: Flink 1.3.2, Yarn, HDFS, RocksDB backend
>Reporter: Shashank Agarwal
>Priority: Blocker
> Fix For: 1.4.0
>
>
> When i try to use RocksDBStateBackend on my staging cluster (which is using 
> HDFS as file system) it crashes. But When i use FsStateBackend on staging 
> (which is using HDFS as file system) it is working fine.
> On local with local file system it's working fine in both cases.
> Please check attached logs. I have around 20-25 tasks in my app.
> {code:java}
> 2017-09-29 14:21:31,639 INFO  
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink  - No state 
> to restore for the BucketingSink (taskIdx=0).
> 2017-09-29 14:21:31,640 INFO  
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend  - 
> Initializing RocksDB keyed state backend from snapshot.
> 2017-09-29 14:21:32,020 INFO  
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink  - No state 
> to restore for the BucketingSink (taskIdx=1).
> 2017-09-29 14:21:32,022 INFO  
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend  - 
> Initializing RocksDB keyed state backend from snapshot.
> 2017-09-29 14:21:32,078 INFO  com.datastax.driver.core.NettyUtil  
>   - Found Netty's native epoll transport in the classpath, using 
> it
> 2017-09-29 14:21:34,177 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - Attempting to fail task externally Co-Flat Map (1/2) 
> (b879f192c4e8aae6671cdafb3a24c00a).
> 2017-09-29 14:21:34,177 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - Attempting to fail task externally Map (2/2) 
> (1ea5aef6ccc7031edc6b37da2912d90b).
> 2017-09-29 14:21:34,178 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - Attempting to fail task externally Co-Flat Map (2/2) 
> (4bac8e764c67520d418a4c755be23d4d).
> 2017-09-29 14:21:34,178 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - Co-Flat Map (1/2) (b879f192c4e8aae6671cdafb3a24c00a) switched 
> from RUNNING to FAILED.
> AsynchronousException{java.lang.Exception: Could not materialize checkpoint 2 
> for operator Co-Flat Map (1/2).}
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:970)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Could not materialize checkpoint 2 for 
> operator Co-Flat Map (1/2).
>   ... 6 more
> Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:897)
>   ... 5 more
>   Suppressed: java.lang.Exception: Could not properly cancel managed 
> keyed state future.
>   at 
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:90)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1023)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:961)
>   ... 5 more
>   Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at 

[jira] [Closed] (FLINK-6321) RocksDB state backend Checkpointing is not working with KeyedCEP.

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-6321.
-
Resolution: Fixed

Hi, 

We just merged 2 in the master fixes for the following issues. 

https://issues.apache.org/jira/browse/FLINK-7835
https://issues.apache.org/jira/browse/FLINK-7484

I believe that these also fix the issue in this JIRA, this is why I am closing 
it.
Please try the master, and if the problem persists, feel free to re-open it.

> RocksDB state backend Checkpointing is not working with KeyedCEP.
> -
>
> Key: FLINK-6321
> URL: https://issues.apache.org/jira/browse/FLINK-6321
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Affects Versions: 1.2.0
> Environment: yarn-cluster, RocksDB State backend, Checkpointing every 
> 1000 ms
>Reporter: Shashank Agarwal
>Assignee: Kostas Kloudas
>Priority: Blocker
> Fix For: 1.4.0
>
>
> Checkpointing is not working with RocksDBStateBackend when using CEP. It's 
> working fine with FsStateBackend and MemoryStateBackend. Application failing 
> every-time.
> {code}
> 04/18/2017 21:53:20   Job execution switched to status FAILING.
> AsynchronousException{java.lang.Exception: Could not materialize checkpoint 
> 46 for operator KeyedCEPPatternOperator -> Map (1/4).}
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:980)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Could not materialize checkpoint 46 for 
> operator KeyedCEPPatternOperator -> Map (1/4).
>   ... 6 more
> Caused by: java.util.concurrent.CancellationException
>   at java.util.concurrent.FutureTask.report(FutureTask.java:121)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:915)
>   ... 5 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7838) Kafka 0.11 producer exactly-once test failure

[Piotr Nowojski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Nowojski updated FLINK-7838:
--
Description: See attached log  (was: 
https://s3.amazonaws.com/archive.travis-ci.org/jobs/287502990/log.txt?X-Amz-Expires=30=20171013T125617Z=AWS4-HMAC-SHA256=AKIAJRYRXRSVGNKPKO5A/20171013/us-east-1/s3/aws4_request=host=c93f6270805b97e693b396d1283c28a62a99e6fff152a54c6fd5a42507f48e5d)

> Kafka 0.11 producer exactly-once test failure
> -
>
> Key: FLINK-7838
> URL: https://issues.apache.org/jira/browse/FLINK-7838
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.4.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Blocker
> Attachments: log.txt
>
>
> See attached log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7838) Kafka 0.11 producer exactly-once test failure

[Piotr Nowojski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Nowojski updated FLINK-7838:
--
Attachment: log.txt

> Kafka 0.11 producer exactly-once test failure
> -
>
> Key: FLINK-7838
> URL: https://issues.apache.org/jira/browse/FLINK-7838
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.4.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Blocker
> Attachments: log.txt
>
>
> See attached log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7435) FsStateBackend with incremental backup enable does not work with Keyed CEP

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-7435.
-
Resolution: Fixed

Hi, 

We just merged 2 in the master fixes for the following issues. 

https://issues.apache.org/jira/browse/FLINK-7835
https://issues.apache.org/jira/browse/FLINK-7484

I believe that these also fix the issue in this JIRA, this is why I am closing 
it.
Please try the master, and if the problem persists, feel free to re-open it.

> FsStateBackend with incremental backup enable does not work with Keyed CEP
> --
>
> Key: FLINK-7435
> URL: https://issues.apache.org/jira/browse/FLINK-7435
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Affects Versions: 1.3.1, 1.3.2
> Environment: AWS EMR YARN, use CEP with pattern start -> next 
> (oneOrMore.Optional.Consective) -> next(end). Store it with FsStatebackend 
> with Incremental option open. 
>Reporter: daiqing
>Assignee: Kostas Kloudas
>Priority: Blocker
> Fix For: 1.4.0
>
>
> java.lang.RuntimeException: Exception occurred while processing valve output 
> watermark: 
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor$ForwardingValveOutputHandler.handleWatermark(StreamInputProcessor.java:289)
>   at 
> org.apache.flink.streaming.runtime.streamstatus.StatusWatermarkValve.findAndOutputNewMinWatermarkAcrossAlignedChannels(StatusWatermarkValve.java:173)
>   at 
> org.apache.flink.streaming.runtime.streamstatus.StatusWatermarkValve.inputWatermark(StatusWatermarkValve.java:108)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:188)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Could not copy NFA.
>   at org.apache.flink.cep.nfa.NFA$NFASerializer.copy(NFA.java:908)
>   at org.apache.flink.cep.nfa.NFA$NFASerializer.copy(NFA.java:852)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:279)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:296)
>   at 
> org.apache.flink.runtime.state.heap.HeapValueState.value(HeapValueState.java:55)
>   at 
> org.apache.flink.cep.operator.AbstractKeyedCEPPatternOperator.getNFA(AbstractKeyedCEPPatternOperator.java:268)
>   at 
> org.apache.flink.cep.operator.AbstractKeyedCEPPatternOperator.onEventTime(AbstractKeyedCEPPatternOperator.java:230)
>   at 
> org.apache.flink.streaming.api.operators.HeapInternalTimerService.advanceWatermark(HeapInternalTimerService.java:275)
>   at 
> org.apache.flink.streaming.api.operators.InternalTimeServiceManager.advanceWatermark(InternalTimeServiceManager.java:107)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.processWatermark(AbstractStreamOperator.java:946)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor$ForwardingValveOutputHandler.handleWatermark(StreamInputProcessor.java:286)
>   ... 7 more
> Caused by: java.io.StreamCorruptedException: invalid type code: 00
>   at java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1620)
>   at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:719)
>   at 
> java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:882)
>   at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1815)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
>   at 
> org.apache.flink.cep.nfa.NFA$NFASerializer.deserializeCondition(NFA.java:1211)
>   at 
> org.apache.flink.cep.nfa.NFA$NFASerializer.deserializeStates(NFA.java:1169)
>   at org.apache.flink.cep.nfa.NFA$NFASerializer.deserialize(NFA.java:957)
>   at org.apache.flink.cep.nfa.NFA$NFASerializer.copy(NFA.java:903)
>   ... 17 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7760) Restore failing from external checkpointing metadata.

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-7760.
-
Resolution: Fixed

Hi, 

We just merged 2 in the master fixes for the following issues. 

https://issues.apache.org/jira/browse/FLINK-7835
https://issues.apache.org/jira/browse/FLINK-7484

I believe that these also fix the issue in this JIRA, this is why I am closing 
it.
Please try the master, and if the problem persists, feel free to re-open it.

> Restore failing from external checkpointing metadata.
> -
>
> Key: FLINK-7760
> URL: https://issues.apache.org/jira/browse/FLINK-7760
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP, State Backends, Checkpointing
>Affects Versions: 1.3.2
> Environment: Yarn, Flink 1.3.2, HDFS,  FsStateBackend
>Reporter: Shashank Agarwal
>Priority: Blocker
> Fix For: 1.4.0
>
>
> My job failed due to failure of cassandra. I have enabled 
> ExternalizedCheckpoints. But when job tried to restore from that checkpoint 
> it's failing continuously with following error.
> {code:java}
> 2017-10-04 09:39:20,611 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - KeyedCEPPatternOperator -> Map (1/2) 
> (8ff7913f820ead571c8b54ccc6b16045) switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Could not initialize keyed state backend.
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:676)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:663)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:252)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.StreamCorruptedException: invalid type code: 00
>   at 
> java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2519)
>   at 
> java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2553)
>   at 
> java.io.ObjectInputStream$BlockDataInputStream.skipBlockData(ObjectInputStream.java:2455)
>   at java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:1951)
>   at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1621)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>   at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>   at 
> org.apache.flink.cep.nfa.NFA$NFASerializer.deserializeCondition(NFA.java:1211)
>   at 
> org.apache.flink.cep.nfa.NFA$NFASerializer.deserializeStates(NFA.java:1169)
>   at org.apache.flink.cep.nfa.NFA$NFASerializer.deserialize(NFA.java:957)
>   at org.apache.flink.cep.nfa.NFA$NFASerializer.deserialize(NFA.java:852)
>   at 
> org.apache.flink.runtime.state.heap.StateTableByKeyGroupReaders$StateTableByKeyGroupReaderV2V3.readMappingsInKeyGroup(StateTableByKeyGroupReaders.java:132)
>   at 
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restorePartitionedState(HeapKeyedStateBackend.java:518)
>   at 
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restore(HeapKeyedStateBackend.java:397)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:772)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
>   ... 6 more
> {code}
> I have tried to start new job also after failure with parameter {code:java} 
> -s [checkpoint meta data path]{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7838) Kafka 0.11 producer exactly-once test failure

[Piotr Nowojski (JIRA)]

Piotr Nowojski created FLINK-7838:
-

 Summary: Kafka 0.11 producer exactly-once test failure
 Key: FLINK-7838
 URL: https://issues.apache.org/jira/browse/FLINK-7838
 Project: Flink
  Issue Type: Bug
  Components: Kafka Connector
Affects Versions: 1.4.0
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
Priority: Blocker


https://s3.amazonaws.com/archive.travis-ci.org/jobs/287502990/log.txt?X-Amz-Expires=30=20171013T125617Z=AWS4-HMAC-SHA256=AKIAJRYRXRSVGNKPKO5A/20171013/us-east-1/s3/aws4_request=host=c93f6270805b97e693b396d1283c28a62a99e6fff152a54c6fd5a42507f48e5d



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7837) AggregatingFunction does not work with immutable types

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-7837:
---

 Summary: AggregatingFunction does not work with immutable types
 Key: FLINK-7837
 URL: https://issues.apache.org/jira/browse/FLINK-7837
 Project: Flink
  Issue Type: Bug
  Components: DataStream API
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Priority: Blocker
 Fix For: 1.4.0


Currently {{add()}} has this signature:
{code}
void add(IN value, ACC accumulator);
{code}

meaning that a value can only be added to an accumulator by modifying the 
accumulator. This should be extended to:
{code}
ACC add(IN value, ACC accumulator);
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-5968) Document WindowedStream.aggregate()

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-5968:
---

Assignee: Aljoscha Krettek

> Document WindowedStream.aggregate()
> ---
>
> Key: FLINK-5968
> URL: https://issues.apache.org/jira/browse/FLINK-5968
> Project: Flink
>  Issue Type: Sub-task
>  Components: DataStream API, Documentation
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203642#comment-16203642
 ] 

ASF GitHub Bot commented on FLINK-7648:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4824

[FLINK-7648] [flip6] Add TaskManagersHandler

## What is the purpose of the change

Add the `TaskManagersHandler` for the new REST server endpoint.

## Brief change log

- Added `TaskManagersHandler` and `TaskManagersHeaders` for serving 
`TaskManagersInfo`
- Changed the serialization of the `legacy.TaskManagersHandler` to reflect 
`TaskManagerInfo` serialization where the `HardwareDescription` is stored in a 
nested JSON object.

## Verifying this change

Added tests:

- `TaskManagerInfoTest` and `TaskManagersInfoTest` for (un)marshalling

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink addTaskManagersHandler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4824


commit 77ece83513d9a79f2f0c3e7024d91243026c334c
Author: Till 
Date:   2017-10-13T07:19:51Z

[FLINK-7832] [flip6] Extend SlotManager to report free slots per TM

Fail if slot belongs to a unregistered TaskManager

Add more sanity checks

Make the TaskManagerSlot state transitions clearer

Introduce proper TaskManagerSlot state enum

Refactor SlotManager for better maintainability

commit 72ae21be368fc3de32a12e486c73a4fb69a76abf
Author: Till 
Date:   2017-10-12T16:24:02Z

[FLINK-7831] Make last received heartbeat retrievable

This commit adds functionality to retrieve the last received heartbeat from
the HeartbeatManager.

commit fceae2d4fa1bda55f2f9b412388606747e9b24af
Author: Till Rohrmann 
Date:   2017-10-11T22:40:17Z

[FLINK-7648] [flip6] Add TaskManagersHandler

Send dataPort and HardwareDescription to RM

Instantiate RM leader retriever

commit c7f637da68d9627b6dcc73ac4a88d278d7b4b5b5
Author: Till 
Date:   2017-10-13T14:32:56Z

[FLINK-7648] [flip6] Write HardwareDescription in sub object of 
legacy.TaskManagersHandler




> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4824: [FLINK-7648] [flip6] Add TaskManagersHandler

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4824

[FLINK-7648] [flip6] Add TaskManagersHandler

## What is the purpose of the change

Add the `TaskManagersHandler` for the new REST server endpoint.

## Brief change log

- Added `TaskManagersHandler` and `TaskManagersHeaders` for serving 
`TaskManagersInfo`
- Changed the serialization of the `legacy.TaskManagersHandler` to reflect 
`TaskManagerInfo` serialization where the `HardwareDescription` is stored in a 
nested JSON object.

## Verifying this change

Added tests:

- `TaskManagerInfoTest` and `TaskManagersInfoTest` for (un)marshalling

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink addTaskManagersHandler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4824


commit 77ece83513d9a79f2f0c3e7024d91243026c334c
Author: Till 
Date:   2017-10-13T07:19:51Z

[FLINK-7832] [flip6] Extend SlotManager to report free slots per TM

Fail if slot belongs to a unregistered TaskManager

Add more sanity checks

Make the TaskManagerSlot state transitions clearer

Introduce proper TaskManagerSlot state enum

Refactor SlotManager for better maintainability

commit 72ae21be368fc3de32a12e486c73a4fb69a76abf
Author: Till 
Date:   2017-10-12T16:24:02Z

[FLINK-7831] Make last received heartbeat retrievable

This commit adds functionality to retrieve the last received heartbeat from
the HeartbeatManager.

commit fceae2d4fa1bda55f2f9b412388606747e9b24af
Author: Till Rohrmann 
Date:   2017-10-11T22:40:17Z

[FLINK-7648] [flip6] Add TaskManagersHandler

Send dataPort and HardwareDescription to RM

Instantiate RM leader retriever

commit c7f637da68d9627b6dcc73ac4a88d278d7b4b5b5
Author: Till 
Date:   2017-10-13T14:32:56Z

[FLINK-7648] [flip6] Write HardwareDescription in sub object of 
legacy.TaskManagersHandler




---

[jira] [Commented] (FLINK-7812) Log system resources as metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203618#comment-16203618
 ] 

ASF GitHub Bot commented on FLINK-7812:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4801
  
Eclipse Public License is not impossible, but tricky.

I am not a lawyer, but this is what I picked up over the year: EPL is weak 
copyleft, meaning linking is okay, but modifying not (from Apache License 
compatibility) . Shading the code in the library (which we do when building the 
flink dist jar) is a bit of an gray zone. It does not violate the spirit of the 
license, but a court may see that differently.

Various Flink users that approached us to avoid weak copyleft as much as 
possible because of that uncertainty, so avoiding this dependency would be 
desirable.

Making it an optional dependency that users explicitly have to add is 
possible, because then we do shade it into the Flink distribution jar.


> Log system resources as metrics
> ---
>
> Key: FLINK-7812
> URL: https://issues.apache.org/jira/browse/FLINK-7812
> Project: Flink
>  Issue Type: New Feature
>  Components: Metrics
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4801: [FLINK-7812] Log system resources metrics

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4801
  
Eclipse Public License is not impossible, but tricky.

I am not a lawyer, but this is what I picked up over the year: EPL is weak 
copyleft, meaning linking is okay, but modifying not (from Apache License 
compatibility) . Shading the code in the library (which we do when building the 
flink dist jar) is a bit of an gray zone. It does not violate the spirit of the 
license, but a court may see that differently.

Various Flink users that approached us to avoid weak copyleft as much as 
possible because of that uncertainty, so avoiding this dependency would be 
desirable.

Making it an optional dependency that users explicitly have to add is 
possible, because then we do shade it into the Flink distribution jar.


---

[jira] [Closed] (FLINK-7484) CaseClassSerializer.duplicate() does not perform proper deep copy

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-7484.
---
Resolution: Fixed

Fixed on release-1.3 in
664a04c8977c04698576d788b878b4c89f16acc6

Fixed on master in
90be5774e481af87355b9f475562180923039a93

> CaseClassSerializer.duplicate() does not perform proper deep copy
> -
>
> Key: FLINK-7484
> URL: https://issues.apache.org/jira/browse/FLINK-7484
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP, DataStream API, Scala API
>Affects Versions: 1.3.2
> Environment: Flink 1.3.2 , Yarn Cluster, FsStateBackend
>Reporter: Shashank Agarwal
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.4.0
>
>
> I am using many CEP's and Global Window. I am getting following error 
> sometimes and application  crashes. I have checked logically there's no flow 
> in the program. Here ItemPojo is a Pojo class and we are using 
> java.utill.List[ItemPojo]. We are using Scala DataStream API please find 
> attached logs.
> {code}
> 2017-08-17 10:04:12,814 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - TriggerWindow(GlobalWindows(), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@6d36aa3c},
>  co.thirdwatch.trigger.TransactionTrigger@5707c1cb, 
> WindowedStream.apply(WindowedStream.scala:582)) -> Flat Map -> Map -> Sink: 
> Saving CSV Features Sink (1/2) (06c0d4d231bc620ba9e7924b9b0da8d1) switched 
> from RUNNING to FAILED.
> com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: 
> Index: 7, Size: 5
> Serialization trace:
> category (co.thirdwatch.pojo.ItemPojo)
> underlying (scala.collection.convert.Wrappers$SeqWrapper)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:43)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:21)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:657)
>   at 
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:190)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:101)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:74)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:34)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:279)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:296)
>   at 
> org.apache.flink.runtime.state.heap.HeapListState.add(HeapListState.java:77)
>   at 
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:442)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:206)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 7, Size: 5
>   at java.util.ArrayList.rangeCheck(ArrayList.java:653)
>   at java.util.ArrayList.get(ArrayList.java:429)
>   at 
> com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
>   at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805)
>   at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:728)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>   ... 22 more
> 2017-08-17 10:04:12,816 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - Freeing task resources for TriggerWindow(GlobalWindows(), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@6d36aa3c},
>  co.thirdwatch.trigger.TransactionTrigger@5707c1cb, 
> 

[GitHub] flink issue #4816: [hotfix][docs] CEP docs review to remove weasel words, fi...

[ChrisChinchilla]

Github user ChrisChinchilla commented on the issue:

https://github.com/apache/flink/pull/4816
  
@StephanEwen I noticed that just as I submitted it, will look next week :)


---

[jira] [Commented] (FLINK-7832) SlotManager should return number of registered slots

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203555#comment-16203555
 ] 

ASF GitHub Bot commented on FLINK-7832:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4823

[FLINK-7832] [flip6] Extend SlotManager to report free slots per TM

## What is the purpose of the change

Extend the `SlotManager` such that we count the free slots per 
`TaskManager`. This has the advantage that we don't have to iterate over all 
registered slots and aggregate their state in order to decide whether a 
TaskManager is idle or not. Moreover, it allows to easily query how many free 
slots every `TaskManager` still has.

## Brief change log

- Fail if slot belongs to a unregistered TaskManager
- Add more sanity checks
- Make the TaskManagerSlot state transitions clearer
- Introduce proper TaskManagerSlot state enum and state transitions
- Refactor SlotManager for better maintainability
- Add free slot counting

## Verifying this change

This change is already covered by existing tests, such as `SlotManagerTest`.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink extendSlotManager

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4823


commit 77ece83513d9a79f2f0c3e7024d91243026c334c
Author: Till 
Date:   2017-10-13T07:19:51Z

[FLINK-7832] [flip6] Extend SlotManager to report free slots per TM

Fail if slot belongs to a unregistered TaskManager

Add more sanity checks

Make the TaskManagerSlot state transitions clearer

Introduce proper TaskManagerSlot state enum

Refactor SlotManager for better maintainability




> SlotManager should return number of registered slots
> 
>
> Key: FLINK-7832
> URL: https://issues.apache.org/jira/browse/FLINK-7832
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> The {{SlotManager}} should provide information about the number of registered 
> slots for a {{TaskExecutor}} and how many of these slots are still free.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4823: [FLINK-7832] [flip6] Extend SlotManager to report ...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4823

[FLINK-7832] [flip6] Extend SlotManager to report free slots per TM

## What is the purpose of the change

Extend the `SlotManager` such that we count the free slots per 
`TaskManager`. This has the advantage that we don't have to iterate over all 
registered slots and aggregate their state in order to decide whether a 
TaskManager is idle or not. Moreover, it allows to easily query how many free 
slots every `TaskManager` still has.

## Brief change log

- Fail if slot belongs to a unregistered TaskManager
- Add more sanity checks
- Make the TaskManagerSlot state transitions clearer
- Introduce proper TaskManagerSlot state enum and state transitions
- Refactor SlotManager for better maintainability
- Add free slot counting

## Verifying this change

This change is already covered by existing tests, such as `SlotManagerTest`.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink extendSlotManager

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4823


commit 77ece83513d9a79f2f0c3e7024d91243026c334c
Author: Till 
Date:   2017-10-13T07:19:51Z

[FLINK-7832] [flip6] Extend SlotManager to report free slots per TM

Fail if slot belongs to a unregistered TaskManager

Add more sanity checks

Make the TaskManagerSlot state transitions clearer

Introduce proper TaskManagerSlot state enum

Refactor SlotManager for better maintainability




---

[GitHub] flink issue #4807: [FLINK-7810] Switch from custom Flakka to Akka 2.4.x

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4807
  
I think this is good now, +1


---

[jira] [Commented] (FLINK-7810) Switch from custom Flakka to Akka 2.4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203546#comment-16203546
 ] 

ASF GitHub Bot commented on FLINK-7810:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4807
  
I think this is good now, +1


> Switch from custom Flakka to Akka 2.4.x
> ---
>
> Key: FLINK-7810
> URL: https://issues.apache.org/jira/browse/FLINK-7810
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203540#comment-16203540
 ] 

ASF GitHub Bot commented on FLINK-5706:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4818
  
Thanks @steveloughran for the comments.

I am actually using Hadoop 2.8.1 here with AWS SDK 1.11.95.
The shaded artifacts are only a few MBs large, so this seems okay.


> Implement Flink's own S3 filesystem
> ---
>
> Key: FLINK-5706
> URL: https://issues.apache.org/jira/browse/FLINK-5706
> Project: Flink
>  Issue Type: New Feature
>  Components: filesystem-connector
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> As part of the effort to make Flink completely independent from Hadoop, Flink 
> needs its own S3 filesystem implementation. Currently Flink relies on 
> Hadoop's S3a and S3n file systems.
> An own S3 file system can be implemented using the AWS SDK. As the basis of 
> the implementation, the Hadoop File System can be used (Apache Licensed, 
> should be okay to reuse some code as long as we do a proper attribution).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4818: [FLINK-5706] [file systems] Add S3 file systems without H...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4818
  
Thanks @steveloughran for the comments.

I am actually using Hadoop 2.8.1 here with AWS SDK 1.11.95.
The shaded artifacts are only a few MBs large, so this seems okay.


---

[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203534#comment-16203534
 ] 

ASF GitHub Bot commented on FLINK-5706:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4818#discussion_r144546384
  
--- Diff: 
flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java
 ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.s3hadoop;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.FileSystemFactory;
+import org.apache.flink.runtime.fs.hdfs.HadoopFileSystem;
+import org.apache.flink.runtime.util.HadoopUtils;
+
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+
+/**
+ * Simple factory for the S3 file system.
+ */
+public class S3FileSystemFactory implements FileSystemFactory {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(S3FileSystemFactory.class);
+
+   /** The prefixes that Flink adds to the Hadoop config under 'fs.s3a.'. 
*/
+   private static final String[] CONFIG_PREFIXES = { "s3.", "s3a.", 
"fs.s3a." };
+
+   /** Keys that are replaced (after prefix replacement, to give a more 
uniform experience
+* across different file system implementations. */
+   private static final String[][] MIRRORED_CONFIG_KEYS = {
+   { "fs.s3a.access-key", "fs.s3a.access.key" },
+   { "fs.s3a.secret-key", "fs.s3a.secret.key" }
--- End diff --

Thanks, I think we will probably add more keys here over time.


> Implement Flink's own S3 filesystem
> ---
>
> Key: FLINK-5706
> URL: https://issues.apache.org/jira/browse/FLINK-5706
> Project: Flink
>  Issue Type: New Feature
>  Components: filesystem-connector
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> As part of the effort to make Flink completely independent from Hadoop, Flink 
> needs its own S3 filesystem implementation. Currently Flink relies on 
> Hadoop's S3a and S3n file systems.
> An own S3 file system can be implemented using the AWS SDK. As the basis of 
> the implementation, the Hadoop File System can be used (Apache Licensed, 
> should be okay to reuse some code as long as we do a proper attribution).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4818: [FLINK-5706] [file systems] Add S3 file systems wi...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4818#discussion_r144546384
  
--- Diff: 
flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java
 ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.s3hadoop;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.FileSystemFactory;
+import org.apache.flink.runtime.fs.hdfs.HadoopFileSystem;
+import org.apache.flink.runtime.util.HadoopUtils;
+
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+
+/**
+ * Simple factory for the S3 file system.
+ */
+public class S3FileSystemFactory implements FileSystemFactory {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(S3FileSystemFactory.class);
+
+   /** The prefixes that Flink adds to the Hadoop config under 'fs.s3a.'. 
*/
+   private static final String[] CONFIG_PREFIXES = { "s3.", "s3a.", 
"fs.s3a." };
+
+   /** Keys that are replaced (after prefix replacement, to give a more 
uniform experience
+* across different file system implementations. */
+   private static final String[][] MIRRORED_CONFIG_KEYS = {
+   { "fs.s3a.access-key", "fs.s3a.access.key" },
+   { "fs.s3a.secret-key", "fs.s3a.secret.key" }
--- End diff --

Thanks, I think we will probably add more keys here over time.


---

[jira] [Commented] (FLINK-6615) tmp directory not cleaned up on shutdown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203518#comment-16203518
 ] 

ASF GitHub Bot commented on FLINK-6615:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4787
  
I think here is an interesting example of why I am often hesitant with 
cleanup refactorings, unless there is a pressing need to clean up.

It is very hard to judge if the cleaned up version is really as robust as 
the original code. This change looked correct at a first glance, but the 
concurrency tests caught a subtle difference in some runs.


> tmp directory not cleaned up on shutdown
> 
>
> Key: FLINK-6615
> URL: https://issues.apache.org/jira/browse/FLINK-6615
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Bowen Li
>
> Steps to reproduce:
> 1) Stop task manager gracefully (kill -6 )
> 2) In the logs:
> {code}
> 2017-05-17 13:35:50,147 INFO  org.apache.zookeeper.ClientCnxn 
>   - EventThread shut down [main-EventThread]
> 2017-05-17 13:35:50,200 ERROR 
> org.apache.flink.runtime.io.disk.iomanager.IOManager  - IOManager 
> failed to properly clean up temp file directory: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47 
> [flink-akka.actor.default-dispatcher-2]
> java.nio.file.DirectoryNotEmptyException: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:154)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.postStop(TaskManager.scala:241)
>   at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> {code}
> Expected:
> * on shutdown delete non-empty directory anyway. 
> Notes:
> * after process terminated, I've checked 
> "/mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47" directory and 
> didn't find anything there. So it looks like timing issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4787: [FLINK-6615][core] simplify FileUtils

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4787
  
I think here is an interesting example of why I am often hesitant with 
cleanup refactorings, unless there is a pressing need to clean up.

It is very hard to judge if the cleaned up version is really as robust as 
the original code. This change looked correct at a first glance, but the 
concurrency tests caught a subtle difference in some runs.


---

[GitHub] flink issue #4787: [FLINK-6615][core] simplify FileUtils

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4787
  
Actually, going back here. I would like to not merge this after all. The 
reason being that in my test run, I found that this does not handle concurrent 
deletes correctly after all:
https://travis-ci.org/StephanEwen/incubator-flink/jobs/287502995


---

[jira] [Commented] (FLINK-6615) tmp directory not cleaned up on shutdown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203514#comment-16203514
 ] 

ASF GitHub Bot commented on FLINK-6615:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4787
  
Actually, going back here. I would like to not merge this after all. The 
reason being that in my test run, I found that this does not handle concurrent 
deletes correctly after all:
https://travis-ci.org/StephanEwen/incubator-flink/jobs/287502995


> tmp directory not cleaned up on shutdown
> 
>
> Key: FLINK-6615
> URL: https://issues.apache.org/jira/browse/FLINK-6615
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Bowen Li
>
> Steps to reproduce:
> 1) Stop task manager gracefully (kill -6 )
> 2) In the logs:
> {code}
> 2017-05-17 13:35:50,147 INFO  org.apache.zookeeper.ClientCnxn 
>   - EventThread shut down [main-EventThread]
> 2017-05-17 13:35:50,200 ERROR 
> org.apache.flink.runtime.io.disk.iomanager.IOManager  - IOManager 
> failed to properly clean up temp file directory: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47 
> [flink-akka.actor.default-dispatcher-2]
> java.nio.file.DirectoryNotEmptyException: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:154)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.postStop(TaskManager.scala:241)
>   at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> {code}
> Expected:
> * on shutdown delete non-empty directory anyway. 
> Notes:
> * after process terminated, I've checked 
> "/mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47" directory and 
> didn't find anything there. So it looks like timing issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-7484) CaseClassSerializer.duplicate() does not perform proper deep copy

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-7484:
---

Assignee: Aljoscha Krettek

> CaseClassSerializer.duplicate() does not perform proper deep copy
> -
>
> Key: FLINK-7484
> URL: https://issues.apache.org/jira/browse/FLINK-7484
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP, DataStream API, Scala API
>Affects Versions: 1.3.2
> Environment: Flink 1.3.2 , Yarn Cluster, FsStateBackend
>Reporter: Shashank Agarwal
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.4.0
>
>
> I am using many CEP's and Global Window. I am getting following error 
> sometimes and application  crashes. I have checked logically there's no flow 
> in the program. Here ItemPojo is a Pojo class and we are using 
> java.utill.List[ItemPojo]. We are using Scala DataStream API please find 
> attached logs.
> {code}
> 2017-08-17 10:04:12,814 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - TriggerWindow(GlobalWindows(), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@6d36aa3c},
>  co.thirdwatch.trigger.TransactionTrigger@5707c1cb, 
> WindowedStream.apply(WindowedStream.scala:582)) -> Flat Map -> Map -> Sink: 
> Saving CSV Features Sink (1/2) (06c0d4d231bc620ba9e7924b9b0da8d1) switched 
> from RUNNING to FAILED.
> com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: 
> Index: 7, Size: 5
> Serialization trace:
> category (co.thirdwatch.pojo.ItemPojo)
> underlying (scala.collection.convert.Wrappers$SeqWrapper)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:43)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:21)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:657)
>   at 
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:190)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:101)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:74)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:34)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:279)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:296)
>   at 
> org.apache.flink.runtime.state.heap.HeapListState.add(HeapListState.java:77)
>   at 
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:442)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:206)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 7, Size: 5
>   at java.util.ArrayList.rangeCheck(ArrayList.java:653)
>   at java.util.ArrayList.get(ArrayList.java:429)
>   at 
> com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
>   at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805)
>   at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:728)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>   ... 22 more
> 2017-08-17 10:04:12,816 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - Freeing task resources for TriggerWindow(GlobalWindows(), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@6d36aa3c},
>  co.thirdwatch.trigger.TransactionTrigger@5707c1cb, 
> WindowedStream.apply(WindowedStream.scala:582)) -> Flat Map -> Map -> Sink: 
> Saving CSV Features Sink (1/2) 

[jira] [Commented] (FLINK-7388) ProcessFunction.onTimer() sets processing time as timestamp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203500#comment-16203500
 ] 

ASF GitHub Bot commented on FLINK-7388:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4786
  
@fhueske is this what you had in mind?


> ProcessFunction.onTimer() sets processing time as timestamp
> ---
>
> Key: FLINK-7388
> URL: https://issues.apache.org/jira/browse/FLINK-7388
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Fabian Hueske
>Assignee: Bowen Li
> Fix For: 1.4.0
>
>
> The {{ProcessFunction.onTimer()}} method sets the current processing time as 
> event-time timestamp when it is called from a processing time timer.
> I don't think this behavior is useful. Processing time timestamps won't be 
> aligned with watermarks and are not deterministic. The only reason would be 
> to have _some_ value in the timestamp field. However, the behavior is very 
> subtle and might not be noticed by users.
> IMO, it would be better to erase the timestamp. This will cause downstream 
> operator that rely on timestamps to fail and notify the users that the logic 
> they implemented was probably not what they intended to do.
> What do you think [~aljoscha]?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4786: [FLINK-7388][DataStream API] ProcessFunction.onTimer() se...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4786
  
@fhueske is this what you had in mind?


---

[jira] [Closed] (FLINK-7414) Hardcode scala.version to 2.11 in flink-quickstart-scala

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-7414.
---
Resolution: Fixed

Fixed in 85b2f27067ce2fe5b16b3126c816b27e916477d8

> Hardcode scala.version to 2.11 in flink-quickstart-scala
> 
>
> Key: FLINK-7414
> URL: https://issues.apache.org/jira/browse/FLINK-7414
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.4.0
>
>
> Currently, the {{scala.binary.version}} of the Scala Quickstart is derived 
> from the {{scala.binary.version}} of Flink at the time when the Quickstart is 
> built. This means that whatever Scala version is active when we push the 
> Quickstart takes precedence. Currently, when deploying SNAPSHOT versions we 
> deploy 2.10 first, then 2.11, i.e. the 1.4-SNAPSHOT Scala Quickstart has 
> Scala version 2.11. The release script deploys first 2.11 and then 2.10, 
> meaning the final 1.4.0 Scala Quickstart would have 2.10.
> Simply fixing it to the latest supported (by Flink) will circumvent that 
> issue and users can easily change the Scala version in the Quickstart.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6273) Client can't connect to jobmanager whose hostname contains capital letters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203485#comment-16203485
 ] 

ASF GitHub Bot commented on FLINK-6273:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3688
  
@barcahead could you please close this because it has been superseded by 
#4812 


> Client can't connect to jobmanager whose hostname contains capital letters
> --
>
> Key: FLINK-6273
> URL: https://issues.apache.org/jira/browse/FLINK-6273
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Yelei Feng
>Assignee: Yelei Feng
>Priority: Blocker
> Fix For: 1.4.0
>
>
> In non-HA mode, if we set jobmanager.rpc.address to a hostname with some 
> capital letters, flink client can't connect to jobmananger.
> ERROR | [flink-akka.actor.default-dispatcher-4] | dropping message [class 
> akka.actor.ActorSelectionMessage] for non-local recipient 
> [Actor[akka.tcp://flink@szv1000258958:32586/]] arriving at 
> [akka.tcp://flink@szv1000258958:32586] inbound addresses are 
> [akka.tcp://flink@SZV1000258958:32586] | akka.remote.EndpointWriter 
> (Slf4jLogger.scala:65)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-6273) Client can't connect to jobmanager whose hostname contains capital letters

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-6273.
---
Resolution: Duplicate

This has been superseded by FLINK-7540

> Client can't connect to jobmanager whose hostname contains capital letters
> --
>
> Key: FLINK-6273
> URL: https://issues.apache.org/jira/browse/FLINK-6273
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Yelei Feng
>Assignee: Yelei Feng
>Priority: Blocker
> Fix For: 1.4.0
>
>
> In non-HA mode, if we set jobmanager.rpc.address to a hostname with some 
> capital letters, flink client can't connect to jobmananger.
> ERROR | [flink-akka.actor.default-dispatcher-4] | dropping message [class 
> akka.actor.ActorSelectionMessage] for non-local recipient 
> [Actor[akka.tcp://flink@szv1000258958:32586/]] arriving at 
> [akka.tcp://flink@szv1000258958:32586] inbound addresses are 
> [akka.tcp://flink@SZV1000258958:32586] | akka.remote.EndpointWriter 
> (Slf4jLogger.scala:65)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #3688: [FLINK-6273] [core] don't convert hostname to lower cases...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3688
  
@barcahead could you please close this because it has been superseded by 
#4812 


---

[GitHub] flink issue #4801: [FLINK-7812] Log system resources metrics

[pnowojski]

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/4801
  
As on dev mailing list discussion, this feature uses 
https://github.com/oshi/oshi library licensed under EPL 1.0. It seems to be 
compatible with ours: https://www.apache.org/legal/resolved.html .

It has minimal external dependencies. Question is whether we want to shade 
everything that we add?

Definitely we could unify config options if we want to do that.


---

[jira] [Closed] (FLINK-7835) Fix duplicate() method in NFASerializer

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-7835.
-
Resolution: Fixed

Merged on master with ff9cefb36c70a9b6c55f607fc2b56644c57f7057

> Fix duplicate() method in NFASerializer
> ---
>
> Key: FLINK-7835
> URL: https://issues.apache.org/jira/browse/FLINK-7835
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Affects Versions: 1.3.0, 1.3.1, 1.3.2
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7763) TableSinkITCase fails with "object reuse" enabled

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203440#comment-16203440
 ] 

Fabian Hueske commented on FLINK-7763:
--

Fixed for 1.4.0 with 57333c62271253248bf3699be31ae7224e97de75

Will port the fix for 1.3.x later.

> TableSinkITCase fails with "object reuse" enabled
> -
>
> Key: FLINK-7763
> URL: https://issues.apache.org/jira/browse/FLINK-7763
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Aljoscha Krettek
>Assignee: Fabian Hueske
>Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> Set {{objectReuse}} to {{true}} in {{ExecutionConfig}} to reproduce the 
> failing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7835) Fix duplicate() method in NFASerializer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203428#comment-16203428
 ] 

ASF GitHub Bot commented on FLINK-7835:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4821


> Fix duplicate() method in NFASerializer
> ---
>
> Key: FLINK-7835
> URL: https://issues.apache.org/jira/browse/FLINK-7835
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Affects Versions: 1.3.0, 1.3.1, 1.3.2
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7812) Log system resources as metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203432#comment-16203432
 ] 

ASF GitHub Bot commented on FLINK-7812:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/4801
  
As on dev mailing list discussion, this feature uses 
https://github.com/oshi/oshi library licensed under EPL 1.0. It seems to be 
compatible with ours: https://www.apache.org/legal/resolved.html .

It has minimal external dependencies. Question is whether we want to shade 
everything that we add?

Definitely we could unify config options if we want to do that.


> Log system resources as metrics
> ---
>
> Key: FLINK-7812
> URL: https://issues.apache.org/jira/browse/FLINK-7812
> Project: Flink
>  Issue Type: New Feature
>  Components: Metrics
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4821: [FLINK-7835][cep] Fix duplicate() in NFASerializer...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4821


---

[jira] [Commented] (FLINK-7835) Fix duplicate() method in NFASerializer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203427#comment-16203427
 ] 

ASF GitHub Bot commented on FLINK-7835:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4821
  
Merging this.


> Fix duplicate() method in NFASerializer
> ---
>
> Key: FLINK-7835
> URL: https://issues.apache.org/jira/browse/FLINK-7835
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Affects Versions: 1.3.0, 1.3.1, 1.3.2
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4821: [FLINK-7835][cep] Fix duplicate() in NFASerializer.

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4821
  
Merging this.


---

[jira] [Commented] (FLINK-7810) Switch from custom Flakka to Akka 2.4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203412#comment-16203412
 ] 

ASF GitHub Bot commented on FLINK-7810:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144530220
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/taskmanager/TaskManagerComponentsStartupShutdownTest.java
 ---
@@ -202,8 +202,6 @@ protected void run() {
assertTrue(ioManager.isProperlyShutDown());
assertTrue(memManager.isShutdown());
} finally {
-   
TestingUtils.stopActorsGracefully(Arrays.asList(jobManager, taskManager));
--- End diff --

If the test succeeds, they will have been shutdown (the previous asserts 
also assert (hehe) this.

It seems the previous Akka version didn't mind another shutdown attempt but 
2.4 fails with an exception when you try to shutdown after already being shut 
down.


> Switch from custom Flakka to Akka 2.4.x
> ---
>
> Key: FLINK-7810
> URL: https://issues.apache.org/jira/browse/FLINK-7810
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4807: [FLINK-7810] Switch from custom Flakka to Akka 2.4...

[aljoscha]

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144530220
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/taskmanager/TaskManagerComponentsStartupShutdownTest.java
 ---
@@ -202,8 +202,6 @@ protected void run() {
assertTrue(ioManager.isProperlyShutDown());
assertTrue(memManager.isShutdown());
} finally {
-   
TestingUtils.stopActorsGracefully(Arrays.asList(jobManager, taskManager));
--- End diff --

If the test succeeds, they will have been shutdown (the previous asserts 
also assert (hehe) this.

It seems the previous Akka version didn't mind another shutdown attempt but 
2.4 fails with an exception when you try to shutdown after already being shut 
down.


---

[GitHub] flink pull request #4807: [FLINK-7810] Switch from custom Flakka to Akka 2.4...

[aljoscha]

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144529895
  
--- Diff: .travis.yml ---
@@ -61,31 +61,6 @@ matrix:
 - TEST="misc"
 - PROFILE="-Dhadoop.version=2.8.0"
 - CACHE_NAME=JDK8_H280_M
-- jdk: "openjdk8"
--- End diff --

You're right, of course. Undropping ...


---

[jira] [Commented] (FLINK-7810) Switch from custom Flakka to Akka 2.4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203410#comment-16203410
 ] 

ASF GitHub Bot commented on FLINK-7810:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144529895
  
--- Diff: .travis.yml ---
@@ -61,31 +61,6 @@ matrix:
 - TEST="misc"
 - PROFILE="-Dhadoop.version=2.8.0"
 - CACHE_NAME=JDK8_H280_M
-- jdk: "openjdk8"
--- End diff --

You're right, of course. Undropping ...


> Switch from custom Flakka to Akka 2.4.x
> ---
>
> Key: FLINK-7810
> URL: https://issues.apache.org/jira/browse/FLINK-7810
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5944) Flink should support reading Snappy Files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203408#comment-16203408
 ] 

ASF GitHub Bot commented on FLINK-5944:
---

Github user mlipkovich commented on the issue:

https://github.com/apache/flink/pull/4683
  
@aljoscha @haohui ,
thank you for your comments. Marked hadoop dependency as provided, set 
Hadoop codec as default one


> Flink should support reading Snappy Files
> -
>
> Key: FLINK-5944
> URL: https://issues.apache.org/jira/browse/FLINK-5944
> Project: Flink
>  Issue Type: New Feature
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ilya Ganelin
>Assignee: Mikhail Lipkovich
>  Labels: features
>
> Snappy is an extremely performant compression format that's widely used 
> offering fast decompression/compression. 
> This can be easily implemented by creating a SnappyInflaterInputStreamFactory 
> and updating the initDefaultInflateInputStreamFactories in FileInputFormat.
> Flink already includes the Snappy dependency in the project. 
> There is a minor gotcha in this. If we wish to use this with Hadoop, then we 
> must provide two separate implementations since Hadoop uses a different 
> version of the snappy format than Snappy Java (which is the xerial/snappy 
> included in Flink). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4683: [FLINK-5944] Support reading of Snappy files

[mlipkovich]

Github user mlipkovich commented on the issue:

https://github.com/apache/flink/pull/4683
  
@aljoscha @haohui ,
thank you for your comments. Marked hadoop dependency as provided, set 
Hadoop codec as default one


---

[GitHub] flink pull request #4807: [FLINK-7810] Switch from custom Flakka to Akka 2.4...

[aljoscha]

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144528124
  
--- Diff: flink-queryable-state/flink-queryable-state-java/pom.xml ---
@@ -110,8 +110,8 @@ under the License.

 

-   com.data-artisans
-   
flakka-testkit_${scala.binary.version}
+   com.typesafe.akka
--- End diff --

There was a test in there that was using Akka but I think that's not needed 
anymore. Will remove.

On a side node, currently, the queryable-state-client module contains both 
the client and server implementations (that's why there was the akka test) 
while flunk-runtime only has interfaces. Fully tweezing the client and server 
apart is still in progress.


---

[jira] [Commented] (FLINK-7810) Switch from custom Flakka to Akka 2.4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203402#comment-16203402
 ] 

ASF GitHub Bot commented on FLINK-7810:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144528124
  
--- Diff: flink-queryable-state/flink-queryable-state-java/pom.xml ---
@@ -110,8 +110,8 @@ under the License.

 

-   com.data-artisans
-   
flakka-testkit_${scala.binary.version}
+   com.typesafe.akka
--- End diff --

There was a test in there that was using Akka but I think that's not needed 
anymore. Will remove.

On a side node, currently, the queryable-state-client module contains both 
the client and server implementations (that's why there was the akka test) 
while flunk-runtime only has interfaces. Fully tweezing the client and server 
apart is still in progress.


> Switch from custom Flakka to Akka 2.4.x
> ---
>
> Key: FLINK-7810
> URL: https://issues.apache.org/jira/browse/FLINK-7810
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4822: [FLINK-7484] Perform proper deep copy in CaseClass...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/4822

[FLINK-7484] Perform proper deep copy in CaseClassSerializer.duplicate()

R: @kl0u 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink 
jira-7484-fix-caseclassserializer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4822.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4822


commit 9e1f435631c6b0af0e0327cb5dce76caedba5c4c
Author: Aljoscha Krettek 
Date:   2017-10-13T11:11:42Z

[FLINK-7484] Perform proper deep copy in CaseClassSerializer.duplicate()

This also adds a test that verifies the deep copy.




---

[jira] [Commented] (FLINK-7484) CaseClassSerializer.duplicate() does not perform proper deep copy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203390#comment-16203390
 ] 

ASF GitHub Bot commented on FLINK-7484:
---

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/4822

[FLINK-7484] Perform proper deep copy in CaseClassSerializer.duplicate()

R: @kl0u 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink 
jira-7484-fix-caseclassserializer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4822.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4822


commit 9e1f435631c6b0af0e0327cb5dce76caedba5c4c
Author: Aljoscha Krettek 
Date:   2017-10-13T11:11:42Z

[FLINK-7484] Perform proper deep copy in CaseClassSerializer.duplicate()

This also adds a test that verifies the deep copy.




> CaseClassSerializer.duplicate() does not perform proper deep copy
> -
>
> Key: FLINK-7484
> URL: https://issues.apache.org/jira/browse/FLINK-7484
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP, DataStream API, Scala API
>Affects Versions: 1.3.2
> Environment: Flink 1.3.2 , Yarn Cluster, FsStateBackend
>Reporter: Shashank Agarwal
>Priority: Blocker
> Fix For: 1.4.0
>
>
> I am using many CEP's and Global Window. I am getting following error 
> sometimes and application  crashes. I have checked logically there's no flow 
> in the program. Here ItemPojo is a Pojo class and we are using 
> java.utill.List[ItemPojo]. We are using Scala DataStream API please find 
> attached logs.
> {code}
> 2017-08-17 10:04:12,814 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - TriggerWindow(GlobalWindows(), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@6d36aa3c},
>  co.thirdwatch.trigger.TransactionTrigger@5707c1cb, 
> WindowedStream.apply(WindowedStream.scala:582)) -> Flat Map -> Map -> Sink: 
> Saving CSV Features Sink (1/2) (06c0d4d231bc620ba9e7924b9b0da8d1) switched 
> from RUNNING to FAILED.
> com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: 
> Index: 7, Size: 5
> Serialization trace:
> category (co.thirdwatch.pojo.ItemPojo)
> underlying (scala.collection.convert.Wrappers$SeqWrapper)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:43)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:21)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:657)
>   at 
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:190)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:101)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:74)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:34)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:279)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:296)
>   at 
> org.apache.flink.runtime.state.heap.HeapListState.add(HeapListState.java:77)
>   at 
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:442)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:206)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 7, Size: 5
>   at java.util.ArrayList.rangeCheck(ArrayList.java:653)
>   at 

[jira] [Updated] (FLINK-7484) CaseClassSerializer.duplicate() does not perform proper deep copy

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-7484:

Summary: CaseClassSerializer.duplicate() does not perform proper deep copy  
(was: com.esotericsoftware.kryo.KryoException: 
java.lang.IndexOutOfBoundsException: Index: 7, Size: 5)

> CaseClassSerializer.duplicate() does not perform proper deep copy
> -
>
> Key: FLINK-7484
> URL: https://issues.apache.org/jira/browse/FLINK-7484
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP, DataStream API, Scala API
>Affects Versions: 1.3.2
> Environment: Flink 1.3.2 , Yarn Cluster, FsStateBackend
>Reporter: Shashank Agarwal
>Priority: Blocker
> Fix For: 1.4.0
>
>
> I am using many CEP's and Global Window. I am getting following error 
> sometimes and application  crashes. I have checked logically there's no flow 
> in the program. Here ItemPojo is a Pojo class and we are using 
> java.utill.List[ItemPojo]. We are using Scala DataStream API please find 
> attached logs.
> {code}
> 2017-08-17 10:04:12,814 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - TriggerWindow(GlobalWindows(), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@6d36aa3c},
>  co.thirdwatch.trigger.TransactionTrigger@5707c1cb, 
> WindowedStream.apply(WindowedStream.scala:582)) -> Flat Map -> Map -> Sink: 
> Saving CSV Features Sink (1/2) (06c0d4d231bc620ba9e7924b9b0da8d1) switched 
> from RUNNING to FAILED.
> com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: 
> Index: 7, Size: 5
> Serialization trace:
> category (co.thirdwatch.pojo.ItemPojo)
> underlying (scala.collection.convert.Wrappers$SeqWrapper)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:43)
>   at com.twitter.chill.TraversableSerializer.read(Traversable.scala:21)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:657)
>   at 
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:190)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:101)
>   at 
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:74)
>   at 
> org.apache.flink.runtime.state.ArrayListSerializer.copy(ArrayListSerializer.java:34)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:279)
>   at 
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:296)
>   at 
> org.apache.flink.runtime.state.heap.HeapListState.add(HeapListState.java:77)
>   at 
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:442)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:206)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 7, Size: 5
>   at java.util.ArrayList.rangeCheck(ArrayList.java:653)
>   at java.util.ArrayList.get(ArrayList.java:429)
>   at 
> com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
>   at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805)
>   at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:728)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>   ... 22 more
> 2017-08-17 10:04:12,816 INFO  org.apache.flink.runtime.taskmanager.Task   
>   - Freeing task resources for TriggerWindow(GlobalWindows(), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@6d36aa3c},
>  co.thirdwatch.trigger.TransactionTrigger@5707c1cb, 
> 

[GitHub] flink issue #4816: [hotfix][docs] CEP docs review to remove weasel words, fi...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4816
  
Would be great to also have this in the 1.3 docs, but cherry-picking the 
committ does not work...


---

[GitHub] flink issue #4816: [hotfix][docs] CEP docs review to remove weasel words, fi...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4816
  
Thanks, merging this...


---

[GitHub] flink pull request #4807: [FLINK-7810] Switch from custom Flakka to Akka 2.4...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144518821
  
--- Diff: flink-queryable-state/flink-queryable-state-java/pom.xml ---
@@ -110,8 +110,8 @@ under the License.

 

-   com.data-artisans
-   
flakka-testkit_${scala.binary.version}
+   com.typesafe.akka
--- End diff --

I could not find any use of akka testkit in the queryable state client (and 
there should not be, design wise).

Was this just copy/paste forwarded and can be dropped?


---

[jira] [Commented] (FLINK-7810) Switch from custom Flakka to Akka 2.4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203365#comment-16203365
 ] 

ASF GitHub Bot commented on FLINK-7810:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144518199
  
--- Diff: .travis.yml ---
@@ -61,31 +61,6 @@ matrix:
 - TEST="misc"
 - PROFILE="-Dhadoop.version=2.8.0"
 - CACHE_NAME=JDK8_H280_M
-- jdk: "openjdk8"
--- End diff --

How about retaining these Travis profiles for now for the Hadoop and JDK 
versions?
Just bump the Scala dependency.


> Switch from custom Flakka to Akka 2.4.x
> ---
>
> Key: FLINK-7810
> URL: https://issues.apache.org/jira/browse/FLINK-7810
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4807: [FLINK-7810] Switch from custom Flakka to Akka 2.4...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144518199
  
--- Diff: .travis.yml ---
@@ -61,31 +61,6 @@ matrix:
 - TEST="misc"
 - PROFILE="-Dhadoop.version=2.8.0"
 - CACHE_NAME=JDK8_H280_M
-- jdk: "openjdk8"
--- End diff --

How about retaining these Travis profiles for now for the Hadoop and JDK 
versions?
Just bump the Scala dependency.


---

[jira] [Commented] (FLINK-7810) Switch from custom Flakka to Akka 2.4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203366#comment-16203366
 ] 

ASF GitHub Bot commented on FLINK-7810:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144519154
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/taskmanager/TaskManagerComponentsStartupShutdownTest.java
 ---
@@ -202,8 +202,6 @@ protected void run() {
assertTrue(ioManager.isProperlyShutDown());
assertTrue(memManager.isShutdown());
} finally {
-   
TestingUtils.stopActorsGracefully(Arrays.asList(jobManager, taskManager));
--- End diff --

Is this now happening automatically?


> Switch from custom Flakka to Akka 2.4.x
> ---
>
> Key: FLINK-7810
> URL: https://issues.apache.org/jira/browse/FLINK-7810
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7810) Switch from custom Flakka to Akka 2.4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203364#comment-16203364
 ] 

ASF GitHub Bot commented on FLINK-7810:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144518821
  
--- Diff: flink-queryable-state/flink-queryable-state-java/pom.xml ---
@@ -110,8 +110,8 @@ under the License.

 

-   com.data-artisans
-   
flakka-testkit_${scala.binary.version}
+   com.typesafe.akka
--- End diff --

I could not find any use of akka testkit in the queryable state client (and 
there should not be, design wise).

Was this just copy/paste forwarded and can be dropped?


> Switch from custom Flakka to Akka 2.4.x
> ---
>
> Key: FLINK-7810
> URL: https://issues.apache.org/jira/browse/FLINK-7810
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4807: [FLINK-7810] Switch from custom Flakka to Akka 2.4...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4807#discussion_r144519154
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/taskmanager/TaskManagerComponentsStartupShutdownTest.java
 ---
@@ -202,8 +202,6 @@ protected void run() {
assertTrue(ioManager.isProperlyShutDown());
assertTrue(memManager.isShutdown());
} finally {
-   
TestingUtils.stopActorsGracefully(Arrays.asList(jobManager, taskManager));
--- End diff --

Is this now happening automatically?


---

[GitHub] flink issue #4801: [FLINK-7812] Log system resources metrics

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4801
  
Thanks for this addition. Few comments:

  - Please try to follow the common (though not enforced) code style when 
it comes to empty lines between class declarations, fields, methods, etc.

  - What is the license of the new dependency? Can you send a link so we 
can verify that?

  - If possible, let's shade that dependency, avoid increasing the 
dependency footprint.

  - Can we harmonize the config keys between the memory logger and the new 
system metrics?


---

[jira] [Commented] (FLINK-7812) Log system resources as metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203354#comment-16203354
 ] 

ASF GitHub Bot commented on FLINK-7812:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4801
  
Thanks for this addition. Few comments:

  - Please try to follow the common (though not enforced) code style when 
it comes to empty lines between class declarations, fields, methods, etc.

  - What is the license of the new dependency? Can you send a link so we 
can verify that?

  - If possible, let's shade that dependency, avoid increasing the 
dependency footprint.

  - Can we harmonize the config keys between the memory logger and the new 
system metrics?


> Log system resources as metrics
> ---
>
> Key: FLINK-7812
> URL: https://issues.apache.org/jira/browse/FLINK-7812
> Project: Flink
>  Issue Type: New Feature
>  Components: Metrics
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6505) Proactively cleanup local FS for RocksDBKeyedStateBackend on startup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203348#comment-16203348
 ] 

ASF GitHub Bot commented on FLINK-6505:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4798
  
Probably good change for now.

I think in the long run, the TaskManager should give each Task a 
sub-directory and make sure that sub directory is cleared whenever tasks 
finish/cancel/fail. That way this safety net is more general.


> Proactively cleanup local FS for RocksDBKeyedStateBackend on startup
> 
>
> Key: FLINK-6505
> URL: https://issues.apache.org/jira/browse/FLINK-6505
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Stefan Richter
>Assignee: Bowen Li
> Fix For: 1.4.0
>
>
> In {{RocksDBKeyedStateBackend}}, the {{instanceBasePath}} is cleared on 
> {{dispose()}}. I think it might make sense to also clear this directory when 
> the backend is created, in case something crashed and the backend never 
> reached {{dispose()}}. At least for previous runs of the same job, we can 
> know what to delete on restart. 
> In general, it is very important for this backend to clean up the local FS, 
> because the local quota might be very limited compared to the DFS. And a node 
> that runs out of local disk space can bring down the whole job, with no way 
> to recover (it might always get rescheduled to that node).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4798: [FLINK-6505] Proactively cleanup local FS for RocksDBKeye...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4798
  
Probably good change for now.

I think in the long run, the TaskManager should give each Task a 
sub-directory and make sure that sub directory is cleared whenever tasks 
finish/cancel/fail. That way this safety net is more general.


---

[jira] [Commented] (FLINK-6615) tmp directory not cleaned up on shutdown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203341#comment-16203341
 ] 

ASF GitHub Bot commented on FLINK-6615:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4787
  
Fair enough. Seems there are tests for the behavior already, so +1 to merge 
this

Merging...


> tmp directory not cleaned up on shutdown
> 
>
> Key: FLINK-6615
> URL: https://issues.apache.org/jira/browse/FLINK-6615
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Bowen Li
>
> Steps to reproduce:
> 1) Stop task manager gracefully (kill -6 )
> 2) In the logs:
> {code}
> 2017-05-17 13:35:50,147 INFO  org.apache.zookeeper.ClientCnxn 
>   - EventThread shut down [main-EventThread]
> 2017-05-17 13:35:50,200 ERROR 
> org.apache.flink.runtime.io.disk.iomanager.IOManager  - IOManager 
> failed to properly clean up temp file directory: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47 
> [flink-akka.actor.default-dispatcher-2]
> java.nio.file.DirectoryNotEmptyException: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:154)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.postStop(TaskManager.scala:241)
>   at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> {code}
> Expected:
> * on shutdown delete non-empty directory anyway. 
> Notes:
> * after process terminated, I've checked 
> "/mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47" directory and 
> didn't find anything there. So it looks like timing issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4787: [FLINK-6615][core] simplify FileUtils

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/4787
  
Fair enough. Seems there are tests for the behavior already, so +1 to merge 
this

Merging...


---

[jira] [Commented] (FLINK-7835) Fix duplicate() method in NFASerializer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203338#comment-16203338
 ] 

ASF GitHub Bot commented on FLINK-7835:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4821#discussion_r144514570
  
--- Diff: 
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/NonDuplicatingTypeSerializer.java
 ---
@@ -190,13 +190,6 @@ public int hashCode() {
return typeSerializer.hashCode();
}
 
-   private void readObject(ObjectInputStream ois) throws IOException, 
ClassNotFoundException {
--- End diff --

removing this means that `elementList` and `identityMap` will not be 
initialised after deserialisation.


> Fix duplicate() method in NFASerializer
> ---
>
> Key: FLINK-7835
> URL: https://issues.apache.org/jira/browse/FLINK-7835
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Affects Versions: 1.3.0, 1.3.1, 1.3.2
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)