[jira] [Commented] (FLINK-6703) Document how to take a savepoint on YARN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178534#comment-16178534
 ] 

ASF GitHub Bot commented on FLINK-6703:
---

Github user bowenli86 closed the pull request at:

https://github.com/apache/flink/pull/4684


> Document how to take a savepoint on YARN
> 
>
> Key: FLINK-6703
> URL: https://issues.apache.org/jira/browse/FLINK-6703
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, YARN
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Bowen Li
>
> The documentation should have a separate entry for savepoint related CLI 
> commands in combination with YARN. It is currently not documented that you 
> have to supply the application id, nor how you can pass it.
> {code}
> ./bin/flink savepoint  -m yarn-cluster (-yid|-yarnapplicationId) 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4684: [FLINK-6703][savepoint/doc] Document how to take a savepo...

[bowenli86]

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4684
  
Sorry, I realized I used the wrong jira ticket number. I'm closing this 
one, and the new PR is at https://github.com/apache/flink/pull/4721


---

[jira] [Commented] (FLINK-6703) Document how to take a savepoint on YARN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178533#comment-16178533
 ] 

ASF GitHub Bot commented on FLINK-6703:
---

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4684
  
Sorry, I realized I used the wrong jira ticket number. I'm closing this 
one, and the new PR is at https://github.com/apache/flink/pull/4721


> Document how to take a savepoint on YARN
> 
>
> Key: FLINK-6703
> URL: https://issues.apache.org/jira/browse/FLINK-6703
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, YARN
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Bowen Li
>
> The documentation should have a separate entry for savepoint related CLI 
> commands in combination with YARN. It is currently not documented that you 
> have to supply the application id, nor how you can pass it.
> {code}
> ./bin/flink savepoint  -m yarn-cluster (-yid|-yarnapplicationId) 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6703) Document how to take a savepoint on YARN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178532#comment-16178532
 ] 

ASF GitHub Bot commented on FLINK-6703:
---

GitHub user bowenli86 opened a pull request:

https://github.com/apache/flink/pull/4721

[FLINK-6703][savepoint/doc] Document how to take a savepoint on YARN

## What is the purpose of the change

The documentation should have a separate entry for savepoint related CLI 
commands in combination with YARN. It is currently not documented that you have 
to supply the application id, nor how you can pass it.

## Brief change log

- *add instruction of taking savepoints on YARN to both Savepoint and CLI 
doc*

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

## Does this pull request potentially affect one of the following parts:

none

## Documentation

none



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bowenli86/flink FLINK-6703

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4721.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4721


commit 1d445fc5c995971dabece8e956c015832a676c18
Author: Bowen Li 
Date:   2017-09-25T04:56:58Z

[FLINK-6703][savepoint/doc] Document how to take a savepoint on YARN




> Document how to take a savepoint on YARN
> 
>
> Key: FLINK-6703
> URL: https://issues.apache.org/jira/browse/FLINK-6703
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, YARN
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Bowen Li
>
> The documentation should have a separate entry for savepoint related CLI 
> commands in combination with YARN. It is currently not documented that you 
> have to supply the application id, nor how you can pass it.
> {code}
> ./bin/flink savepoint  -m yarn-cluster (-yid|-yarnapplicationId) 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4684: [FLINK-6703][savepoint/doc] Document how to take a...

[bowenli86]

Github user bowenli86 closed the pull request at:

https://github.com/apache/flink/pull/4684


---

[GitHub] flink pull request #4721: [FLINK-6703][savepoint/doc] Document how to take a...

[bowenli86]

GitHub user bowenli86 opened a pull request:

https://github.com/apache/flink/pull/4721

[FLINK-6703][savepoint/doc] Document how to take a savepoint on YARN

## What is the purpose of the change

The documentation should have a separate entry for savepoint related CLI 
commands in combination with YARN. It is currently not documented that you have 
to supply the application id, nor how you can pass it.

## Brief change log

- *add instruction of taking savepoints on YARN to both Savepoint and CLI 
doc*

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

## Does this pull request potentially affect one of the following parts:

none

## Documentation

none



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bowenli86/flink FLINK-6703

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4721.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4721


commit 1d445fc5c995971dabece8e956c015832a676c18
Author: Bowen Li 
Date:   2017-09-25T04:56:58Z

[FLINK-6703][savepoint/doc] Document how to take a savepoint on YARN




---

[jira] [Updated] (FLINK-7394) Manage exclusive buffers in RemoteInputChannel

[zhijiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang updated FLINK-7394:

Summary: Manage exclusive buffers in RemoteInputChannel  (was: Implement 
basic InputChannel for credit-based logic)

> Manage exclusive buffers in RemoteInputChannel
> --
>
> Key: FLINK-7394
> URL: https://issues.apache.org/jira/browse/FLINK-7394
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: zhijiang
>Assignee: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control. 
> The basic works are:
> * Exclusive buffers are assigned to {{RemoteInputChannel}} after created by 
> {{SingleInputGate}}.
> * {{RemoteInputChannel}} implements {{BufferRecycler}} interface to manage 
> the exclusive buffers itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7394) Implement basic InputChannel for credit-based logic

[zhijiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang updated FLINK-7394:

Description: 
This is a part of work for credit-based network flow control. 

The basic works are:

* Exclusive buffers are assigned to {{RemoteInputChannel}} after created by 
{{SingleInputGate}}.
* {{RemoteInputChannel}} implements {{BufferRecycler}} interface to manage the 
exclusive buffers itself.

  was:
This is a part of work for credit-based network flow control. 

The basic works are:

* Propose the {{BufferListener}} interface for notifying buffer availability 
and buffer destroyed.
* {{RemoteInputChannel}} implements {{BufferRecycler}} interface to manage the 
exclusive buffers itself.
* {{RemoteInputChannel}} implements {{BufferListener}} interface to be notified 
repeatedly .
* {{RemoteInputChannel}} maintains and notifies of unannounced credit.
* {{RemoteInputChannel}} maintains current sender backlog to trigger requests 
of floating buffers.



> Implement basic InputChannel for credit-based logic
> ---
>
> Key: FLINK-7394
> URL: https://issues.apache.org/jira/browse/FLINK-7394
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: zhijiang
>Assignee: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control. 
> The basic works are:
> * Exclusive buffers are assigned to {{RemoteInputChannel}} after created by 
> {{SingleInputGate}}.
> * {{RemoteInputChannel}} implements {{BufferRecycler}} interface to manage 
> the exclusive buffers itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178493#comment-16178493
 ] 

mingleizhang commented on FLINK-3829:
-

[~yew1eb] Me to. You are very welcome ~

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3829) flink-java

[Hai Zhou_UTC+8 (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178487#comment-16178487
 ] 

Hai Zhou_UTC+8 commented on FLINK-3829:
---

Ha ha, [~mingleizhang] you are interesting, I am very happy to work with you.

Peace,
Rice

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178484#comment-16178484
 ] 

mingleizhang commented on FLINK-3829:
-

Hey,[~yew1eb] I am also a newbie. So, that is WHY I am angry. Not so angry you 
said, just a little bit angry instead. we should be friendly to each other. 
Only experts not angry with us because they have all they want. We need to grow 
with Flink. ;)(*);)

Peace,
Rice

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3829) flink-java

[Hai Zhou_UTC+8 (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178480#comment-16178480
 ] 

Hai Zhou_UTC+8 commented on FLINK-3829:
---

[~mingleizhang], Thank you very much, can you tell me why you are so angry.;)

from now on, I will correct my bad behavior.

BTW, in issue [FLINK-6444], We should discuss first, instead of directly modify 
the issue assigned to their own.  you are familiar with the apache way, maybe 
you should be friendly to newbie.:D

Best,
Hai Zhou

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4720: Release 1.3.2 rc3

[yaochunnian]

GitHub user yaochunnian opened a pull request:

https://github.com/apache/flink/pull/4720

Release 1.3.2 rc3

*Thank you very much for contributing to Apache Flink - we are happy that 
you want to help us improve Flink. To help the community review your 
contribution in the best possible way, please go through the checklist below, 
which will get the contribution into a shape in which it can be best reviewed.*

*Please understand that we do not do this to make contributions to Flink a 
hassle. In order to uphold a high standard of quality for code contributions, 
while at the same time managing a large number of contributions, we need 
contributors to prepare the contributions well, and give reviewers enough 
contextual information for the review. Please also understand that 
contributions that do not follow this guide will take longer to review and thus 
typically be picked up with lower priority by the community.*

## Contribution Checklist

  - Make sure that the pull request corresponds to a [JIRA 
issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are 
made for typos in JavaDoc or documentation files, which need no JIRA issue.
  
  - Name the pull request in the form "[FLINK-] [component] Title of 
the pull request", where *FLINK-* should be replaced by the actual issue 
number. Skip *component* if you are unsure about which is the best component.
  Typo fixes that have no associated JIRA issue should be named following 
this pattern: `[hotfix] [docs] Fix typo in event time introduction` or 
`[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.

  - Fill out the template below to describe the changes contributed by the 
pull request. That will give reviewers the context they need to do the review.
  
  - Make sure that the change passes the automated tests, i.e., `mvn clean 
verify` passes. You can set up Travis CI to do that following [this 
guide](http://flink.apache.org/contribute-code.html#best-practices).

  - Each pull request should address only one issue, not mix up code from 
multiple issues.
  
  - Each commit in the pull request has a meaningful commit message 
(including the JIRA id)

  - Once all items of the checklist are addressed, remove the above text 
and this checklist, leaving only the filled out template below.


**(The sections below can be removed for hotfixes of typos)**

## What is the purpose of the change

*(For example: This pull request makes task deployment go through the blob 
server, rather than through RPC. That way we avoid re-transferring them on each 
deployment (during recovery).)*


## Brief change log

*(for example:)*
  - *The TaskInfo is stored in the blob store on job creation time as a 
persistent artifact*
  - *Deployments RPC transmits only the blob storage reference*
  - *TaskManagers retrieve the TaskInfo from the blob cache*


## Verifying this change

*(Please pick either of the following options)*

This change is a trivial rework / code cleanup without any test coverage.

*(or)*

This change is already covered by existing tests, such as *(please describe 
tests)*.

*(or)*

This change added tests and can be verified as follows:

*(example:)*
  - *Added integration tests for end-to-end deployment with large payloads 
(100MB)*
  - *Extended integration test for recovery after master (JobManager) 
failure*
  - *Added test that validates that TaskInfo is transferred only once 
across recoveries*
  - *Manually verified the change by running a 4 node cluser with 2 
JobManagers and 4 TaskManagers, a stateful streaming program, and killing one 
JobManager and two TaskManagers during the execution, verifying that recovery 
happens correctly.*

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes / no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / no)
  - The serializers: (yes / no / don't know)
  - The runtime per-record code paths (performance sensitive): (yes / no / 
don't know)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)

## Documentation

  - Does this pull request introduce a new feature? (yes / no)
  - If yes, how is the feature documented? (not applicable / docs / 
JavaDocs / not documented)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/flink release-1.3.2-rc3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4720.patch

To close this pull request, make a commit to your 

[jira] [Commented] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178461#comment-16178461
 ] 

mingleizhang commented on FLINK-3829:
-

[~yew1eb] I was not pointing to this jira. You know what jira number we were 
talking. Parent issue belongs to you, but not indicate it's sub issue all 
belongs to you. Yes,We should talk first and then carry out our work next time. 
Remember. Dude.

Peace.
Rice.

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3829) flink-java

[Hai Zhou_UTC+8 (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178454#comment-16178454
 ] 

Hai Zhou_UTC+8 commented on FLINK-3829:
---

[~mingleizhang], I do not know what to say:P

Long ago, this issue's parent issue[FLINK-3827] assigned to me. If you love 
this issue, Maybe we'll talk first.;)

or you can create and fix other modules eg. flink-yarn, flink-yarn-tests, 
flink-metrics...

Best, 
Hai Zhou


> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang reassigned FLINK-3829:
---

Assignee: (was: mingleizhang)

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178437#comment-16178437
 ] 

mingleizhang commented on FLINK-3829:
-

Hey, [~yew1eb] You can assign it to yourself now as I find you gave a patch to 
this jira. And I will unassign it soon. BTW I know what an apache way looks 
like. But please remember it is you do not know what called polite first. I 
think you gotta consult with others when someone else has the intention to do 
this thing before, rather than directly rushed over here and say, WOW. WOW. 
WOW." You should first learn how to respect. And then carry out your work.

Good luck, Man. I quit from this.

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: mingleizhang
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7393) Move unit tests of KinesisConfigUtil from FlinkKinesisConsumerTest to KinesisConfigUtilTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178436#comment-16178436
 ] 

ASF GitHub Bot commented on FLINK-7393:
---

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4708
  
cc @tzulitai 


> Move unit tests of KinesisConfigUtil from FlinkKinesisConsumerTest to 
> KinesisConfigUtilTest
> ---
>
> Key: FLINK-7393
> URL: https://issues.apache.org/jira/browse/FLINK-7393
> Project: Flink
>  Issue Type: Test
>  Components: Kinesis Connector
>Affects Versions: 1.3.2
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Minor
> Fix For: 1.4.0
>
>
> Right now, 
> [{{FlinkKinesisConsumerTest}}|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumerTest.java]
>  has lots of tests that actually should belong to {{KinesisConfigUtil}}, e.g. 
> all the {{validateXxxConfiguration()}}
> We need to move those tests out to a new file {{KinesisConfigUtilTest}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4708: [FLINK-7393][kinesis connector] Move unit tests that shou...

[bowenli86]

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4708
  
cc @tzulitai 


---

[jira] [Assigned] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang reassigned FLINK-3829:
---

Assignee: mingleizhang

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: mingleizhang
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6444) Add a check that '@VisibleForTesting' methods are only used in tests

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178428#comment-16178428
 ] 

mingleizhang commented on FLINK-6444:
-

Hey, [~yew1eb]. I think you can just use reflections do it in a simple way. And 
I guess it is more easier than your method as it is just a insignificant 
change. Not suggest to write many codes for it. And it will take potential bug 
instead probably.

> Add a check that '@VisibleForTesting' methods are only used in tests
> 
>
> Key: FLINK-6444
> URL: https://issues.apache.org/jira/browse/FLINK-6444
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>
> Some methods are annotated with {{@VisibleForTesting}}. These methods should 
> only be called from tests.
> This is currently not enforced / checked during the build. We should add such 
> a check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLINK-4004) Do not pass custom flink kafka connector properties to Kafka to avoid warnings

[Luffy Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178409#comment-16178409
 ] 

Luffy Tsai edited comment on FLINK-4004 at 9/25/17 12:06 AM:
-

Hi,


Before I started working, I thought the solution of the issue was defined a 
filter function {{filterUnusedProperties}} and then just changed the line 
{{this.kafkaProperties = checkNotNull(kafkaProperties);}} in the 
[KafkaConsumerThread.java#L132|https://github.com/apache/flink/blob/4afca4b3a13b61c2754bc839c77ba4d4eb1d2da2/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java#L132],
 and finally the warning log would disappear.

However, I have had some problems.
There is a lot of room for improvement in the following solutions of getting 
properties of consumer config.

h6. 1. Get config from Kafka ConsumerConfig.

Unfortunately, the public method {{configNames}} in Kafka which was added from 
v0.10.1. It couldn't be used in the flink-connector-kafka-0.8 and 
flink-connector-kafka-0.9.
[ConsumerConfig#L417|https://github.com/apache/kafka/blob/2b663790733527488d7f33ebc47f383ff18b5a83/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L417]

Maybe we could get all public fields by the following code, but it's not a 
appropriate design pattern.
{{Field[] fields = ConsumerConfig.class.getFields();}}

h6. 2. Maintain consumer config in the Flink.

It sounds workable but dirty.


Could you give me some suggestions?
Thank you!


was (Author: paristsai):
Hi,


Before I started working, I thought the solution of the issue was defined a 
filter function {{filterUnusedProperties}} and then just changed the line 
{{this.kafkaProperties = checkNotNull(kafkaProperties);}} in the 
[KafkaConsumerThread.java#L132|https://github.com/apache/flink/blob/4afca4b3a13b61c2754bc839c77ba4d4eb1d2da2/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java#L132],
 and finally the warning log would disappear.

However, I have had some problems.
There is a lot of room for improvement in the following solutions of getting 
properties of consumer config.

h6. 1. Get config from Kafka ConsumerConfig.

Unfortunately, the public method {{configNames}} which was added from v0.10.1. 
It couldn't be used in the flink-connector-kafka-0.8 and 
flink-connector-kafka-0.9.
[ConsumerConfig#L417|https://github.com/apache/kafka/blob/2b663790733527488d7f33ebc47f383ff18b5a83/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L417]

Maybe we could get all public fields by the following code, but it's not a 
appropriate design pattern.
{{Field[] fields = ConsumerConfig.class.getFields();}}

h6. 2. Maintain consumer config in the Flink.

It sounds workable but dirty.


Could you give me some suggestions?
Thank you!

> Do not pass custom flink kafka connector properties to Kafka to avoid warnings
> --
>
> Key: FLINK-4004
> URL: https://issues.apache.org/jira/browse/FLINK-4004
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Luffy Tsai
>
> The FlinkKafkaConsumer has some custom properties, which we pass to the 
> KafkaConsumer as well (such as {{flink.poll-timeout}}). This causes Kafka to 
> log warnings about unused properties.
> We should not pass Flink-internal properties to Kafka, to avoid those 
> warnings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLINK-4004) Do not pass custom flink kafka connector properties to Kafka to avoid warnings

[Luffy Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178409#comment-16178409
 ] 

Luffy Tsai edited comment on FLINK-4004 at 9/25/17 12:05 AM:
-

Hi,


Before I started working, I thought the solution of the issue was defined a 
filter function {{filterUnusedProperties}} and then just changed the line 
{{this.kafkaProperties = checkNotNull(kafkaProperties);}} in the 
[KafkaConsumerThread.java#L132|https://github.com/apache/flink/blob/4afca4b3a13b61c2754bc839c77ba4d4eb1d2da2/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java#L132],
 and finally the warning log would disappear.

However, I have had some problems.
There is a lot of room for improvement in the following solutions of getting 
properties of consumer config.

h6. 1. Get config from Kafka ConsumerConfig.

Unfortunately, the public method {{configNames}} which was added from v0.10.1. 
It couldn't be used in the flink-connector-kafka-0.8 and 
flink-connector-kafka-0.9.
[ConsumerConfig#L417|https://github.com/apache/kafka/blob/2b663790733527488d7f33ebc47f383ff18b5a83/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L417]

Maybe we could get all public fields by the following code, but it's not a 
appropriate design pattern.
{{Field[] fields = ConsumerConfig.class.getFields();}}

h6. 2. Maintain consumer config in the Flink.

It sounds workable but dirty.


Could you give me some suggestions?
Thank you!


was (Author: paristsai):
Hi,


Before I started working, I thought the solution of the issue was defined a 
filter function {{filterUnusedProperties}} and then just changed the line 
{{this.kafkaProperties = checkNotNull(kafkaProperties);}} in the 
[KafkaConsumerThread.java#L132|https://github.com/apache/flink/blob/4afca4b3a13b61c2754bc839c77ba4d4eb1d2da2/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java#L132],
 and finally the warning log would disappear.

However, I have had some problems. The following methods are not good enough.

h6. 1. Get config from Kafka ConsumerConfig.

Unfortunately, the public method {{configNames}} which was added from v0.10.1. 
It couldn't be used in the flink-connector-kafka-0.8 and 
flink-connector-kafka-0.9.
[ConsumerConfig#L417|https://github.com/apache/kafka/blob/2b663790733527488d7f33ebc47f383ff18b5a83/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L417]

Maybe we could get all public fields by the following code, but it's not a 
appropriate design pattern.
{{Field[] fields = ConsumerConfig.class.getFields();}}

h6. 2. Maintain consumer config in the Flink.

It sounds workable but dirty.


Could you give me some suggestions?
Thank you!

> Do not pass custom flink kafka connector properties to Kafka to avoid warnings
> --
>
> Key: FLINK-4004
> URL: https://issues.apache.org/jira/browse/FLINK-4004
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Luffy Tsai
>
> The FlinkKafkaConsumer has some custom properties, which we pass to the 
> KafkaConsumer as well (such as {{flink.poll-timeout}}). This causes Kafka to 
> log warnings about unused properties.
> We should not pass Flink-internal properties to Kafka, to avoid those 
> warnings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-4004) Do not pass custom flink kafka connector properties to Kafka to avoid warnings

[Luffy Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178409#comment-16178409
 ] 

Luffy Tsai commented on FLINK-4004:
---

Hi,


Before I started working, I thought the solution of the issue was defined a 
filter function {{filterUnusedProperties}} and then just changed the line 
{{this.kafkaProperties = checkNotNull(kafkaProperties);}} in the 
[KafkaConsumerThread.java#L132|https://github.com/apache/flink/blob/4afca4b3a13b61c2754bc839c77ba4d4eb1d2da2/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java#L132],
 and finally the warning log would disappear.

However, I have had some problems. The following methods are not good enough.

h6. 1. Get config from Kafka ConsumerConfig.

Unfortunately, the public method {{configNames}} which was added from v0.10.1. 
It couldn't be used in the flink-connector-kafka-0.8 and 
flink-connector-kafka-0.9.
[ConsumerConfig#L417|https://github.com/apache/kafka/blob/2b663790733527488d7f33ebc47f383ff18b5a83/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L417]

Maybe we could get all public fields by the following code, but it's not a 
appropriate design pattern.
{{Field[] fields = ConsumerConfig.class.getFields();}}

h6. 2. Maintain consumer config in the Flink.

It sounds workable but dirty.


Could you give me some suggestions?
Thank you!

> Do not pass custom flink kafka connector properties to Kafka to avoid warnings
> --
>
> Key: FLINK-4004
> URL: https://issues.apache.org/jira/browse/FLINK-4004
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Luffy Tsai
>
> The FlinkKafkaConsumer has some custom properties, which we pass to the 
> KafkaConsumer as well (such as {{flink.poll-timeout}}). This causes Kafka to 
> log warnings about unused properties.
> We should not pass Flink-internal properties to Kafka, to avoid those 
> warnings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7488) TaskManagerHeapSizeCalculationJavaBashTest sometimes fails

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-7488:
--
Description: 

{code}
compareNetworkBufShellScriptWithJava(org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest)
  Time elapsed: 0.239 sec  <<< FAILURE!
org.junit.ComparisonFailure: Different network buffer memory sizes with 
configuration: {taskmanager.network.memory.fraction=0.1, 
taskmanager.memory.off-heap=false, taskmanager.memory.fraction=0.7, 
taskmanager.memory.size=-1, taskmanager.network.memory.max=1073741824, 
taskmanager.heap.mb=1000, taskmanager.network.memory.min=67108864} 
expected:<[]104857600> but was:<[Setting HADOOP_CONF_DIR=/etc/hadoop/conf 
because no HADOOP_CONF_DIR was set.Using the result of 'hadoop classpath' to 
augment the Hadoop classpath: 
/usr/hdp/2.5.0.0-1245/hadoop/conf:/usr/hdp/2.5.0.0-1245/hadoop/lib/*:/usr/hdp/2.5.0.0-1245/hadoop/.//*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/./:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/.//*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/.//*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/.//*:/usr/hdp/2.5.0.0-1245/tez/*:/usr/hdp/2.5.0.0-1245/tez/lib/*:/usr/hdp/2.5.0.0-1245/tez/conf]104857600>
  at org.junit.Assert.assertEquals(Assert.java:115)
  at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.compareNetworkBufJavaVsScript(TaskManagerHeapSizeCalculationJavaBashTest.java:235)
  at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.compareNetworkBufShellScriptWithJava(TaskManagerHeapSizeCalculationJavaBashTest.java:81)

compareHeapSizeShellScriptWithJava(org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest)
  Time elapsed: 0.16 sec  <<< FAILURE!
org.junit.ComparisonFailure: Different heap sizes with configuration: 
{taskmanager.network.memory.fraction=0.1, taskmanager.memory.off-heap=false, 
taskmanager.memory.fraction=0.7, taskmanager.memory.size=-1, 
taskmanager.network.memory.max=1073741824, taskmanager.heap.mb=1000, 
taskmanager.network.memory.min=67108864} expected:<[]1000> but was:<[Setting 
HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.Using the 
result of 'hadoop classpath' to augment the Hadoop classpath: 
/usr/hdp/2.5.0.0-1245/hadoop/conf:/usr/hdp/2.5.0.0-1245/hadoop/lib/*:/usr/hdp/2.5.0.0-1245/hadoop/.//*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/./:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/.//*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/.//*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/.//*:/usr/hdp/2.5.0.0-1245/tez/*:/usr/hdp/2.5.0.0-1245/tez/lib/*:/usr/hdp/2.5.0.0-1245/tez/conf]1000>
  at org.junit.Assert.assertEquals(Assert.java:115)
  at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.compareHeapSizeJavaVsScript(TaskManagerHeapSizeCalculationJavaBashTest.java:275)
  at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.compareHeapSizeShellScriptWithJava(TaskManagerHeapSizeCalculationJavaBashTest.java:110)
{code}
$HADOOP_CONF_DIR was not set prior to running the test.

  was:
{code}
compareNetworkBufShellScriptWithJava(org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest)
  Time elapsed: 0.239 sec  <<< FAILURE!
org.junit.ComparisonFailure: Different network buffer memory sizes with 
configuration: {taskmanager.network.memory.fraction=0.1, 
taskmanager.memory.off-heap=false, taskmanager.memory.fraction=0.7, 
taskmanager.memory.size=-1, taskmanager.network.memory.max=1073741824, 
taskmanager.heap.mb=1000, taskmanager.network.memory.min=67108864} 
expected:<[]104857600> but was:<[Setting HADOOP_CONF_DIR=/etc/hadoop/conf 
because no HADOOP_CONF_DIR was set.Using the result of 'hadoop classpath' to 
augment the Hadoop classpath: 
/usr/hdp/2.5.0.0-1245/hadoop/conf:/usr/hdp/2.5.0.0-1245/hadoop/lib/*:/usr/hdp/2.5.0.0-1245/hadoop/.//*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/./:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/.//*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/.//*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/.//*:/usr/hdp/2.5.0.0-1245/tez/*:/usr/hdp/2.5.0.0-1245/tez/lib/*:/usr/hdp/2.5.0.0-1245/tez/conf]104857600>
  at org.junit.Assert.assertEquals(Assert.java:115)
  at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.compareNetworkBufJavaVsScript(TaskManagerHeapSizeCalculationJavaBashTest.java:235)
  at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.compareNetworkBufShellScriptWithJava(TaskManagerHeapSizeCalculationJavaBashTest.java:81)

compareHeapSizeShellScriptWithJava(org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest)
  Time elapsed: 0.16 sec  <<< FAILURE!
org.junit.ComparisonFailure: Different 

[jira] [Updated] (FLINK-7495) AbstractUdfStreamOperator#initializeState() should be called in AsyncWaitOperator#initializeState()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-7495:
--
Description: 
{code}
recoveredStreamElements = context
  .getOperatorStateStore()
  .getListState(new ListStateDescriptor<>(STATE_NAME, 
inStreamElementSerializer));
{code}

Call to AbstractUdfStreamOperator#initializeState() should be added in the 
beginning

  was:
{code}
recoveredStreamElements = context
  .getOperatorStateStore()
  .getListState(new ListStateDescriptor<>(STATE_NAME, 
inStreamElementSerializer));
{code}
Call to AbstractUdfStreamOperator#initializeState() should be added in the 
beginning


> AbstractUdfStreamOperator#initializeState() should be called in 
> AsyncWaitOperator#initializeState()
> ---
>
> Key: FLINK-7495
> URL: https://issues.apache.org/jira/browse/FLINK-7495
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Fang Yong
>Priority: Minor
>
> {code}
> recoveredStreamElements = context
>   .getOperatorStateStore()
>   .getListState(new ListStateDescriptor<>(STATE_NAME, 
> inStreamElementSerializer));
> {code}
> Call to AbstractUdfStreamOperator#initializeState() should be added in the 
> beginning



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7525) Add config option to disable Cancel functionality on UI

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-7525:
--
Description: 
In this email thread 
http://search-hadoop.com/m/Flink/VkLeQlf0QOnc7YA?subj=Security+Control+of+running+Flink+Jobs+on+Flink+UI
 , Raja was asking for a way to control how users cancel Job(s).

Robert proposed adding a config option which disables the Cancel functionality.

  was:
In this email thread 
http://search-hadoop.com/m/Flink/VkLeQlf0QOnc7YA?subj=Security+Control+of+running+Flink+Jobs+on+Flink+UI
 , Raja was asking for a way to control how users cancel Job(s).


Robert proposed adding a config option which disables the Cancel functionality.


> Add config option to disable Cancel functionality on UI
> ---
>
> Key: FLINK-7525
> URL: https://issues.apache.org/jira/browse/FLINK-7525
> Project: Flink
>  Issue Type: Improvement
>  Components: Web Client, Webfrontend
>Reporter: Ted Yu
>
> In this email thread 
> http://search-hadoop.com/m/Flink/VkLeQlf0QOnc7YA?subj=Security+Control+of+running+Flink+Jobs+on+Flink+UI
>  , Raja was asking for a way to control how users cancel Job(s).
> Robert proposed adding a config option which disables the Cancel 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7631) Ineffective check in PageRank#open()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178375#comment-16178375
 ] 

Ted Yu commented on FLINK-7631:
---

Looking at rest of the class, the use of uniformlyDistributedScore doesn't seem 
to handle {{Infinity}}

> Ineffective check in PageRank#open()
> 
>
> Key: FLINK-7631
> URL: https://issues.apache.org/jira/browse/FLINK-7631
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> From 
> flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/linkanalysis/PageRank.java
>  :
> {code}
>   this.vertexCount = vertexCountIterator.hasNext() ? 
> vertexCountIterator.next().getValue() : 0;
>   this.uniformlyDistributedScore = ((1 - dampingFactor) + dampingFactor * 
> sumOfSinks) / this.vertexCount;
> {code}
> The check for vertexCountIterator.hasNext() should enclose the assignments to 
> both this.vertexCount and this.uniformlyDistributedScore
> Otherwise there may be divide by zero error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (FLINK-6333) Utilize Bloomfilters in RocksDb

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved FLINK-6333.
---
Resolution: Later

> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
>   new BlockBasedTableConfig()
>   .setBlockCacheSize(blockCacheSize)
>   .setBlockSize(blockSize)
>   .setFilter(new BloomFilter())
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #:

[heytitle]

Github user heytitle commented on the pull request:


https://github.com/apache/flink/commit/9016cce503b4d471b5a49f0abccc196945ada97e#commitcomment-24529391
  
In 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java:
In 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 on line 188:
Can we simplify it to the code below?
```
Class generatedClass = null
WeakReference fromCache = generatedClassCache.getOrDefault(cacheKey, 
null);

generatedClass = fromCache != null ? fromCache.get() : null;

if ( genenetedClass == null ) {
// cache miss
...
}
```

So, we don't need to introduce `cacheHit` variable.


---

[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter

[ggevay]

Github user ggevay commented on the issue:

https://github.com/apache/flink/pull/3511
  
Thanks @fhueske and @heytitle!

Most of the comments have been addressed. The two outstanding issues are
1. Addressing 
https://github.com/apache/flink/pull/3511#discussion_r139933797
2. Making sure that 
https://github.com/apache/flink/pull/3511#discussion_r139926965 indeed fixes 
the leak and doesn't introduce any new issues.

I think I can do both next week. Or, @heytitle , if you have time you could 
also work on them. (Please drop me an email if you start working on 1., to 
avoid both of us doing it. For 2., the more eyes on it, the better.)


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140664772
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.TemplateException;
+
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private TemplateManager templateManager;
+   private HashMap constructorCache;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.templateManager = TemplateManager.getInstance();
+   this.classCompiler = new SimpleCompiler();
+   this.constructorCache = new HashMap<>();
+   }
+
+   /**
+* A method to get a singleton instance
+* or create one if it hasn't been created yet.
+* @return
+*/
+   public static synchronized SorterFactory getInstance() {
+   if (sorterFactory == null){
+   sorterFactory = new SorterFactory();
+   }
+
+   return sorterFactory;
+   }
+
+
+   /**
+* Create a sorter for the given type comparator and
+* assign serializer, comparator and memory to the sorter.
+* @param serializer
+* @param comparator
+* @param memory
+* @return
+*/
+   public  InMemorySorter createSorter(ExecutionConfig config, 
TypeSerializer serializer, 

[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter

[ggevay]

Github user ggevay commented on the issue:

https://github.com/apache/flink/pull/3511
  
Oh, you are right @heytitle, thanks! Fixed in 
9016cce503b4d471b5a49f0abccc196945ada97e
These `WeakReferences` are scary :)


---

[GitHub] flink pull request #:

[heytitle]

Github user heytitle commented on the pull request:


https://github.com/apache/flink/commit/cce40c5cd4aaff446bb4bec8918d2fda37649e0a#commitcomment-24528598
  
In 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java:
In 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 on line 180:
Is it possible that the class will be disappear after the first `get()` 
from if's condition?


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[heytitle]

Github user heytitle commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140664033
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.Configuration;
+import freemarker.template.Template;
+import freemarker.template.TemplateException;
+import freemarker.template.TemplateExceptionHandler;
+import freemarker.template.Version;
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private HashMap constructorCache;
+   private final Template template;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.classCompiler = new SimpleCompiler();
+   
this.classCompiler.setParentClassLoader(this.getClass().getClassLoader());
--- End diff --

Ok, I got it now. I thought there is some things behind this parent class 
loader field and influence the existence of `classComplier`.


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663665
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.TemplateException;
+
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private TemplateManager templateManager;
+   private HashMap constructorCache;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.templateManager = TemplateManager.getInstance();
+   this.classCompiler = new SimpleCompiler();
+   this.constructorCache = new HashMap<>();
+   }
+
+   /**
+* A method to get a singleton instance
+* or create one if it hasn't been created yet.
+* @return
+*/
+   public static synchronized SorterFactory getInstance() {
+   if (sorterFactory == null){
+   sorterFactory = new SorterFactory();
+   }
+
+   return sorterFactory;
+   }
+
+
+   /**
+* Create a sorter for the given type comparator and
+* assign serializer, comparator and memory to the sorter.
+* @param serializer
+* @param comparator
+* @param memory
+* @return
+*/
+   public  InMemorySorter createSorter(ExecutionConfig config, 
TypeSerializer serializer, 

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663639
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterTemplateModel.java
 ---
@@ -0,0 +1,424 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * {@link SorterTemplateModel} is a class that implements code generation 
logic for a given
+ * {@link TypeComparator}.
+ *
+ * The swap and compare methods in {@link NormalizedKeySorter} work on 
a sequence of bytes.
+ * We speed up these operations by splitting this sequence of bytes into 
chunks that can
+ * be handled by primitive operations such as Integer and Long 
operations.
+ */
+class SorterTemplateModel {
+
+   // 

+   //   Constants
+   // 

+
+   static final String TEMPLATE_NAME = "sorter.ftlh";
+
+   /** We don't split to chunks above this size. */
+   private static final int SPLITTING_THRESHOLD = 32;
+
+   /**
+* POSSIBLE_CHUNK_SIZES must be in descending order,
+* because methods that using it are using greedy approach.
+*/
+   private static final Integer[] POSSIBLE_CHUNK_SIZES = {8, 4, 2, 1};
+
+   /** Mapping from chunk sizes to primitive operators. */
+   private static final HashMap byteOperatorMapping = new 
HashMap(){
+   {
+   put(8, "Long");
+   put(4, "Int");
+   put(2, "Short");
+   put(1, "Byte");
+   }
+   };
+
+   // 

+   //   Attributes
+   // 

+
+   private final TypeComparator typeComparator;
+
+   /**
+* Sizes of the chunks. Empty, if we are not splitting to chunks. (See 
calculateChunks())
+*/
+   private final ArrayList primitiveChunks;
+
+   private final String sorterName;
+
+   /**
+* Shows whether the order of records can be completely determined by 
the normalized
+* sorting key, or the sorter has to also deserialize records if their 
keys are equal to
+* really confirm the order.
+*/
+   private final boolean normalizedKeyFullyDetermines;
+
+   /**
+* Constructor.
+* @param typeComparator
+*The type information of underlying data
+*/
+   SorterTemplateModel(TypeComparator typeComparator){
+   this.typeComparator = typeComparator;
+
+   // number of bytes of the sorting key
+   int numKeyBytes;
+
+   // compute no. bytes for sorting records and check whether 
these bytes are just a prefix or not.
+   if (this.typeComparator.supportsNormalizedKey()) {
+   // compute the max normalized key length
+   int numPartialKeys;
+   try {
+   numPartialKeys = 
this.typeComparator.getFlatComparators().length;
+   } catch (Throwable t) {
+   numPartialKeys = 1;
+   }
+
+   int maxLen = 
Math.min(NormalizedKeySorter.DEFAULT_MAX_NORMALIZED_KEY_LEN, 
NormalizedKeySorter.MAX_NORMALIZED_KEY_LEN_PER_ELEMENT * numPartialKeys);
+
+   numKeyBytes = 

[jira] [Commented] (FLINK-3829) flink-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178304#comment-16178304
 ] 

ASF GitHub Bot commented on FLINK-3829:
---

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4719

[FLINK-3829][build] POM Cleanup flink-java

## What is the purpose of the change
This PR changes the flink-java pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-java 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]commons-cli:commons-cli:jar:1.3.1:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-java 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3829

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4719


commit 54f09e83b040ffe80176b856052ccd7f1eaff969
Author: yew1eb 
Date:   2017-09-24T18:13:43Z

POM Cleanup flink-java




> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663635
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterTemplateModel.java
 ---
@@ -0,0 +1,424 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * {@link SorterTemplateModel} is a class that implements code generation 
logic for a given
+ * {@link TypeComparator}.
+ *
+ * The swap and compare methods in {@link NormalizedKeySorter} work on 
a sequence of bytes.
+ * We speed up these operations by splitting this sequence of bytes into 
chunks that can
+ * be handled by primitive operations such as Integer and Long 
operations.
+ */
+class SorterTemplateModel {
+
+   // 

+   //   Constants
+   // 

+
+   static final String TEMPLATE_NAME = "sorter.ftlh";
+
+   /** We don't split to chunks above this size. */
+   private static final int SPLITTING_THRESHOLD = 32;
+
+   /**
+* POSSIBLE_CHUNK_SIZES must be in descending order,
+* because methods that using it are using greedy approach.
+*/
+   private static final Integer[] POSSIBLE_CHUNK_SIZES = {8, 4, 2, 1};
+
+   /** Mapping from chunk sizes to primitive operators. */
+   private static final HashMap byteOperatorMapping = new 
HashMap(){
+   {
+   put(8, "Long");
+   put(4, "Int");
+   put(2, "Short");
+   put(1, "Byte");
+   }
+   };
+
+   // 

+   //   Attributes
+   // 

+
+   private final TypeComparator typeComparator;
+
+   /**
+* Sizes of the chunks. Empty, if we are not splitting to chunks. (See 
calculateChunks())
+*/
+   private final ArrayList primitiveChunks;
+
+   private final String sorterName;
+
+   /**
+* Shows whether the order of records can be completely determined by 
the normalized
+* sorting key, or the sorter has to also deserialize records if their 
keys are equal to
+* really confirm the order.
+*/
+   private final boolean normalizedKeyFullyDetermines;
+
+   /**
+* Constructor.
+* @param typeComparator
+*The type information of underlying data
+*/
+   SorterTemplateModel(TypeComparator typeComparator){
+   this.typeComparator = typeComparator;
+
+   // number of bytes of the sorting key
+   int numKeyBytes;
+
+   // compute no. bytes for sorting records and check whether 
these bytes are just a prefix or not.
+   if (this.typeComparator.supportsNormalizedKey()) {
+   // compute the max normalized key length
+   int numPartialKeys;
+   try {
+   numPartialKeys = 
this.typeComparator.getFlatComparators().length;
+   } catch (Throwable t) {
+   numPartialKeys = 1;
+   }
+
+   int maxLen = 
Math.min(NormalizedKeySorter.DEFAULT_MAX_NORMALIZED_KEY_LEN, 
NormalizedKeySorter.MAX_NORMALIZED_KEY_LEN_PER_ELEMENT * numPartialKeys);
+
+   numKeyBytes = 

[jira] [Commented] (FLINK-7680) Add "Performance Tuning" section to docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178302#comment-16178302
 ] 

ASF GitHub Bot commented on FLINK-7680:
---

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663596
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -127,6 +127,12 @@
private long autoWatermarkInterval = 0;
 
/**
+* The flag determines whether a custom NormalizedKeySorter will be 
dynamically created
+* for underlying data
+*/
+   private boolean codeGenerationForSorterEnabled = false;
--- End diff --

36419beee7938768453d2335daa9d7c29af720c1

I created a Jira for performance tweaking:
https://issues.apache.org/jira/browse/FLINK-7680


> Add "Performance Tuning" section to docs
> 
>
> Key: FLINK-7680
> URL: https://issues.apache.org/jira/browse/FLINK-7680
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Gabor Gevay
>Priority: Minor
> Fix For: 1.4.0
>
>
> We could have a separate section in the docs about performance tuning (maybe 
> separately for batch and streaming jobs).
> It could include for example:
> - object reuse
> - serializer issues
> - semantic annotations
> - optimizer hints
> - sorter code generation (Flink-5734)
> See [~fhueske]'s suggestion here:
> https://github.com/apache/flink/pull/3511#discussion_r139917275



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4719: [FLINK-3829][build] POM Cleanup flink-java

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4719

[FLINK-3829][build] POM Cleanup flink-java

## What is the purpose of the change
This PR changes the flink-java pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-java 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]commons-cli:commons-cli:jar:1.3.1:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-java 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3829

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4719


commit 54f09e83b040ffe80176b856052ccd7f1eaff969
Author: yew1eb 
Date:   2017-09-24T18:13:43Z

POM Cleanup flink-java




---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663591
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.TemplateException;
+
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private TemplateManager templateManager;
+   private HashMap constructorCache;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.templateManager = TemplateManager.getInstance();
+   this.classCompiler = new SimpleCompiler();
--- End diff --

909b59e3174a5348c2ee550806d4711475f5c268
cce40c5cd4aaff446bb4bec8918d2fda37649e0a

This was a bit tricky, I hope I haven't messed it up. I will think about 
this a bit more in the next few days, and maybe do some more testing, to see 
that we are not keeping alive anything from past jobs. Unfortunately, I don't 
know how this could be tested in an automated way. I think I will test it 
manually by just submitting hundreds of jobs, and watching in a profiler that 
object counts are not growing.


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663593
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/NormalizedKeySorter.java
 ---
@@ -309,7 +309,7 @@ public boolean write(T record) throws IOException {
//   Access Utilities
// 


-   private long readPointer(int logicalPosition) {
+   protected long readPointer(int logicalPosition) {
--- End diff --

Thanks, I've made them `protected final`:
db14ac58d137ed39f63bc4f8d724622f4cae518f
5e31cf011d5b2d3a9e986414411e312310867c44


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663604
  
--- Diff: flink-runtime/pom.xml ---
@@ -204,6 +204,18 @@ under the License.
reflections

 
+   
+   org.freemarker
--- End diff --

84cb9a83efdb384d91d4c967b237fc12dca68774


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663599
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/TemplateManager.java
 ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import freemarker.template.Configuration;
+import freemarker.template.Template;
+import freemarker.template.TemplateException;
+import freemarker.template.TemplateExceptionHandler;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.io.Writer;
+import java.util.Map;
+
+/**
+ * {@link TemplateManager} is a singleton class that provides template 
rendering functionalities for code generation.
+ * Such functionalities are caching, writing generated code to a file.
--- End diff --

Sorry, you are right, we don't write to a file anymore. Also, the caching 
is not handled by this class anymore (it is handled by `SorterFactory`). So 
I've just realized that this class is not really needed, and moved its 
functionality into `SorterFactory`.
b8f1e53fa46b0ec7f9b7baefbb388c29be72f7ee


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663596
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -127,6 +127,12 @@
private long autoWatermarkInterval = 0;
 
/**
+* The flag determines whether a custom NormalizedKeySorter will be 
dynamically created
+* for underlying data
+*/
+   private boolean codeGenerationForSorterEnabled = false;
--- End diff --

36419beee7938768453d2335daa9d7c29af720c1

I created a Jira for performance tweaking:
https://issues.apache.org/jira/browse/FLINK-7680


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663603
  
--- Diff: flink-runtime/pom.xml ---
@@ -204,6 +204,18 @@ under the License.
reflections

 
+   
+   org.freemarker
+   freemarker
+   2.3.20
+   
+
+   
+   org.codehaus.janino
+   janino
+   2.7.5
--- End diff --

84cb9a83efdb384d91d4c967b237fc12dca68774


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140663585
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.TemplateException;
+
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private TemplateManager templateManager;
+   private HashMap constructorCache;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.templateManager = TemplateManager.getInstance();
+   this.classCompiler = new SimpleCompiler();
+   this.constructorCache = new HashMap<>();
+   }
+
+   /**
+* A method to get a singleton instance
+* or create one if it hasn't been created yet.
+* @return
+*/
+   public static synchronized SorterFactory getInstance() {
+   if (sorterFactory == null){
+   sorterFactory = new SorterFactory();
+   }
+
+   return sorterFactory;
+   }
+
+
+   /**
+* Create a sorter for the given type comparator and
+* assign serializer, comparator and memory to the sorter.
+* @param serializer
+* @param comparator
+* @param memory
+* @return
+*/
+   public  InMemorySorter createSorter(ExecutionConfig config, 
TypeSerializer serializer, 

[jira] [Commented] (FLINK-3832) flink-streaming-scala

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178296#comment-16178296
 ] 

ASF GitHub Bot commented on FLINK-3832:
---

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4718

[FLINK-3832][build] POM Cleanup flink-streaming-scala

## What is the purpose of the change
This PR changes the flink-streaming-scala pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-scala_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-runtime_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
[WARNING]org.scala-lang:scala-compiler:jar:2.11.11:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.scala-lang:scala-reflect:jar:2.11.11:compile
[WARNING]org.slf4j:slf4j-api:jar:1.7.7:compile
[WARNING]
org.apache.flink:flink-runtime_2.11:test-jar:tests:1.4-SNAPSHOT:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-scala_2.11 ---
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-api:jar:1.7.7:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3832

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4718.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4718


commit 8e0ff9db6c1b9c0896655135c1fdf2240d9bc3f4
Author: yew1eb 
Date:   2017-09-24T18:06:34Z

POM Cleanup flink-streaming-scala




> flink-streaming-scala
> -
>
> Key: FLINK-3832
> URL: https://issues.apache.org/jira/browse/FLINK-3832
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
> flink-streaming-scala_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-runtime_2.11:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> [WARNING]org.scala-lang:scala-compiler:jar:2.11.11:compile
> [WARNING]

[GitHub] flink pull request #4718: [FLINK-3832][build] POM Cleanup flink-streaming-sc...

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4718

[FLINK-3832][build] POM Cleanup flink-streaming-scala

## What is the purpose of the change
This PR changes the flink-streaming-scala pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-scala_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-runtime_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
[WARNING]org.scala-lang:scala-compiler:jar:2.11.11:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.scala-lang:scala-reflect:jar:2.11.11:compile
[WARNING]org.slf4j:slf4j-api:jar:1.7.7:compile
[WARNING]
org.apache.flink:flink-runtime_2.11:test-jar:tests:1.4-SNAPSHOT:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-scala_2.11 ---
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-api:jar:1.7.7:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3832

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4718.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4718


commit 8e0ff9db6c1b9c0896655135c1fdf2240d9bc3f4
Author: yew1eb 
Date:   2017-09-24T18:06:34Z

POM Cleanup flink-streaming-scala




---

[jira] [Created] (FLINK-7680) Add "Performance Tuning" section to docs

[Gabor Gevay (JIRA)]

Gabor Gevay created FLINK-7680:
--

 Summary: Add "Performance Tuning" section to docs
 Key: FLINK-7680
 URL: https://issues.apache.org/jira/browse/FLINK-7680
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Gabor Gevay
Priority: Minor
 Fix For: 1.4.0


We could have a separate section in the docs about performance tuning (maybe 
separately for batch and streaming jobs).

It could include for example:
- object reuse
- serializer issues
- semantic annotations
- optimizer hints
- sorter code generation (Flink-5734)

See [~fhueske]'s suggestion here:
https://github.com/apache/flink/pull/3511#discussion_r139917275



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-3832) flink-streaming-scala

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 updated FLINK-3832:
--
Description: 
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-scala_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-runtime_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
[WARNING]org.scala-lang:scala-compiler:jar:2.11.11:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.scala-lang:scala-reflect:jar:2.11.11:compile
[WARNING]org.slf4j:slf4j-api:jar:1.7.7:compile
[WARNING]
org.apache.flink:flink-runtime_2.11:test-jar:tests:1.4-SNAPSHOT:test


> flink-streaming-scala
> -
>
> Key: FLINK-3832
> URL: https://issues.apache.org/jira/browse/FLINK-3832
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
> flink-streaming-scala_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-runtime_2.11:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> [WARNING]org.scala-lang:scala-compiler:jar:2.11.11:compile
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]org.mockito:mockito-all:jar:1.10.19:test
> [WARNING]org.scala-lang:scala-reflect:jar:2.11.11:compile
> [WARNING]org.slf4j:slf4j-api:jar:1.7.7:compile
> [WARNING]
> org.apache.flink:flink-runtime_2.11:test-jar:tests:1.4-SNAPSHOT:test



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3831) flink-streaming-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178295#comment-16178295
 ] 

ASF GitHub Bot commented on FLINK-3831:
---

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4717

[FLINK-3831][build] POM Cleanup  flink-streaming-java

## What is the purpose of the change
This PR changes the flink-streaming-java pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-java_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]com.fasterxml.jackson.core:jackson-databind:jar:2.7.4:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
[WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.commons:commons-lang3:jar:3.3.2:compile
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-java_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test

```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3831

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4717


commit bdc0392e245de23ba1095f404b3fc7cf3c37d515
Author: yew1eb 
Date:   2017-09-04T15:50:44Z

 fix flink-streaming-java undeclared/unused dependencies




> flink-streaming-java
> 
>
> Key: FLINK-3831
> URL: https://issues.apache.org/jira/browse/FLINK-3831
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
> Environment: Apache Maven 3.3.9, Java version: 1.8.0_144
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
> flink-streaming-java_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
> [WARNING]commons-io:commons-io:jar:2.4:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]com.fasterxml.jackson.core:jackson-databind:jar:2.7.4:compile
> [WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
> [WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
> [WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.commons:commons-lang3:jar:3.3.2:compile
> [WARNING]org.powermock:powermock-core:jar:1.6.5:test
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA

[GitHub] flink pull request #4717: [FLINK-3831][build] POM Cleanup flink-streaming-ja...

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4717

[FLINK-3831][build] POM Cleanup  flink-streaming-java

## What is the purpose of the change
This PR changes the flink-streaming-java pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-java_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]com.fasterxml.jackson.core:jackson-databind:jar:2.7.4:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
[WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.commons:commons-lang3:jar:3.3.2:compile
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-streaming-java_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test

```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3831

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4717


commit bdc0392e245de23ba1095f404b3fc7cf3c37d515
Author: yew1eb 
Date:   2017-09-04T15:50:44Z

 fix flink-streaming-java undeclared/unused dependencies




---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140662898
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.Configuration;
+import freemarker.template.Template;
+import freemarker.template.TemplateException;
+import freemarker.template.TemplateExceptionHandler;
+import freemarker.template.Version;
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private HashMap constructorCache;
+   private final Template template;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.classCompiler = new SimpleCompiler();
+   
this.classCompiler.setParentClassLoader(this.getClass().getClassLoader());
--- End diff --

Why would `classCompiler` be removed?

The "parent classloader" of a `SimpleCompiler` is just a field in it, and 
`classCompiler.setParentClassLoader` just modifies this field. So I think it 
should be OK to call `classCompiler.setParentClassLoader` again and again with 
different user code classloaders for every job.


---

[GitHub] flink pull request #4716: [FLINK-3833][build] POM Cleanup flink-test-utils

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4716

[FLINK-3833][build] POM Cleanup flink-test-utils

## What is the purpose of the change
This PR changes the flink-test-utils pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-test-utils_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
[WARNING]
org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-1.0:compile
[WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:flink-clients_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.apache.curator:curator-test:jar:2.12.0:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-test-utils_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3833

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4716


commit b65fc918d05a5f930748fe4153d96a258726bb67
Author: yew1eb 
Date:   2017-09-24T17:44:27Z

POM Cleanup flink-test-utils




---

[jira] [Commented] (FLINK-3833) flink-test-utils

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178292#comment-16178292
 ] 

ASF GitHub Bot commented on FLINK-3833:
---

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4716

[FLINK-3833][build] POM Cleanup flink-test-utils

## What is the purpose of the change
This PR changes the flink-test-utils pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-test-utils_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
[WARNING]
org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-1.0:compile
[WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:flink-clients_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.apache.curator:curator-test:jar:2.12.0:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-test-utils_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3833

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4716


commit b65fc918d05a5f930748fe4153d96a258726bb67
Author: yew1eb 
Date:   2017-09-24T17:44:27Z

POM Cleanup flink-test-utils




> flink-test-utils
> 
>
> Key: FLINK-3833
> URL: https://issues.apache.org/jira/browse/FLINK-3833
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
> flink-test-utils_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
> [WARNING]io.netty:netty:jar:3.8.0.Final:compile
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
> [WARNING]commons-io:commons-io:jar:2.4:compile
> [WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
> [WARNING]org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-1.0:compile
> [WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.flink:flink-clients_2.11:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]

[jira] [Updated] (FLINK-3833) flink-test-utils

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 updated FLINK-3833:
--
Description: 
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-test-utils_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
[WARNING]org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-1.0:compile
[WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:flink-clients_2.11:jar:1.4-SNAPSHOT:compile
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.apache.curator:curator-test:jar:2.12.0:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test


> flink-test-utils
> 
>
> Key: FLINK-3833
> URL: https://issues.apache.org/jira/browse/FLINK-3833
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
> flink-test-utils_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
> [WARNING]io.netty:netty:jar:3.8.0.Final:compile
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]org.scala-lang:scala-library:jar:2.11.11:compile
> [WARNING]commons-io:commons-io:jar:2.4:compile
> [WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
> [WARNING]org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-1.0:compile
> [WARNING]org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-java:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.flink:flink-clients_2.11:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.mockito:mockito-all:jar:1.10.19:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.curator:curator-test:jar:2.12.0:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3830) flink-scala

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178285#comment-16178285
 ] 

ASF GitHub Bot commented on FLINK-3830:
---

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4715

[FLINK-3830][build] POM Cleanup flink-scala 

## What is the purpose of the change
This PR changes the flink-scala pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] — maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-scala_2.11 —
[WARNING] Used undeclared dependencies found:
[WARNING] org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING] org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING] org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
[WARNING] com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING] org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING] org.apache.commons:commons-lang3:jar:3.3.2:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING] org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING] org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING] com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING] log4j:log4j:jar:1.2.17:test
[WARNING] org.mockito:mockito-all:jar:1.10.19:test
[WARNING] org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING] 
org.apache.flink:flink-test-utils_2.11:test-jar:tests:1.4-SNAPSHOT:test
[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-scala_2.11 ---
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3830

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4715


commit c3acee6913a9bcb6382d3429e540a9536690802f
Author: yew1eb 
Date:   2017-09-24T17:31:05Z

fix flink-scala undeclared/unused dependencies




> flink-scala
> ---
>
> Key: FLINK-3830
> URL: https://issues.apache.org/jira/browse/FLINK-3830
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
> flink-scala_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.commons:commons-lang3:jar:3.3.2:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.mockito:mockito-all:jar:1.10.19:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]
> org.apache.flink:flink-test-utils_2.11:test-jar:tests:1.4-SNAPSHOT:test
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[heytitle]

Github user heytitle commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140662538
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.Configuration;
+import freemarker.template.Template;
+import freemarker.template.TemplateException;
+import freemarker.template.TemplateExceptionHandler;
+import freemarker.template.Version;
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private HashMap constructorCache;
+   private final Template template;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.classCompiler = new SimpleCompiler();
+   
this.classCompiler.setParentClassLoader(this.getClass().getClassLoader());
--- End diff --

Thanks for the comment.  I have a further comment on this

What I understand is that If `SorterFactory` is a singleton object and its 
parent class loader isn't the user one, it might happen that the next job will 
fail to create new code-generated sorters. This is due to the fact that 
`classComplier` is removed after the first job finishes and it is instantiated 
only once when `SorterFactory is created. 

Please correct me if I'm wrong.


---

[GitHub] flink pull request #4715: [FLINK-3830][build] POM Cleanup flink-scala

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4715

[FLINK-3830][build] POM Cleanup flink-scala 

## What is the purpose of the change
This PR changes the flink-scala pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] — maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-scala_2.11 —
[WARNING] Used undeclared dependencies found:
[WARNING] org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING] org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING] org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
[WARNING] com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING] org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING] org.apache.commons:commons-lang3:jar:3.3.2:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING] org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING] org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING] com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING] log4j:log4j:jar:1.2.17:test
[WARNING] org.mockito:mockito-all:jar:1.10.19:test
[WARNING] org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING] 
org.apache.flink:flink-test-utils_2.11:test-jar:tests:1.4-SNAPSHOT:test
[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-scala_2.11 ---
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3830

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4715


commit c3acee6913a9bcb6382d3429e540a9536690802f
Author: yew1eb 
Date:   2017-09-24T17:31:05Z

fix flink-scala undeclared/unused dependencies




---

[jira] [Updated] (FLINK-3830) flink-scala

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 updated FLINK-3830:
--
Description: 
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-scala_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.commons:commons-lang3:jar:3.3.2:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.mockito:mockito-all:jar:1.10.19:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]
org.apache.flink:flink-test-utils_2.11:test-jar:tests:1.4-SNAPSHOT:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test


> flink-scala
> ---
>
> Key: FLINK-3830
> URL: https://issues.apache.org/jira/browse/FLINK-3830
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
> flink-scala_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.flink:flink-test-utils-junit:jar:1.4-SNAPSHOT:test
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]org.apache.commons:commons-lang3:jar:3.3.2:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.mockito:mockito-all:jar:1.10.19:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]
> org.apache.flink:flink-test-utils_2.11:test-jar:tests:1.4-SNAPSHOT:test
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7577) flink-core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178283#comment-16178283
 ] 

ASF GitHub Bot commented on FLINK-7577:
---

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4714

[FLINK-7577][build] POM Cleanup flink-core

## What is the purpose of the change
This PR changes the flink-core pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-core 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING]org.objenesis:objenesis:jar:2.1:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.joda:joda-convert:jar:1.7:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-core 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING]org.objenesis:objenesis:jar:2.1:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-7577

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4714


commit 7ede14db9f96185c064535c77602c6230bd3e080
Author: yew1eb 
Date:   2017-09-04T15:30:58Z

fix flink-core undeclared/unused dependencies




> flink-core
> --
>
> Key: FLINK-7577
> URL: https://issues.apache.org/jira/browse/FLINK-7577
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
> Environment: Apache Maven 3.3.9, Java version: 1.8.0_144
>Reporter: Hai Zhou_UTC+8
>Assignee: Hai Zhou_UTC+8
>
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-core ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.powermock:powermock-core:jar:1.6.5:test
> [WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
> [WARNING]org.objenesis:objenesis:jar:2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.joda:joda-convert:jar:1.7:test
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4714: [FLINK-7577][build] POM Cleanup flink-core

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4714

[FLINK-7577][build] POM Cleanup flink-core

## What is the purpose of the change
This PR changes the flink-core pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-core 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING]org.objenesis:objenesis:jar:2.1:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.joda:joda-convert:jar:1.7:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ flink-core 
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING]org.objenesis:objenesis:jar:2.1:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-7577

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4714


commit 7ede14db9f96185c064535c77602c6230bd3e080
Author: yew1eb 
Date:   2017-09-04T15:30:58Z

fix flink-core undeclared/unused dependencies




---

[jira] [Commented] (FLINK-3828) flink-runtime

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178281#comment-16178281
 ] 

ASF GitHub Bot commented on FLINK-3828:
---

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4713

[FLINK-3828][build] POM Cleanup flink-runtime

## What is the purpose of the change

This PR changes the flink-runtime pom to

not contain unused dependencies
contain all used dependencies

## Brief change log

## Verifying this change
_mvn dependency:analyze_
```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-runtime_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.curator:curator-client:jar:2.12.0:compile
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING]com.google.code.findbugs:annotations:jar:2.0.1:test
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]com.typesafe:config:jar:1.2.1:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-api-support:jar:1.6.5:test
[WARNING]javax.xml.bind:jaxb-api:jar:2.2.2:compile
[WARNING]commons-collections:commons-collections:jar:3.2.2:compile
[WARNING]
com.fasterxml.jackson.core:jackson-annotations:jar:2.7.4:compile
[WARNING]org.apache.hadoop:hadoop-common:jar:2.4.1:compile
[WARNING]org.apache.curator:curator-recipes:jar:2.12.0:compile
[WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.1:compile
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.apache.curator:curator-framework:jar:2.12.0:compile
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]com.data-artisans:flakka-slf4j_2.11:jar:2.3-custom:compile
[WARNING]org.reflections:reflections:jar:0.9.10:test
[WARNING]org.javassist:javassist:jar:3.18.2-GA:compile
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]com.twitter:chill_2.11:jar:0.7.4:compile
[WARNING]
org.apache.flink:flink-shaded-curator-recipes:jar:1.4-SNAPSHOT:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```
After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-runtime_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.curator:curator-client:jar:2.12.0:compile
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING]com.typesafe:config:jar:1.2.1:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-api-support:jar:1.6.5:test
[WARNING]javax.xml.bind:jaxb-api:jar:2.2.2:compile
[WARNING]org.apache.hadoop:hadoop-common:jar:2.4.1:compile
[WARNING]org.apache.curator:curator-recipes:jar:2.12.0:compile
[WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.1:compile
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.apache.curator:curator-framework:jar:2.12.0:compile
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.javassist:javassist:jar:3.18.2-GA:compile
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]com.twitter:chill_2.11:jar:0.7.4:compile
[WARNING]
org.apache.flink:flink-shaded-curator-recipes:jar:1.4-SNAPSHOT:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```
## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3828

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4713


commit c73e9eaea192bb51ecdc639a1dbfb78fce7ef95f
Author: yew1eb 
Date:   2017-09-24T16:45:12Z

 POM Cleanup flink-runtime




> flink-runtime
> -
>
> Key: FLINK-3828
> 

[GitHub] flink pull request #4713: [FLINK-3828][build] POM Cleanup flink-runtime

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4713

[FLINK-3828][build] POM Cleanup flink-runtime

## What is the purpose of the change

This PR changes the flink-runtime pom to

not contain unused dependencies
contain all used dependencies

## Brief change log

## Verifying this change
_mvn dependency:analyze_
```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-runtime_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]org.apache.curator:curator-client:jar:2.12.0:compile
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING]com.google.code.findbugs:annotations:jar:2.0.1:test
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]com.typesafe:config:jar:1.2.1:compile
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-api-support:jar:1.6.5:test
[WARNING]javax.xml.bind:jaxb-api:jar:2.2.2:compile
[WARNING]commons-collections:commons-collections:jar:3.2.2:compile
[WARNING]
com.fasterxml.jackson.core:jackson-annotations:jar:2.7.4:compile
[WARNING]org.apache.hadoop:hadoop-common:jar:2.4.1:compile
[WARNING]org.apache.curator:curator-recipes:jar:2.12.0:compile
[WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.1:compile
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.apache.curator:curator-framework:jar:2.12.0:compile
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.flink:flink-shaded-hadoop2:jar:1.4-SNAPSHOT:compile
[WARNING]com.data-artisans:flakka-slf4j_2.11:jar:2.3-custom:compile
[WARNING]org.reflections:reflections:jar:0.9.10:test
[WARNING]org.javassist:javassist:jar:3.18.2-GA:compile
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]com.twitter:chill_2.11:jar:0.7.4:compile
[WARNING]
org.apache.flink:flink-shaded-curator-recipes:jar:1.4-SNAPSHOT:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```
After the change:

```
[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-runtime_2.11 ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.curator:curator-client:jar:2.12.0:compile
[WARNING]io.netty:netty:jar:3.8.0.Final:compile
[WARNING]com.typesafe:config:jar:1.2.1:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.powermock:powermock-api-support:jar:1.6.5:test
[WARNING]javax.xml.bind:jaxb-api:jar:2.2.2:compile
[WARNING]org.apache.hadoop:hadoop-common:jar:2.4.1:compile
[WARNING]org.apache.curator:curator-recipes:jar:2.12.0:compile
[WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.1:compile
[WARNING]org.powermock:powermock-core:jar:1.6.5:test
[WARNING]org.apache.curator:curator-framework:jar:2.12.0:compile
[WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
[WARNING] Unused declared dependencies found:
[WARNING]org.javassist:javassist:jar:3.18.2-GA:compile
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]com.twitter:chill_2.11:jar:0.7.4:compile
[WARNING]
org.apache.flink:flink-shaded-curator-recipes:jar:1.4-SNAPSHOT:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```
## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-3828

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4713


commit c73e9eaea192bb51ecdc639a1dbfb78fce7ef95f
Author: yew1eb 
Date:   2017-09-24T16:45:12Z

 POM Cleanup flink-runtime




---

[jira] [Assigned] (FLINK-3834) flink-statebackend-rocksdb

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 reassigned FLINK-3834:
-

Assignee: Chesnay Schepler  (was: Hai Zhou_UTC+8)

> flink-statebackend-rocksdb
> --
>
> Key: FLINK-3834
> URL: https://issues.apache.org/jira/browse/FLINK-3834
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-3834) flink-statebackend-rocksdb

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 closed FLINK-3834.
-
Resolution: Fixed

> flink-statebackend-rocksdb
> --
>
> Key: FLINK-3834
> URL: https://issues.apache.org/jira/browse/FLINK-3834
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4712: fix flink-clients undeclared/unused dependencies

[yew1eb]

GitHub user yew1eb opened a pull request:

https://github.com/apache/flink/pull/4712

fix flink-clients undeclared/unused dependencies

## What is the purpose of the change
This PR changes the flink-clients pom to

- not contain unused dependencies
- contain all used dependencies

## Brief change log
## Verifying this change
_mvn dependency:analyze_

```
[INFO] — maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-clients_2.11 —
[WARNING] Used undeclared dependencies found:
[WARNING] org.scala-lang:scala-library:jar:2.11.11:compile
[WARNING] com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING] org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING] org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING] com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING] log4j:log4j:jar:1.2.17:test
[WARNING] org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.7:test
```
After the change:
```

[INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ 
flink-clients_2.11 ---
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes)

## Documentation
none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yew1eb/flink FLINK-7574

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4712.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4712


commit 0a3c8c1c90d094ba869c1168a2f575299638f5f7
Author: yew1eb 
Date:   2017-09-04T13:44:57Z

fix flink-clients undeclared/unused dependencies




---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140660299
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.Configuration;
+import freemarker.template.Template;
+import freemarker.template.TemplateException;
+import freemarker.template.TemplateExceptionHandler;
+import freemarker.template.Version;
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private HashMap constructorCache;
+   private final Template template;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.classCompiler = new SimpleCompiler();
+   
this.classCompiler.setParentClassLoader(this.getClass().getClassLoader());
--- End diff --

I think that the `SorterFactory` class is not instantiated by the user code 
classloader, so `this.getClass().getClassLoader()` gets the wrong classloader 
here.

Instead, we should get the user code classloader from the caller of 
`createSorter`.

I'll make this change, and also do the modifications to the cache, as 
discussed above.


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140660095
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -81,6 +81,7 @@
 */
private SorterFactory() {
this.classCompiler = new SimpleCompiler();
+   
this.classCompiler.setParentClassLoader(this.getClass().getClassLoader());
--- End diff --

I think that the `SorterFactory` class is not instantiated by the user code 
classloader, so `this.getClass().getClassLoader()` gets the wrong classloader 
here.

Instead, we should get the user code classloader from the caller of 
`createSorter`.

I'll make this change, and also do the modifications to the cache, as 
discussed above.


---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[ggevay]

Github user ggevay commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140659933
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
 ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.TemplateException;
+
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides 
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to 
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+   // 

+   //   Constants
+   // 

+   private static final Logger LOG = 
LoggerFactory.getLogger(SorterFactory.class);
+
+   /** Fixed length records with a length below this threshold will be 
in-place sorted, if possible. */
+   private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+   // 

+   //   Singleton Attribute
+   // 

+   private static SorterFactory sorterFactory;
+
+   // 

+   //   Attributes
+   // 

+   private SimpleCompiler classCompiler;
+   private TemplateManager templateManager;
+   private HashMap constructorCache;
+
+   /**
+* This is only for testing. If an error occurs, we want to fail the 
test, instead of falling back
+* to a non-generated sorter.
+*/
+   public boolean forceCodeGeneration = false;
+
+   /**
+* Constructor.
+*/
+   private SorterFactory() {
+   this.templateManager = TemplateManager.getInstance();
+   this.classCompiler = new SimpleCompiler();
--- End diff --

Thanks for raising this issue. Actually it's a bit more complicated than 
just setting the parent classloader, because of the caching of the generated 
classes. The problem is that the `constructorCache` would keep the old user 
code classloader alive. I'm currently thinking of the following solution:
1. include also the user code classloader in the key of the cache, and
2. use `WeakReference` for both the values and keys of the cache.

This will ensure that we don't try to reuse generated classes from a 
previous job, since the classloader will be different across jobs, so we will 
have different keys. And the cache won't keep anything alive, since it will 
only have `WeakReferences`.
(For 2., I will use the Guava `CacheBuilder`.)


---

[jira] [Comment Edited] (FLINK-6444) Add a check that '@VisibleForTesting' methods are only used in tests

[Hai Zhou_UTC+8 (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178254#comment-16178254
 ] 

Hai Zhou_UTC+8 edited comment on FLINK-6444 at 9/24/17 3:43 PM:


[~StephanEwen]  I implemented a maven module "flink-spotbugs-plugin" that 
contains a *VisibleForTestingDetector* class used to detect whether 
'@VisibleForTesting' methods was used illegally.

commit: 
https://github.com/yew1eb/flink/commit/d8edc6625ad98302cca54d80c5321d7fe3948b82


But,I am troubled. this module needs to be deployed to a remote maven 
repository, and It dep the “flink-annotation” module, the version of 
flink-annotation is not certain. 
Can you give me some suggestions? 
Or have no choice but to use the second plan like 
*CheckForbiddenMethodsUsage.java*.

BTW, via VisibleForTestingDetector detector, I did find that many 
'@VisibleForTesting' methods are called not in the test code.


was (Author: yew1eb):
[~StephanEwen]  I implemented a maven module "flink-spotbugs-plugin" that 
contains a *VisibleForTestingDetector* class used to detect whether 
'@VisibleForTesting' methods was used illegally.
commit: 
https://github.com/yew1eb/flink/commit/d8edc6625ad98302cca54d80c5321d7fe3948b82

But,I am troubled. this module needs to be deployed to a remote maven 
repository, and It dep the “flink-annotation” module, the version of 
flink-annotation is not certain. 
Can you give me some suggestions? 
Or have no choice but to use the second plan like 
*CheckForbiddenMethodsUsage.java*.

BTW, via VisibleForTestingDetector detector, I did find that many 
'@VisibleForTesting' methods are called not in the test code.

> Add a check that '@VisibleForTesting' methods are only used in tests
> 
>
> Key: FLINK-6444
> URL: https://issues.apache.org/jira/browse/FLINK-6444
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>
> Some methods are annotated with {{@VisibleForTesting}}. These methods should 
> only be called from tests.
> This is currently not enforced / checked during the build. We should add such 
> a check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLINK-6444) Add a check that '@VisibleForTesting' methods are only used in tests

[Hai Zhou_UTC+8 (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178254#comment-16178254
 ] 

Hai Zhou_UTC+8 edited comment on FLINK-6444 at 9/24/17 3:42 PM:


[~StephanEwen]  I implemented a maven module "flink-spotbugs-plugin" that 
contains a *VisibleForTestingDetector* class used to detect whether 
'@VisibleForTesting' methods was used illegally.
commit: 
https://github.com/yew1eb/flink/commit/d8edc6625ad98302cca54d80c5321d7fe3948b82

But,I am troubled. this module needs to be deployed to a remote maven 
repository, and It dep the “flink-annotation” module, the version of 
flink-annotation is not certain. 
Can you give me some suggestions? 
Or have no choice but to use the second plan like 
*CheckForbiddenMethodsUsage.java*.

BTW, via VisibleForTestingDetector detector, I did find that many 
'@VisibleForTesting' methods are called not in the test code.


was (Author: yew1eb):
[~StephanEwen]  I implemented a maven module "flink-spotbugs-plugin" that 
contains a *VisibleForTestingDetector* class used to detect whether 
'@VisibleForTesting' methods was used illegally.

But,I am troubled. this module needs to be deployed to a remote maven 
repository, and It dep the “flink-annotation” module, the version of 
flink-annotation is not certain. 
Can you give me some suggestions? 
Or have no choice but to use the second plan like 
*CheckForbiddenMethodsUsage.java*.

BTW, via VisibleForTestingDetector detector, I did find that many 
'@VisibleForTesting' methods are called not in the test code.

> Add a check that '@VisibleForTesting' methods are only used in tests
> 
>
> Key: FLINK-6444
> URL: https://issues.apache.org/jira/browse/FLINK-6444
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>
> Some methods are annotated with {{@VisibleForTesting}}. These methods should 
> only be called from tests.
> This is currently not enforced / checked during the build. We should add such 
> a check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6444) Add a check that '@VisibleForTesting' methods are only used in tests

[Hai Zhou_UTC+8 (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178254#comment-16178254
 ] 

Hai Zhou_UTC+8 commented on FLINK-6444:
---

[~StephanEwen]  I implemented a maven module "flink-spotbugs-plugin" that 
contains a *VisibleForTestingDetector* class used to detect whether 
'@VisibleForTesting' methods was used illegally.

But,I am troubled. this module needs to be deployed to a remote maven 
repository, and It dep the “flink-annotation” module, the version of 
flink-annotation is not certain. 
Can you give me some suggestions? 
Or have no choice but to use the second plan like 
*CheckForbiddenMethodsUsage.java*.

BTW, via VisibleForTestingDetector detector, I did find that many 
'@VisibleForTesting' methods are called not in the test code.

> Add a check that '@VisibleForTesting' methods are only used in tests
> 
>
> Key: FLINK-6444
> URL: https://issues.apache.org/jira/browse/FLINK-6444
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>
> Some methods are annotated with {{@VisibleForTesting}}. These methods should 
> only be called from tests.
> This is currently not enforced / checked during the build. We should add such 
> a check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

[heytitle]

Github user heytitle commented on a diff in the pull request:

https://github.com/apache/flink/pull/3511#discussion_r140659544
  
--- Diff: 
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java
 ---
@@ -190,6 +190,7 @@ private String getSystemOutput(String[] args) throws 
Exception {
switch (mode) {
case CLUSTER:
case COLLECTION:
+   case CLUSTER_WITH_CODEGENERATION_ENABLED:
--- End diff --

First of all, I'm not sure whether this is a good way to get this 
estimation. 

I estimated the build time by running all tests from `flink-gelly-examples` 
inside IntelliJ IDEA.

With `CLUSTER_WITH_CODEGENERATION_ENABLED` : `2m 20s`
Without `CLUSTER_WITH_CODEGENERATION_ENABLED`: `1m 27s`

Patch for disabling `CLUSTER_WITH_CODEGENERATION_ENABLED` case : 
https://gist.github.com/heytitle/89961fcaabcf326eadee190b9d6085a6



---

[jira] [Closed] (FLINK-7677) Add side outputs to ProcessWindowFunction

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi closed FLINK-7677.
--

> Add side outputs to ProcessWindowFunction
> -
>
> Key: FLINK-7677
> URL: https://issues.apache.org/jira/browse/FLINK-7677
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.4.0
>Reporter: Ufuk Celebi
>
> As per discussion on the user mailing list [1], we should add the required 
> context to collect to side output in ProcessWindowFunctions similar to 
> ProcessFunction.
> [1] 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/question-on-sideoutput-from-ProcessWindow-function-td15500.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6233) Support rowtime inner equi-join between two streams in the SQL API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178209#comment-16178209
 ] 

ASF GitHub Bot commented on FLINK-6233:
---

Github user xccui commented on the issue:

https://github.com/apache/flink/pull/4625
  
Hi @fhueske, the PR has been updated. Temporarily, I keep the logic for 
dealing with the late data, as well as the fine-grained cache. 

For the late data semantics problem, I think we need to rethink it and make 
a final decision (maybe we should consult others). For the cache optimization 
problem, I want to leave it a future work. 


> Support rowtime inner equi-join between two streams in the SQL API
> --
>
> Key: FLINK-6233
> URL: https://issues.apache.org/jira/browse/FLINK-6233
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: hongyuhong
>Assignee: Xingcan Cui
>
> The goal of this issue is to add support for inner equi-join on proc time 
> streams to the SQL interface.
> Queries similar to the following should be supported:
> {code}
> SELECT o.rowtime , o.productId, o.orderId, s.rowtime AS shipTime 
> FROM Orders AS o 
> JOIN Shipments AS s 
> ON o.orderId = s.orderId 
> AND o.rowtime BETWEEN s.rowtime AND s.rowtime + INTERVAL '1' HOUR;
> {code}
> The following restrictions should initially apply:
> * The join hint only support inner join
> * The ON clause should include equi-join condition
> * The time-condition {{o.rowtime BETWEEN s.rowtime AND s.rowtime + INTERVAL 
> '1' HOUR}} only can use rowtime that is a system attribute, the time 
> condition only support bounded time range like {{o.rowtime BETWEEN s.rowtime 
> - INTERVAL '1' HOUR AND s.rowtime + INTERVAL '1' HOUR}}, not support 
> unbounded like {{o.rowtime  s.rowtime}} ,  and  should include both two 
> stream's rowtime attribute, {{o.rowtime between rowtime () and rowtime () + 
> 1}} should also not be supported.
> An row-time streams join will not be able to handle late data, because this 
> would mean in insert a row into a sorted order shift all other computations. 
> This would be too expensive to maintain. Therefore, we will throw an error if 
> a user tries to use an row-time stream join with late data handling.
> This issue includes:
> * Design of the DataStream operator to deal with stream join
> * Translation from Calcite's RelNode representation (LogicalJoin). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4625: [FLINK-6233] [table] Support time-bounded stream inner jo...

[xccui]

Github user xccui commented on the issue:

https://github.com/apache/flink/pull/4625
  
Hi @fhueske, the PR has been updated. Temporarily, I keep the logic for 
dealing with the late data, as well as the fine-grained cache. 

For the late data semantics problem, I think we need to rethink it and make 
a final decision (maybe we should consult others). For the cache optimization 
problem, I want to leave it a future work. 


---

[jira] [Assigned] (FLINK-3834) flink-statebackend-rocksdb

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 reassigned FLINK-3834:
-

Assignee: Hai Zhou_UTC+8

> flink-statebackend-rocksdb
> --
>
> Key: FLINK-3834
> URL: https://issues.apache.org/jira/browse/FLINK-3834
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-3833) flink-test-utils

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 reassigned FLINK-3833:
-

Assignee: Hai Zhou_UTC+8

> flink-test-utils
> 
>
> Key: FLINK-3833
> URL: https://issues.apache.org/jira/browse/FLINK-3833
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-3830) flink-scala

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 reassigned FLINK-3830:
-

Assignee: Hai Zhou_UTC+8

> flink-scala
> ---
>
> Key: FLINK-3830
> URL: https://issues.apache.org/jira/browse/FLINK-3830
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-3832) flink-streaming-scala

[Hai Zhou_UTC+8 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou_UTC+8 reassigned FLINK-3832:
-

Assignee: Hai Zhou_UTC+8

> flink-streaming-scala
> -
>
> Key: FLINK-3832
> URL: https://issues.apache.org/jira/browse/FLINK-3832
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>Assignee: Hai Zhou_UTC+8
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178152#comment-16178152
 ] 

mingleizhang commented on FLINK-3829:
-

Hey, [~till.rohrmann] One thing that I should mention it, most unused declared 
dependencies are inherited from the parent pom. And can not exclude them like a 
normal way. Do you have any good idea ? Thanks in advance.

> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5944) Flink should support reading Snappy Files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178140#comment-16178140
 ] 

ASF GitHub Bot commented on FLINK-5944:
---

Github user mlipkovich commented on a diff in the pull request:

https://github.com/apache/flink/pull/4683#discussion_r140652438
  
--- Diff: flink-core/pom.xml ---
@@ -52,6 +52,12 @@ under the License.
flink-shaded-asm

 
+   
+   org.apache.flink
+   flink-shaded-hadoop2
+   ${project.version}
+   
--- End diff --

What do you think about adding this dependency to compile-time only?

Regarding to difference between codecs as I understand the thing is that 
Snappy compressed files are not splittable. So Hadoop splits raw files into 
blocks and compresses each block separately using regular Snappy. If you 
download the whole Hadoop Snappy compressed file regular Snappy will not be 
able to decompress it since it's not aware of block boundaries


> Flink should support reading Snappy Files
> -
>
> Key: FLINK-5944
> URL: https://issues.apache.org/jira/browse/FLINK-5944
> Project: Flink
>  Issue Type: New Feature
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ilya Ganelin
>Assignee: Mikhail Lipkovich
>  Labels: features
>
> Snappy is an extremely performant compression format that's widely used 
> offering fast decompression/compression. 
> This can be easily implemented by creating a SnappyInflaterInputStreamFactory 
> and updating the initDefaultInflateInputStreamFactories in FileInputFormat.
> Flink already includes the Snappy dependency in the project. 
> There is a minor gotcha in this. If we wish to use this with Hadoop, then we 
> must provide two separate implementations since Hadoop uses a different 
> version of the snappy format than Snappy Java (which is the xerial/snappy 
> included in Flink). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4683: [FLINK-5944] Support reading of Snappy files

[mlipkovich]

Github user mlipkovich commented on a diff in the pull request:

https://github.com/apache/flink/pull/4683#discussion_r140652438
  
--- Diff: flink-core/pom.xml ---
@@ -52,6 +52,12 @@ under the License.
flink-shaded-asm

 
+   
+   org.apache.flink
+   flink-shaded-hadoop2
+   ${project.version}
+   
--- End diff --

What do you think about adding this dependency to compile-time only?

Regarding to difference between codecs as I understand the thing is that 
Snappy compressed files are not splittable. So Hadoop splits raw files into 
blocks and compresses each block separately using regular Snappy. If you 
download the whole Hadoop Snappy compressed file regular Snappy will not be 
able to decompress it since it's not aware of block boundaries


---

[jira] [Updated] (FLINK-3829) flink-java

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang updated FLINK-3829:

Description: 

{noformat}
[WARNING] Used undeclared dependencies found:
[WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[WARNING]commons-cli:commons-cli:jar:1.3.1:compile
[WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
[WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
[WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
[WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]log4j:log4j:jar:1.2.17:test
[WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
[WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
{noformat}


> flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Till Rohrmann
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-3986) Rename readFileStream from the StreamExecutionEnvironment

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang updated FLINK-3986:

Description: The readFileStream(String filePath, long intervalMillis, 
WatchType watchType) has to be renamed to readFile match the naming conventions 
of the rest of the methods, or even removed.  (was: The readFileStream(String 
filePath, long intervalMillis, WatchType watchType) has to be renamed to to 
readFile match the naming conventions of the rest of the methods, or even 
removed.)

> Rename readFileStream from the StreamExecutionEnvironment
> -
>
> Key: FLINK-3986
> URL: https://issues.apache.org/jira/browse/FLINK-3986
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Kostas Kloudas
> Fix For: 2.0.0
>
>
> The readFileStream(String filePath, long intervalMillis, WatchType watchType) 
> has to be renamed to readFile match the naming conventions of the rest of the 
> methods, or even removed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-3986) Rename readFileStream from the StreamExecutionEnvironment

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178126#comment-16178126
 ] 

mingleizhang commented on FLINK-3986:
-

I would like to work on this. Just not sure which way we choose to implement 
it. remove or rename ?

> Rename readFileStream from the StreamExecutionEnvironment
> -
>
> Key: FLINK-3986
> URL: https://issues.apache.org/jira/browse/FLINK-3986
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Kostas Kloudas
> Fix For: 2.0.0
>
>
> The readFileStream(String filePath, long intervalMillis, WatchType watchType) 
> has to be renamed to to readFile match the naming conventions of the rest of 
> the methods, or even removed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5944) Flink should support reading Snappy Files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178113#comment-16178113
 ] 

ASF GitHub Bot commented on FLINK-5944:
---

Github user haohui commented on a diff in the pull request:

https://github.com/apache/flink/pull/4683#discussion_r140649772
  
--- Diff: flink-core/pom.xml ---
@@ -52,6 +52,12 @@ under the License.
flink-shaded-asm

 
+   
+   org.apache.flink
+   flink-shaded-hadoop2
+   ${project.version}
+   
--- End diff --

Including hadoop as a dependency in flink-core can be problematic for a 
number of downstream projects.

I wonder what is the exact difference between the Hadoop and vanilla snappy 
codec? Is it just due to the fact that there are additional framings in the 
snappy codec in Hadoop?





> Flink should support reading Snappy Files
> -
>
> Key: FLINK-5944
> URL: https://issues.apache.org/jira/browse/FLINK-5944
> Project: Flink
>  Issue Type: New Feature
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ilya Ganelin
>Assignee: Mikhail Lipkovich
>  Labels: features
>
> Snappy is an extremely performant compression format that's widely used 
> offering fast decompression/compression. 
> This can be easily implemented by creating a SnappyInflaterInputStreamFactory 
> and updating the initDefaultInflateInputStreamFactories in FileInputFormat.
> Flink already includes the Snappy dependency in the project. 
> There is a minor gotcha in this. If we wish to use this with Hadoop, then we 
> must provide two separate implementations since Hadoop uses a different 
> version of the snappy format than Snappy Java (which is the xerial/snappy 
> included in Flink). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4683: [FLINK-5944] Support reading of Snappy files

[haohui]

Github user haohui commented on a diff in the pull request:

https://github.com/apache/flink/pull/4683#discussion_r140649772
  
--- Diff: flink-core/pom.xml ---
@@ -52,6 +52,12 @@ under the License.
flink-shaded-asm

 
+   
+   org.apache.flink
+   flink-shaded-hadoop2
+   ${project.version}
+   
--- End diff --

Including hadoop as a dependency in flink-core can be problematic for a 
number of downstream projects.

I wonder what is the exact difference between the Hadoop and vanilla snappy 
codec? Is it just due to the fact that there are additional framings in the 
snappy codec in Hadoop?





---

[jira] [Commented] (FLINK-7446) Support to define an existing field as the rowtime field for TableSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178107#comment-16178107
 ] 

ASF GitHub Bot commented on FLINK-7446:
---

Github user haohui commented on the issue:

https://github.com/apache/flink/pull/4710
  
LGTM overall +1.

One question: since we now cast `ROWTIME` / `PROCTIME` directly to `LONG`, 
I wonder, do we want to revisit the decision that creates dedicated types for 
`ROWTIME` / `PROCTIME`?


> Support to define an existing field as the rowtime field for TableSource
> 
>
> Key: FLINK-7446
> URL: https://issues.apache.org/jira/browse/FLINK-7446
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Jark Wu
>Assignee: Fabian Hueske
>
> Currently {{DefinedRowtimeAttribute}} can define an appended rowtime field 
> for a {{TableSource}}. But it would be helpful if we can support to define an 
> existing field as the rowtime field. Just like registering a DataStream, the 
> rowtime field can be appended but also can replace an existing field.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4710: [FLINK-7446] [table] Change DefinedRowtimeAttribute to wo...

[haohui]

Github user haohui commented on the issue:

https://github.com/apache/flink/pull/4710
  
LGTM overall +1.

One question: since we now cast `ROWTIME` / `PROCTIME` directly to `LONG`, 
I wonder, do we want to revisit the decision that creates dedicated types for 
`ROWTIME` / `PROCTIME`?


---