[jira] [Commented] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem

[Paul Harris (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168480#comment-15168480
 ] 

Paul Harris commented on FLINK-3501:


This seems to be an OSX issue. I have tried with Oracle JDK7 u75, JDK7 u79 and 
JDK8 u84.

There is more information here 
http://stackoverflow.com/questions/10522835/hadoop-java-io-ioexception-mkdirs-failed-to-create-some-path

it relates to the jar containing a file META-INF/LICENSE and also a directory 
called META-INF/license.

If I edit the archive with
{code:none}
zip -d flink-runtime-0.10.2-hadoop1.jar META_INF/license
{code}

then unjar with 
{code:none}
jar xf flink-runtime-0.10.2-hadoop1.jar
{code}

everything works ok, but this means editing my gradle cache which is a bit of a 
pain.

> flink-runtime-0.10.2-hadoop-1 jar problem
> -
>
> Key: FLINK-3501
> URL: https://issues.apache.org/jira/browse/FLINK-3501
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.10.2
>Reporter: Paul Harris
>
> Unjaring flink-runtime-0.10.2-hadoop-1 results in an error.
> Steps to reproduce
> {code:none}
> jar xf flink-runtime-0.10.2-hadoop1.jar
> {code}
> Result...
> {code:none}
> java.io.IOException: META-INF/license : could not create directory
>   at sun.tools.jar.Main.extractFile(Main.java:934)
>   at sun.tools.jar.Main.extract(Main.java:877)
>   at sun.tools.jar.Main.run(Main.java:263)
>   at sun.tools.jar.Main.main(Main.java:1177)
> {code}
> This is the result of the jar containing a file 'license' and a directory 
> license'. The file is unpacked first, preventing the directory being unpacked 
> to the same name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3461) Remove duplicate condition check in ZooKeeperLeaderElectionService

[sahitya pavurala (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168434#comment-15168434
 ] 

sahitya pavurala commented on FLINK-3461:
-

I submitted a pull request for this, can you please review it. Thanks.

> Remove duplicate condition check in ZooKeeperLeaderElectionService
> --
>
> Key: FLINK-3461
> URL: https://issues.apache.org/jira/browse/FLINK-3461
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ufuk Celebi
>Priority: Trivial
>
> In {{ZooKeeperLeaderElectionService}}:
> {code}
> @Override
> public boolean hasLeadership() {
>   if(leaderLatch.getState().equals(LeaderLatch.State.STARTED)) {
> return leaderLatch.hasLeadership();
>   } else {
> return false;
>   }
> }
> {code}
> Looking into the leader {{leaderLatch.hasLeadership()}}, the state gets 
> checked already/again.
> {code}
> return (state.get() == State.STARTED) && hasLeadership.get();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem

[Paul Harris (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168206#comment-15168206
 ] 

Paul Harris edited comment on FLINK-3501 at 2/26/16 1:20 AM:
-

I am on a mac with 
{code:none}
java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
{code}

When I download the jar from Apache I can untar fine.
When I pull via gradle from mvnrepository I have the issue. I am bringing in 
the following dependencies
{code:none}
'org.apache.flink:flink-clients:0.10.2-hadoop1'
'org.apache.flink:flink-scala:0.10.2-hadoop1'
{code}

The file in the gradle cache is the one with the issue (I have deleted the 
cached files and allowed gradle to pull again) .

This sounds more like a gradle-ism than a jar'ing problem. I think we can mark 
this as resolved. I'll re-open if I continue to have issues.


was (Author: trickbooter):
I am on a mac with 
{code:none}
java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
{code}

When I download the jar from Apache I can untar fine.
When I pull via gradle from mvnrepository I have the issue. I am bringing in 
the following dependencies
{code:none}
'org.apache.flink:flink-clients:0.10.2-hadoop1'
'org.apache.flink:flink-scala:0.10.2-hadoop1'
{code}

The file in the gradle cache is the one with the issue (I have deleted the 
cached files and allowed gradle to pull again) .

I don't know much mvn packaging I am afraid, so I don't know if the folllowing 
is relevent, but...
{code:none}
cd /path/to/gradle/cache
md5 flink-runtime-0.10.2-hadoop1.jar
MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2

cd /path/to/mvn/direct/download
MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 
54a4c422a57baecb48d609e7187448d8
{code}

This sounds more like a gradle-ism than a jar'ing problem. I think we can mark 
this as resolved. I'll re-open if I continue to have issues.

> flink-runtime-0.10.2-hadoop-1 jar problem
> -
>
> Key: FLINK-3501
> URL: https://issues.apache.org/jira/browse/FLINK-3501
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.10.2
>Reporter: Paul Harris
>
> Unjaring flink-runtime-0.10.2-hadoop-1 results in an error.
> Steps to reproduce
> {code:none}
> jar xf flink-runtime-0.10.2-hadoop1.jar
> {code}
> Result...
> {code:none}
> java.io.IOException: META-INF/license : could not create directory
>   at sun.tools.jar.Main.extractFile(Main.java:934)
>   at sun.tools.jar.Main.extract(Main.java:877)
>   at sun.tools.jar.Main.run(Main.java:263)
>   at sun.tools.jar.Main.main(Main.java:1177)
> {code}
> This is the result of the jar containing a file 'license' and a directory 
> license'. The file is unpacked first, preventing the directory being unpacked 
> to the same name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem

[Paul Harris (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168206#comment-15168206
 ] 

Paul Harris edited comment on FLINK-3501 at 2/26/16 12:57 AM:
--

I am on a mac with 
{code:none}
java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
{code}

When I download the jar from Apache I can untar fine.
When I pull via gradle from mvnrepository I have the issue. I am bringing in 
the following dependencies
{code:none}
'org.apache.flink:flink-clients:0.10.2-hadoop1'
'org.apache.flink:flink-scala:0.10.2-hadoop1'
{code}

The file in the gradle cache is the one with the issue (I have deleted the 
cached files and allowed gradle to pull again) .

I don't know much mvn packaging I am afraid, so I don't know if the folllowing 
is relevent, but...
{code:none}
cd /path/to/gradle/cache
md5 flink-runtime-0.10.2-hadoop1.jar
MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2

cd /path/to/mvn/direct/download
MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 
54a4c422a57baecb48d609e7187448d8
{code}

This sounds more like a gradle-ism than a jar'ing problem. I think we can mark 
this as resolved. I'll re-open if I continue to have issues.


was (Author: trickbooter):
I am on a mac with 
```
java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
```

When I download the jar from Apache I can untar fine.
When I pull via gradle from mvnrepository I have the issue. I am bringing in 
the following dependencies
```
'org.apache.flink:flink-clients:0.10.2-hadoop1'
'org.apache.flink:flink-scala:0.10.2-hadoop1'
```

The file in the gradle cache is the one with the issue (I have deleted the 
cached files and allowed gradle to pull again) .

I don't know much mvn packaging I am afraid, so I don't know if the folllowing 
is relevent, but...
```
cd /path/to/gradle/cache
md5 flink-runtime-0.10.2-hadoop1.jar
MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2

cd /path/to/mvn/direct/download
MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 
54a4c422a57baecb48d609e7187448d8
```

This sounds more like a gradle-ism than a jar'ing problem. I think we can mark 
this as resolved. I'll re-open if I continue to have issues.

> flink-runtime-0.10.2-hadoop-1 jar problem
> -
>
> Key: FLINK-3501
> URL: https://issues.apache.org/jira/browse/FLINK-3501
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.10.2
>Reporter: Paul Harris
>
> Unjaring flink-runtime-0.10.2-hadoop-1 results in an error.
> Steps to reproduce
> {code:none}
> jar xf flink-runtime-0.10.2-hadoop1.jar
> {code}
> Result...
> {code:none}
> java.io.IOException: META-INF/license : could not create directory
>   at sun.tools.jar.Main.extractFile(Main.java:934)
>   at sun.tools.jar.Main.extract(Main.java:877)
>   at sun.tools.jar.Main.run(Main.java:263)
>   at sun.tools.jar.Main.main(Main.java:1177)
> {code}
> This is the result of the jar containing a file 'license' and a directory 
> license'. The file is unpacked first, preventing the directory being unpacked 
> to the same name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem

[Paul Harris (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168206#comment-15168206
 ] 

Paul Harris commented on FLINK-3501:


I am on a mac with 
```
java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
```

When I download the jar from Apache I can untar fine.
When I pull via gradle from mvnrepository I have the issue. I am bringing in 
the following dependencies
```
'org.apache.flink:flink-clients:0.10.2-hadoop1'
'org.apache.flink:flink-scala:0.10.2-hadoop1'
```

The file in the gradle cache is the one with the issue (I have deleted the 
cached files and allowed gradle to pull again) .

I don't know much mvn packaging I am afraid, so I don't know if the folllowing 
is relevent, but...
```
cd /path/to/gradle/cache
md5 flink-runtime-0.10.2-hadoop1.jar
MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2

cd /path/to/mvn/direct/download
MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 
54a4c422a57baecb48d609e7187448d8
```

This sounds more like a gradle-ism than a jar'ing problem. I think we can mark 
this as resolved. I'll re-open if I continue to have issues.

> flink-runtime-0.10.2-hadoop-1 jar problem
> -
>
> Key: FLINK-3501
> URL: https://issues.apache.org/jira/browse/FLINK-3501
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.10.2
>Reporter: Paul Harris
>
> Unjaring flink-runtime-0.10.2-hadoop-1 results in an error.
> Steps to reproduce
> {code:none}
> jar xf flink-runtime-0.10.2-hadoop1.jar
> {code}
> Result...
> {code:none}
> java.io.IOException: META-INF/license : could not create directory
>   at sun.tools.jar.Main.extractFile(Main.java:934)
>   at sun.tools.jar.Main.extract(Main.java:877)
>   at sun.tools.jar.Main.run(Main.java:263)
>   at sun.tools.jar.Main.main(Main.java:1177)
> {code}
> This is the result of the jar containing a file 'license' and a directory 
> license'. The file is unpacked first, preventing the directory being unpacked 
> to the same name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3485) The SerializedListAccumulator value doest seem to be right

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168142#comment-15168142
 ] 

ASF GitHub Bot commented on FLINK-3485:
---

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/1717

[FLINK-3485] implement toString for SerializedListAccumulator



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-3485

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1717


commit 420b7d164168f8557e82f836ed809f3c81e1983a
Author: Maximilian Michels 
Date:   2016-02-25T23:49:28Z

[FLINK-3485] implement toString for SerializedListAccumulator




> The SerializedListAccumulator value doest seem to be right
> --
>
> Key: FLINK-3485
> URL: https://issues.apache.org/jira/browse/FLINK-3485
> Project: Flink
>  Issue Type: Bug
>  Components: Web Client
>Reporter: sahitya pavurala
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> When run flink examples, the accumulator value keeps changing, as it shows 
> the hashcode value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3485] implement toString for Serialized...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/1717

[FLINK-3485] implement toString for SerializedListAccumulator



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-3485

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1717


commit 420b7d164168f8557e82f836ed809f3c81e1983a
Author: Maximilian Michels 
Date:   2016-02-25T23:49:28Z

[FLINK-3485] implement toString for SerializedListAccumulator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3485) The SerializedListAccumulator value doest seem to be right

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168139#comment-15168139
 ] 

Maximilian Michels commented on FLINK-3485:
---

You're right. The accumulator is printed using the {{toString()}} method. What 
you see is the default Object.toString() method of the 
{{SerializedListAccumuator}}. I suppose we could change this to something nicer 
like printing the number of items. The accumulator is used for aggregating the 
results when you use {{collect()}}.

> The SerializedListAccumulator value doest seem to be right
> --
>
> Key: FLINK-3485
> URL: https://issues.apache.org/jira/browse/FLINK-3485
> Project: Flink
>  Issue Type: Bug
>  Components: Web Client
>Reporter: sahitya pavurala
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> When run flink examples, the accumulator value keeps changing, as it shows 
> the hashcode value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3510) Pattern class class-level comment misses type argument

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi closed FLINK-3510.
--
Resolution: Fixed

Fixed in 930a266.

> Pattern class class-level comment misses type argument
> --
>
> Key: FLINK-3510
> URL: https://issues.apache.org/jira/browse/FLINK-3510
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Trivial
>
> Class-level comment code example reads:
> {code}
>  * {@code
>  * Pattern pattern = Pattern.begin("start")
>  *   .next("middle").subtype(F.class)
>  *   .followedBy("end").where(new MyFilterFunction());
>  * }
>  * 
> {code}
> The 2nd type paramter for {{Pattern}} is missing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3517) Number of job and task managers not checked in scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168098#comment-15168098
 ] 

ASF GitHub Bot commented on FLINK-3517:
---

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1716

[FLINK-3517] [dist] Only count active PIDs in start script

```bash
$ bin/start-cluster.sh
Starting cluster.
Starting jobmanager daemon on host pablo.
Starting taskmanager daemon on host pablo.
$ bin/taskmanager.sh start
[INFO] 1 instance(s) of taskmanager are already running on pablo.
Starting taskmanager daemon on host pablo.
$ bin/taskmanager.sh start
[INFO] 2 instance(s) of taskmanager are already running on pablo.
Starting taskmanager daemon on host pablo.
$ bin/taskmanager.sh start
[INFO] 3 instance(s) of taskmanager are already running on pablo.
Starting taskmanager daemon on host pablo.
$ jps
27328 TaskManager
27140 TaskManager
26949 TaskManager
26523 JobManager
26716 TaskManager
$ kill -9 27140
$ bin/taskmanager.sh start
>>> [INFO] 3 instance(s) of taskmanager are already running on pablo <<< 
Correct now
Starting taskmanager daemon on host pablo.
$ bin/stop-cluster.sh
Stopping taskmanager daemon (pid: 27545) on host pablo.
Stopping jobmanager daemon (pid: 26523) on host pablo.
$ bin/taskmanager.sh stop
Stopping taskmanager daemon (pid: 27328) on host pablo.
$ bin/taskmanager.sh stop
No taskmanager daemon (pid: 27140) is running anymore on pablo.
$ bin/taskmanager.sh stop
Stopping taskmanager daemon (pid: 26949) on host pablo.
$ bin/taskmanager.sh stop
Stopping taskmanager daemon (pid: 26716) on host pablo.
$ bin/taskmanager.sh stop
No taskmanager daemon to stop on host pablo.
```

We can further improve the stop part by repeatedly the PIDs in the pid file 
if a value is not matching an active PID.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink 3517-scripts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1716


commit e037c89404704b8f8bd02911e65dc1dd24b1e836
Author: Ufuk Celebi 
Date:   2016-02-25T23:11:48Z

[FLINK-3517] [dist] Only count active PIDs in start script




> Number of job and task managers not checked in scripts
> --
>
> Key: FLINK-3517
> URL: https://issues.apache.org/jira/browse/FLINK-3517
> Project: Flink
>  Issue Type: Test
>  Components: Start-Stop Scripts
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Minor
>
> The start up scripts determine whether a job or task manager is running via a 
> pids file. If a process, which is part of the pid file, is destroyed (for 
> example on failure) outside of the scripts, a warning for multiple job 
> managers are printed even though they are not running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3517] [dist] Only count active PIDs in ...

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1716

[FLINK-3517] [dist] Only count active PIDs in start script

```bash
$ bin/start-cluster.sh
Starting cluster.
Starting jobmanager daemon on host pablo.
Starting taskmanager daemon on host pablo.
$ bin/taskmanager.sh start
[INFO] 1 instance(s) of taskmanager are already running on pablo.
Starting taskmanager daemon on host pablo.
$ bin/taskmanager.sh start
[INFO] 2 instance(s) of taskmanager are already running on pablo.
Starting taskmanager daemon on host pablo.
$ bin/taskmanager.sh start
[INFO] 3 instance(s) of taskmanager are already running on pablo.
Starting taskmanager daemon on host pablo.
$ jps
27328 TaskManager
27140 TaskManager
26949 TaskManager
26523 JobManager
26716 TaskManager
$ kill -9 27140
$ bin/taskmanager.sh start
>>> [INFO] 3 instance(s) of taskmanager are already running on pablo <<< 
Correct now
Starting taskmanager daemon on host pablo.
$ bin/stop-cluster.sh
Stopping taskmanager daemon (pid: 27545) on host pablo.
Stopping jobmanager daemon (pid: 26523) on host pablo.
$ bin/taskmanager.sh stop
Stopping taskmanager daemon (pid: 27328) on host pablo.
$ bin/taskmanager.sh stop
No taskmanager daemon (pid: 27140) is running anymore on pablo.
$ bin/taskmanager.sh stop
Stopping taskmanager daemon (pid: 26949) on host pablo.
$ bin/taskmanager.sh stop
Stopping taskmanager daemon (pid: 26716) on host pablo.
$ bin/taskmanager.sh stop
No taskmanager daemon to stop on host pablo.
```

We can further improve the stop part by repeatedly the PIDs in the pid file 
if a value is not matching an active PID.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink 3517-scripts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1716


commit e037c89404704b8f8bd02911e65dc1dd24b1e836
Author: Ufuk Celebi 
Date:   2016-02-25T23:11:48Z

[FLINK-3517] [dist] Only count active PIDs in start script




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167956#comment-15167956
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-189004909
  
Pushed. Thanks for the explanation.


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...

[greghogan]

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-189004909
  
Pushed. Thanks for the explanation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167871#comment-15167871
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188985984
  
If you can quickly do it, yes.

Otherwise whoever merges this tomorrow, please remove the sysout statements.
We try to keep the maven output a bit cleaner that way. If you want to 
output something into the logs for the tests, just use a regular slf4j logger...


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188985984
  
If you can quickly do it, yes.

Otherwise whoever merges this tomorrow, please remove the sysout statements.
We try to keep the maven output a bit cleaner that way. If you want to 
output something into the logs for the tests, just use a regular slf4j logger...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167864#comment-15167864
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188982973
  
@StephanEwen just to clarify, since time is of the essence, would you like 
me to update the pull request without the println's?


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...

[greghogan]

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188982973
  
@StephanEwen just to clarify, since time is of the essence, would you like 
me to update the pull request without the println's?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3482) Implement Union Support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167755#comment-15167755
 ] 

ASF GitHub Bot commented on FLINK-3482:
---

GitHub user vasia opened a pull request:

https://github.com/apache/flink/pull/1715

[FLINK-3482] implement union translation

This PR add union support to Table on Calcite. I've implemented a custom 
`JoinUnionTransposeRule`
because Calcite's only matches with LogicalUnion.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vasia/flink union

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1715


commit 6991e601ceabc63ec9f367c4ad2e4f75468aaf2b
Author: vasia 
Date:   2016-02-25T19:52:19Z

[FLINK-3482] implement union translation
- implement custom JoinUnionTransposeRules
because Calcite's only match with LogicalUnion




> Implement Union Support
> ---
>
> Key: FLINK-3482
> URL: https://issues.apache.org/jira/browse/FLINK-3482
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3482] implement union translation

[vasia]

GitHub user vasia opened a pull request:

https://github.com/apache/flink/pull/1715

[FLINK-3482] implement union translation

This PR add union support to Table on Calcite. I've implemented a custom 
`JoinUnionTransposeRule`
because Calcite's only matches with LogicalUnion.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vasia/flink union

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1715


commit 6991e601ceabc63ec9f367c4ad2e4f75468aaf2b
Author: vasia 
Date:   2016-02-25T19:52:19Z

[FLINK-3482] implement union translation
- implement custom JoinUnionTransposeRules
because Calcite's only match with LogicalUnion




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3226] Improvements for expected types

[twalthr]

Github user twalthr closed the pull request at:

https://github.com/apache/flink/pull/1709


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167678#comment-15167678
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user twalthr closed the pull request at:

https://github.com/apache/flink/pull/1709


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167662#comment-15167662
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1709#issuecomment-188933506
  
You can close this now @twalthr. I've merged it.


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3463) Implement Calc Support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167661#comment-15167661
 ] 

ASF GitHub Bot commented on FLINK-3463:
---

Github user vasia closed the pull request at:

https://github.com/apache/flink/pull/1696


> Implement Calc Support
> --
>
> Key: FLINK-3463
> URL: https://issues.apache.org/jira/browse/FLINK-3463
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Improvements for expected types

[vasia]

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1709#issuecomment-188933506
  
You can close this now @twalthr. I've merged it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167659#comment-15167659
 ] 

ASF GitHub Bot commented on FLINK-3513:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1712


> Fix interplay of automatic Operator UID and Changing name of WindowOperator
> ---
>
> Key: FLINK-3513
> URL: https://issues.apache.org/jira/browse/FLINK-3513
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
> Fix For: 1.0.0
>
>
> WindowOperator can have a changing name because it has the TypeSerializer 
> .toString() output in it's name. For some type serializers that don't 
> implement toString() this means that the name changes.
> This means that savepoint restore does not work for the automatically 
> generated UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1712


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3463] implement calc translation

[vasia]

Github user vasia closed the pull request at:

https://github.com/apache/flink/pull/1696


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-3512) Savepoint backend should not revert to "jobmanager"

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi closed FLINK-3512.
--
   Resolution: Fixed
Fix Version/s: 1.0.0

Fixed in c9cba27.

> Savepoint backend should not revert to "jobmanager"
> ---
>
> Key: FLINK-3512
> URL: https://issues.apache.org/jira/browse/FLINK-3512
> Project: Flink
>  Issue Type: Bug
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
> Fix For: 1.0.0
>
>
> When not setting a state backend in the config the savepoint coordinator 
> defaults to use "jobmanager" as savepoint backend, ignoring the setting for 
> the savepoint backend in the config.
> This means that when setting a StateBackend only per job the system always 
> falls back to "jobmanager" even though it could use "filesystem" to make 
> savepoints persistent.
> See SavepointStoreFactory.java:71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi closed FLINK-3513.
--
   Resolution: Fixed
Fix Version/s: 1.0.0

Fixed in d90672f.

> Fix interplay of automatic Operator UID and Changing name of WindowOperator
> ---
>
> Key: FLINK-3513
> URL: https://issues.apache.org/jira/browse/FLINK-3513
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
> Fix For: 1.0.0
>
>
> WindowOperator can have a changing name because it has the TypeSerializer 
> .toString() output in it's name. For some type serializers that don't 
> implement toString() this means that the name changes.
> This means that savepoint restore does not work for the automatically 
> generated UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167654#comment-15167654
 ] 

ASF GitHub Bot commented on FLINK-3513:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1712#issuecomment-188932532
  
I'm merging this to master.


> Fix interplay of automatic Operator UID and Changing name of WindowOperator
> ---
>
> Key: FLINK-3513
> URL: https://issues.apache.org/jira/browse/FLINK-3513
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
>
> WindowOperator can have a changing name because it has the TypeSerializer 
> .toString() output in it's name. For some type serializers that don't 
> implement toString() this means that the name changes.
> This means that savepoint restore does not work for the automatically 
> generated UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1712#issuecomment-188932532
  
I'm merging this to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3517) Number of job and task managers not checked in scripts

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-3517:
--

 Summary: Number of job and task managers not checked in scripts
 Key: FLINK-3517
 URL: https://issues.apache.org/jira/browse/FLINK-3517
 Project: Flink
  Issue Type: Test
  Components: Start-Stop Scripts
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi
Priority: Minor


The start up scripts determine whether a job or task manager is running via a 
pids file. If a process, which is part of the pid file, is destroyed (for 
example on failure) outside of the scripts, a warning for multiple job managers 
are printed even though they are not running.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167641#comment-15167641
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188929386
  
This looks good to me.

+1 for merging, but removing the System.out.println


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188929386
  
This looks good to me.

+1 for merging, but removing the System.out.println


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3516) JobManagerHACheckpointRecoveryITCase testCheckpointedStreamingSumProgram still fails :(

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-3516:
--

 Summary: JobManagerHACheckpointRecoveryITCase 
testCheckpointedStreamingSumProgram still fails :(
 Key: FLINK-3516
 URL: https://issues.apache.org/jira/browse/FLINK-3516
 Project: Flink
  Issue Type: Test
Reporter: Ufuk Celebi


https://s3.amazonaws.com/archive.travis-ci.org/jobs/111782050/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [hotfix] fix PageRank example to run with no a...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1714#issuecomment-188923753
  
+1 good to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [hotfix] fix PageRank example to run with no a...

[vasia]

GitHub user vasia opened a pull request:

https://github.com/apache/flink/pull/1714

[hotfix] fix PageRank example to run with no args; some doc fixes

This PR includes a fix for the java PageRank example to retrieve the 
required parameter `numPages ` when run with no arguments and some 
documentation fixes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vasia/flink examples-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1714


commit d9c89b26d288e75dfc708b42816cc35015dcbf20
Author: vasia 
Date:   2016-02-25T16:36:37Z

[hotfix] fix PageRank example to run with no args; some doc fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3442) Expose savepoint button on web ui

[Gyula Fora (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167599#comment-15167599
 ] 

Gyula Fora commented on FLINK-3442:
---

I thought that this should be equivalent to calling bin/flink savepoint jobid. 
So whatever savepoint path is configured for the given job will be used.

> Expose savepoint button on web ui
> -
>
> Key: FLINK-3442
> URL: https://issues.apache.org/jira/browse/FLINK-3442
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming, Webfrontend
>Reporter: Gyula Fora
>Priority: Minor
>
> Similarly to Cancel there should be a Savepoint button to initiate a 
> savepoint for streaming jobs.
> These 2 buttons should NOT be next to each other :)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3277) Use Value types in Gelly API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167586#comment-15167586
 ] 

ASF GitHub Bot commented on FLINK-3277:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1671#issuecomment-188918217
  
Thanks for checking this out.

Do you have an estimate of how this affects Gelly performance for some 
algorithms? How much does this improve the performance?


> Use Value types in Gelly API
> 
>
> Key: FLINK-3277
> URL: https://issues.apache.org/jira/browse/FLINK-3277
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> This would be a breaking change so the discussion needs to happen before the 
> 1.0.0 release.
> I think it would benefit Flink to use {{Value}} types wherever possible. The 
> {{Graph}} functions {{inDegrees}}, {{outDegrees}}, and {{getDegrees}} each 
> return {{DataSet>}}. Using {{Long}} creates a new heap object 
> for every serialization and deserialization. The mutable {{Value}} types do 
> not suffer from this issue when object reuse is enabled.
> I lean towards a preference for conciseness in documentation and performance 
> in examples and APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3277] Use Value types in Gelly API

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1671#issuecomment-188918217
  
Thanks for checking this out.

Do you have an estimate of how this affects Gelly performance for some 
algorithms? How much does this improve the performance?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3442) Expose savepoint button on web ui

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167572#comment-15167572
 ] 

Stephan Ewen commented on FLINK-3442:
-

This would rely on some pre-configured default safepoint target path?

> Expose savepoint button on web ui
> -
>
> Key: FLINK-3442
> URL: https://issues.apache.org/jira/browse/FLINK-3442
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming, Webfrontend
>Reporter: Gyula Fora
>Priority: Minor
>
> Similarly to Cancel there should be a Savepoint button to initiate a 
> savepoint for streaming jobs.
> These 2 buttons should NOT be next to each other :)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3515) Make the "file monitoring source" exactly-once

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-3515:
---

 Summary: Make the "file monitoring source" exactly-once
 Key: FLINK-3515
 URL: https://issues.apache.org/jira/browse/FLINK-3515
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Affects Versions: 0.10.2
Reporter: Stephan Ewen


The stream source that watches directories for changes is currently not 
"exactly-once".

To make it exactly once, the source (that generates files to be read) and the 
flatMap (that reads the files) need to keep track of where they were at the 
point of a checkpoint.

Assuming that files do not change after creation (HDFS / S3 style), we can make 
this the following way:

  - The source can track the files it already emitted downstream via file 
creation/modification timestamp, assuming that new files always get newer 
timestamps.

  - The flatMappers need to always store the path of their current file 
fragment, plus the byte offset where they were within that file split.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3432) ZookeeperOffsetHandlerTest fails on windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167557#comment-15167557
 ] 

ASF GitHub Bot commented on FLINK-3432:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1664#issuecomment-188911039
  
Looks good to my, +1 to merge


> ZookeeperOffsetHandlerTest fails on windows
> ---
>
> Key: FLINK-3432
> URL: https://issues.apache.org/jira/browse/FLINK-3432
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.0.0
>
>
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandlerTest
>   Time elapsed: 0.351 sec  <<< FAILURE!
> java.lang.AssertionError: cannot create kafka temp dir
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at 
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.prepare(KafkaTestEnvironmentImpl.java:139)
> at 
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.prepare(KafkaTestBase.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandlerTest
>   Time elapsed: 0.352 sec  <<< ERROR!
> java.lang.NullPointerException: null
> at 
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.shutdown(KafkaTestEnvironmentImpl.java:187)
> at 
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.shutDownServices(KafkaTestBase.java:127)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3432] Fixed ZookeeperOffsethandlerTest ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1664#issuecomment-188911039
  
Looks good to my, +1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3514) Add support for slowly changing streaming broadcast variables

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-3514:
---

 Summary: Add support for slowly changing streaming broadcast 
variables
 Key: FLINK-3514
 URL: https://issues.apache.org/jira/browse/FLINK-3514
 Project: Flink
  Issue Type: New Feature
  Components: Streaming
Reporter: Stephan Ewen


With "slow changing inputs", I mean auxiliary data sets that change over time. 
Think about dictionaries, set of rules, etc that are updates at slow rates and 
then applied onto another stream.

They are a bit like broadcast variables in the DataSet API, but not constant 
over the entire lifetime of the job.

One can currently approximate this via connected streams:
{code}
DataStream rules = ...
DataStream events = ...

rules.broadcast().connect(events).flatMap(...);
{code}

The disadvantage is that one has no control that the broadcasted "rules" input 
is materialized in the function before the "events" are delivered.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167506#comment-15167506
 ] 

Greg Hogan commented on FLINK-3511:
---

Is the only difference from {{flink-examples-batch}} that 
{{flink-gelly-examples}} would need to either include Gelly in the single or 
individual jars, or instruct the user to copy or download a Gelly jar into the 
lib folder?

> Flink library examples not runnable withput adding dependencies
> ---
>
> Key: FLINK-3511
> URL: https://issues.apache.org/jira/browse/FLINK-3511
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.0.0
>Reporter: Márton Balassi
>
> Recent changes to the build [1] where many libraries got their core 
> dependencies (the ones included in the flink-dist fat jar) moved to the 
> provided scope.
> The reasoning was that when submitting to the Flink cluster the application 
> already has these dependencies, while when a user writes a program against 
> these libraries she will include the core dependencies explicitly anyway.
> There is one other case of usage however, namely when someone is trying to 
> run an application defined in these libraries depending on the core jars. To 
> give an example if you were to run the Gelly ConnectedComponents example [2] 
> from an IDE after importing Flink (or running with java -jar without 
> including the flink fat jar in the classpath) you would receive the following 
> class not found exception as per the current master:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/flink/api/common/ProgramDescription
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> Where the class missing in runtime is in flink-core, a transitive dependency 
> of the jars moved to the provided scope.
> Funny thing is we have tests in place to run our examples, but those add test 
> scope dependencies re-adding the missing classes, so it is never discovered.
> I agree with the original purpose of PR #1683, but also think that the 
> current state makes for very inconvenient user experience.
> I would like to open a discussion on how and when to resolve the issue given 
> the release of 1.0.0.
> 1. Is it a release blocker?
> 2. Should the change be reverted or is it sufficient to have proper 
> documentation around it? Maybe a maven profile for explicitly for developing 
> Flink without the provided scope?
> Note that the issue was originally reported by Gábor Gévay.
> [1] https://github.com/apache/flink/pull/1683
> [2] 
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3509) Update Hadoop versions in release script and on travis to the latest minor version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167481#comment-15167481
 ] 

ASF GitHub Bot commented on FLINK-3509:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1713

[FLINK-3509] Update Hadoop minor versions to latest & sync travis and 
release scripts



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3509

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1713


commit 45344e162f432003c2aded4cbae2841e7c2e4476
Author: Robert Metzger 
Date:   2016-02-25T15:30:56Z

[FLINK-3509] Update Hadoop minor versions to latest & sync travis and 
release scripts




> Update Hadoop versions in release script and on travis to the latest minor 
> version
> --
>
> Key: FLINK-3509
> URL: https://issues.apache.org/jira/browse/FLINK-3509
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> The Hadoop versions in the travis.yml and the create-release-files.sh script 
> are not updated to the latest Hadoop minor releases. Hadoop 2.7.2 and 2.6.4 
> were released in the meantime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3509] Update Hadoop minor versions to l...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1713

[FLINK-3509] Update Hadoop minor versions to latest & sync travis and 
release scripts



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3509

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1713


commit 45344e162f432003c2aded4cbae2841e7c2e4476
Author: Robert Metzger 
Date:   2016-02-25T15:30:56Z

[FLINK-3509] Update Hadoop minor versions to latest & sync travis and 
release scripts




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167446#comment-15167446
 ] 

Márton Balassi commented on FLINK-3511:
---

Somewhat sensible for me, but it just propagates the problem to 
flink-gelly-examples then, doesn't?

> Flink library examples not runnable withput adding dependencies
> ---
>
> Key: FLINK-3511
> URL: https://issues.apache.org/jira/browse/FLINK-3511
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.0.0
>Reporter: Márton Balassi
>
> Recent changes to the build [1] where many libraries got their core 
> dependencies (the ones included in the flink-dist fat jar) moved to the 
> provided scope.
> The reasoning was that when submitting to the Flink cluster the application 
> already has these dependencies, while when a user writes a program against 
> these libraries she will include the core dependencies explicitly anyway.
> There is one other case of usage however, namely when someone is trying to 
> run an application defined in these libraries depending on the core jars. To 
> give an example if you were to run the Gelly ConnectedComponents example [2] 
> from an IDE after importing Flink (or running with java -jar without 
> including the flink fat jar in the classpath) you would receive the following 
> class not found exception as per the current master:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/flink/api/common/ProgramDescription
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> Where the class missing in runtime is in flink-core, a transitive dependency 
> of the jars moved to the provided scope.
> Funny thing is we have tests in place to run our examples, but those add test 
> scope dependencies re-adding the missing classes, so it is never discovered.
> I agree with the original purpose of PR #1683, but also think that the 
> current state makes for very inconvenient user experience.
> I would like to open a discussion on how and when to resolve the issue given 
> the release of 1.0.0.
> 1. Is it a release blocker?
> 2. Should the change be reverted or is it sufficient to have proper 
> documentation around it? Maybe a maven profile for explicitly for developing 
> Flink without the provided scope?
> Note that the issue was originally reported by Gábor Gévay.
> [1] https://github.com/apache/flink/pull/1683
> [2] 
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3511) Flink library examples not runnable withput adding dependencies

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167446#comment-15167446
 ] 

Márton Balassi edited comment on FLINK-3511 at 2/25/16 5:03 PM:


Somewhat sensible for me, but it just propagates the problem to 
flink-gelly-examples then, doesn't it?


was (Author: mbalassi):
Somewhat sensible for me, but it just propagates the problem to 
flink-gelly-examples then, doesn't?

> Flink library examples not runnable withput adding dependencies
> ---
>
> Key: FLINK-3511
> URL: https://issues.apache.org/jira/browse/FLINK-3511
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.0.0
>Reporter: Márton Balassi
>
> Recent changes to the build [1] where many libraries got their core 
> dependencies (the ones included in the flink-dist fat jar) moved to the 
> provided scope.
> The reasoning was that when submitting to the Flink cluster the application 
> already has these dependencies, while when a user writes a program against 
> these libraries she will include the core dependencies explicitly anyway.
> There is one other case of usage however, namely when someone is trying to 
> run an application defined in these libraries depending on the core jars. To 
> give an example if you were to run the Gelly ConnectedComponents example [2] 
> from an IDE after importing Flink (or running with java -jar without 
> including the flink fat jar in the classpath) you would receive the following 
> class not found exception as per the current master:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/flink/api/common/ProgramDescription
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> Where the class missing in runtime is in flink-core, a transitive dependency 
> of the jars moved to the provided scope.
> Funny thing is we have tests in place to run our examples, but those add test 
> scope dependencies re-adding the missing classes, so it is never discovered.
> I agree with the original purpose of PR #1683, but also think that the 
> current state makes for very inconvenient user experience.
> I would like to open a discussion on how and when to resolve the issue given 
> the release of 1.0.0.
> 1. Is it a release blocker?
> 2. Should the change be reverted or is it sufficient to have proper 
> documentation around it? Maybe a maven profile for explicitly for developing 
> Flink without the provided scope?
> Note that the issue was originally reported by Gábor Gévay.
> [1] https://github.com/apache/flink/pull/1683
> [2] 
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3463] implement calc translation

[vasia]

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1696#issuecomment-188878585
  
Thanks! I'll merge both once Travis passes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3463) Implement Calc Support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167443#comment-15167443
 ] 

ASF GitHub Bot commented on FLINK-3463:
---

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1696#issuecomment-188878585
  
Thanks! I'll merge both once Travis passes.


> Implement Calc Support
> --
>
> Key: FLINK-3463
> URL: https://issues.apache.org/jira/browse/FLINK-3463
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167429#comment-15167429
 ] 

Greg Hogan commented on FLINK-3511:
---

Then a flink-gelly-examples?

> Flink library examples not runnable withput adding dependencies
> ---
>
> Key: FLINK-3511
> URL: https://issues.apache.org/jira/browse/FLINK-3511
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.0.0
>Reporter: Márton Balassi
>
> Recent changes to the build [1] where many libraries got their core 
> dependencies (the ones included in the flink-dist fat jar) moved to the 
> provided scope.
> The reasoning was that when submitting to the Flink cluster the application 
> already has these dependencies, while when a user writes a program against 
> these libraries she will include the core dependencies explicitly anyway.
> There is one other case of usage however, namely when someone is trying to 
> run an application defined in these libraries depending on the core jars. To 
> give an example if you were to run the Gelly ConnectedComponents example [2] 
> from an IDE after importing Flink (or running with java -jar without 
> including the flink fat jar in the classpath) you would receive the following 
> class not found exception as per the current master:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/flink/api/common/ProgramDescription
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> Where the class missing in runtime is in flink-core, a transitive dependency 
> of the jars moved to the provided scope.
> Funny thing is we have tests in place to run our examples, but those add test 
> scope dependencies re-adding the missing classes, so it is never discovered.
> I agree with the original purpose of PR #1683, but also think that the 
> current state makes for very inconvenient user experience.
> I would like to open a discussion on how and when to resolve the issue given 
> the release of 1.0.0.
> 1. Is it a release blocker?
> 2. Should the change be reverted or is it sufficient to have proper 
> documentation around it? Maybe a maven profile for explicitly for developing 
> Flink without the provided scope?
> Note that the issue was originally reported by Gábor Gévay.
> [1] https://github.com/apache/flink/pull/1683
> [2] 
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3399) Count with timeout trigger

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167426#comment-15167426
 ] 

ASF GitHub Bot commented on FLINK-3399:
---

Github user shikhar closed the pull request at:

https://github.com/apache/flink/pull/1636


> Count with timeout trigger
> --
>
> Key: FLINK-3399
> URL: https://issues.apache.org/jira/browse/FLINK-3399
> Project: Flink
>  Issue Type: Improvement
>Reporter: Shikhar Bhushan
>Priority: Minor
>
> I created an implementation of a trigger that I'd like to contribute, 
> https://gist.github.com/shikhar/2cb9f1b792be31b7c16e
> An example application - if a sink function operates more efficiently if it 
> is writing in a batched fashion, then the windowing mechanism + this trigger 
> can be used. Count to have an upper bound on batch size & better control on 
> memory usage, and timeout to ensure timeliness of the outputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger

[shikhar]

Github user shikhar closed the pull request at:

https://github.com/apache/flink/pull/1636


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3399) Count with timeout trigger

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167425#comment-15167425
 ] 

ASF GitHub Bot commented on FLINK-3399:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1636#issuecomment-188874847
  
I would prefer to just add the compassable way, yes. Sorry that you now 
already put effort into this.


> Count with timeout trigger
> --
>
> Key: FLINK-3399
> URL: https://issues.apache.org/jira/browse/FLINK-3399
> Project: Flink
>  Issue Type: Improvement
>Reporter: Shikhar Bhushan
>Priority: Minor
>
> I created an implementation of a trigger that I'd like to contribute, 
> https://gist.github.com/shikhar/2cb9f1b792be31b7c16e
> An example application - if a sink function operates more efficiently if it 
> is writing in a batched fashion, then the windowing mechanism + this trigger 
> can be used. Count to have an upper bound on batch size & better control on 
> memory usage, and timeout to ensure timeliness of the outputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1636#issuecomment-188874847
  
I would prefer to just add the compassable way, yes. Sorry that you now 
already put effort into this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3399) Count with timeout trigger

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167417#comment-15167417
 ] 

ASF GitHub Bot commented on FLINK-3399:
---

Github user shikhar commented on the pull request:

https://github.com/apache/flink/pull/1636#issuecomment-188874026
  
Thanks @StephanEwen, @aljoscha noted on the JIRA that it may be better to 
create a composable mechanism like `Any.of(,)`

Until that exists, I do think this is useful functionality, but let me know 
if we'd prefer to just have the composable way and I should close this PR.


> Count with timeout trigger
> --
>
> Key: FLINK-3399
> URL: https://issues.apache.org/jira/browse/FLINK-3399
> Project: Flink
>  Issue Type: Improvement
>Reporter: Shikhar Bhushan
>Priority: Minor
>
> I created an implementation of a trigger that I'd like to contribute, 
> https://gist.github.com/shikhar/2cb9f1b792be31b7c16e
> An example application - if a sink function operates more efficiently if it 
> is writing in a batched fashion, then the windowing mechanism + this trigger 
> can be used. Count to have an upper bound on batch size & better control on 
> memory usage, and timeout to ensure timeliness of the outputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger

[shikhar]

Github user shikhar commented on the pull request:

https://github.com/apache/flink/pull/1636#issuecomment-188874026
  
Thanks @StephanEwen, @aljoscha noted on the JIRA that it may be better to 
create a composable mechanism like `Any.of(,)`

Until that exists, I do think this is useful functionality, but let me know 
if we'd prefer to just have the composable way and I should close this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs

[Gabor Gevay (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167415#comment-15167415
 ] 

Gabor Gevay commented on FLINK-3322:


It's making it even worse:
false, 500m: 30s
false, 5000m: 115s
true, 500m: 8s
true, 5000m: 10s


> MemoryManager creates too much GC pressure with iterative jobs
> --
>
> Key: FLINK-3322
> URL: https://issues.apache.org/jira/browse/FLINK-3322
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Priority: Critical
> Fix For: 1.0.0
>
>
> When taskmanager.memory.preallocate is false (the default), released memory 
> segments are not added to a pool, but the GC is expected to take care of 
> them. This puts too much pressure on the GC with iterative jobs, where the 
> operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the 
> JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. 
> (It will generate some lookuptables to /tmp on first run for a few minutes.) 
> (I think the slowdown might also depend somewhat on 
> taskmanager.memory.fraction, because more unused non-managed memory results 
> in rarer GCs.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167416#comment-15167416
 ] 

Márton Balassi commented on FLINK-3511:
---

I am against moving the examples of a library out of the library. It would also 
mean that flink-examples-batch would need to depend on gelly, which would add 
gelly automatically as a dependency to flink-dist, something that we would like 
to avoid as long as it is beta.

And even if we did, it would only heal one single symptom instead of curing the 
underlying problem. With the current solution any example in any library is a 
potential victim of this issue.

> Flink library examples not runnable withput adding dependencies
> ---
>
> Key: FLINK-3511
> URL: https://issues.apache.org/jira/browse/FLINK-3511
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.0.0
>Reporter: Márton Balassi
>
> Recent changes to the build [1] where many libraries got their core 
> dependencies (the ones included in the flink-dist fat jar) moved to the 
> provided scope.
> The reasoning was that when submitting to the Flink cluster the application 
> already has these dependencies, while when a user writes a program against 
> these libraries she will include the core dependencies explicitly anyway.
> There is one other case of usage however, namely when someone is trying to 
> run an application defined in these libraries depending on the core jars. To 
> give an example if you were to run the Gelly ConnectedComponents example [2] 
> from an IDE after importing Flink (or running with java -jar without 
> including the flink fat jar in the classpath) you would receive the following 
> class not found exception as per the current master:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/flink/api/common/ProgramDescription
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> Where the class missing in runtime is in flink-core, a transitive dependency 
> of the jars moved to the provided scope.
> Funny thing is we have tests in place to run our examples, but those add test 
> scope dependencies re-adding the missing classes, so it is never discovered.
> I agree with the original purpose of PR #1683, but also think that the 
> current state makes for very inconvenient user experience.
> I would like to open a discussion on how and when to resolve the issue given 
> the release of 1.0.0.
> 1. Is it a release blocker?
> 2. Should the change be reverted or is it sufficient to have proper 
> documentation around it? Maybe a maven profile for explicitly for developing 
> Flink without the provided scope?
> Note that the issue was originally reported by Gábor Gévay.
> [1] https://github.com/apache/flink/pull/1683
> [2] 
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3399) Count with timeout trigger

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167413#comment-15167413
 ] 

ASF GitHub Bot commented on FLINK-3399:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1636#issuecomment-188872638
  
Sorry for the late response, most people are fixing issues for the 1.0 
release candidates.

This actually looks like a nice addition.
+1 for adding a small library/collection of typical triggers and patterns.

Let's merge this into master once the 1.0 release is out, to be released 
with either 1.1.0 or 1.0.1.




> Count with timeout trigger
> --
>
> Key: FLINK-3399
> URL: https://issues.apache.org/jira/browse/FLINK-3399
> Project: Flink
>  Issue Type: Improvement
>Reporter: Shikhar Bhushan
>Priority: Minor
>
> I created an implementation of a trigger that I'd like to contribute, 
> https://gist.github.com/shikhar/2cb9f1b792be31b7c16e
> An example application - if a sink function operates more efficiently if it 
> is writing in a batched fashion, then the windowing mechanism + this trigger 
> can be used. Count to have an upper bound on batch size & better control on 
> memory usage, and timeout to ensure timeliness of the outputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1636#issuecomment-188872638
  
Sorry for the late response, most people are fixing issues for the 1.0 
release candidates.

This actually looks like a nice addition.
+1 for adding a small library/collection of typical triggers and patterns.

Let's merge this into master once the 1.0 release is out, to be released 
with either 1.1.0 or 1.0.1.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167404#comment-15167404
 ] 

Greg Hogan commented on FLINK-3511:
---

Could the Gelly examples be moved to flink-examples-batch?

> Flink library examples not runnable withput adding dependencies
> ---
>
> Key: FLINK-3511
> URL: https://issues.apache.org/jira/browse/FLINK-3511
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.0.0
>Reporter: Márton Balassi
>
> Recent changes to the build [1] where many libraries got their core 
> dependencies (the ones included in the flink-dist fat jar) moved to the 
> provided scope.
> The reasoning was that when submitting to the Flink cluster the application 
> already has these dependencies, while when a user writes a program against 
> these libraries she will include the core dependencies explicitly anyway.
> There is one other case of usage however, namely when someone is trying to 
> run an application defined in these libraries depending on the core jars. To 
> give an example if you were to run the Gelly ConnectedComponents example [2] 
> from an IDE after importing Flink (or running with java -jar without 
> including the flink fat jar in the classpath) you would receive the following 
> class not found exception as per the current master:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/flink/api/common/ProgramDescription
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> Where the class missing in runtime is in flink-core, a transitive dependency 
> of the jars moved to the provided scope.
> Funny thing is we have tests in place to run our examples, but those add test 
> scope dependencies re-adding the missing classes, so it is never discovered.
> I agree with the original purpose of PR #1683, but also think that the 
> current state makes for very inconvenient user experience.
> I would like to open a discussion on how and when to resolve the issue given 
> the release of 1.0.0.
> 1. Is it a release blocker?
> 2. Should the change be reverted or is it sufficient to have proper 
> documentation around it? Maybe a maven profile for explicitly for developing 
> Flink without the provided scope?
> Note that the issue was originally reported by Gábor Gévay.
> [1] https://github.com/apache/flink/pull/1683
> [2] 
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167402#comment-15167402
 ] 

Stephan Ewen commented on FLINK-3322:
-

How does the off-heap memory setting affect these stats?
Do the numbers look similar, or does pre-allocation not make that much of a 
difference there?

> MemoryManager creates too much GC pressure with iterative jobs
> --
>
> Key: FLINK-3322
> URL: https://issues.apache.org/jira/browse/FLINK-3322
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Priority: Critical
> Fix For: 1.0.0
>
>
> When taskmanager.memory.preallocate is false (the default), released memory 
> segments are not added to a pool, but the GC is expected to take care of 
> them. This puts too much pressure on the GC with iterative jobs, where the 
> operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the 
> JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. 
> (It will generate some lookuptables to /tmp on first run for a few minutes.) 
> (I think the slowdown might also depend somewhat on 
> taskmanager.memory.fraction, because more unused non-managed memory results 
> in rarer GCs.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167393#comment-15167393
 ] 

ASF GitHub Bot commented on FLINK-3513:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1712#issuecomment-188862999
  
`mvn clean verify` was successful on my local machine, so if we need this 
for a new RC, we can merge it before the Travis run finishes.


> Fix interplay of automatic Operator UID and Changing name of WindowOperator
> ---
>
> Key: FLINK-3513
> URL: https://issues.apache.org/jira/browse/FLINK-3513
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
>
> WindowOperator can have a changing name because it has the TypeSerializer 
> .toString() output in it's name. For some type serializers that don't 
> implement toString() this means that the name changes.
> This means that savepoint restore does not work for the automatically 
> generated UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1712#issuecomment-188862999
  
`mvn clean verify` was successful on my local machine, so if we need this 
for a new RC, we can merge it before the Travis run finishes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs

[Gabor Gevay (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167377#comment-15167377
 ] 

Gabor Gevay commented on FLINK-3322:


I have constructed a much simpler example to demonstrate the problem: 
ConnectedComponents on a graph that is an 1000 length path: 1->2, 2->3, 3->4, 
4->5, ... 999->1000:
https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc-2
The class to run is org.apache.flink.graph.example.ConnectedComponents. Try 
increasing the memory from eg. 500m to 5000m, and look at the difference when  
taskmanager.memory.preallocate is true and false (TaskManager.scala:1713).
I measured the following times on my laptop:
false, 500m:  14s
false, 5000m: 115s
true, 500m:   8s
true, 5000m:  13s

(I guess that the difference between the two runs where preallocate is true is 
due to the time it takes for the JVM to allocate the memory once, but this 
should also be checked that it isn't for some other unexpected reason.)

So the bottom line is that the problem gets worse when there are more 
iterations. (We have 1001 iterations in the linked example.)

> MemoryManager creates too much GC pressure with iterative jobs
> --
>
> Key: FLINK-3322
> URL: https://issues.apache.org/jira/browse/FLINK-3322
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Priority: Critical
> Fix For: 1.0.0
>
>
> When taskmanager.memory.preallocate is false (the default), released memory 
> segments are not added to a pool, but the GC is expected to take care of 
> them. This puts too much pressure on the GC with iterative jobs, where the 
> operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the 
> JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. 
> (It will generate some lookuptables to /tmp on first run for a few minutes.) 
> (I think the slowdown might also depend somewhat on 
> taskmanager.memory.fraction, because more unused non-managed memory results 
> in rarer GCs.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167373#comment-15167373
 ] 

ASF GitHub Bot commented on FLINK-3513:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1712#issuecomment-188858495
  
:+1: 


> Fix interplay of automatic Operator UID and Changing name of WindowOperator
> ---
>
> Key: FLINK-3513
> URL: https://issues.apache.org/jira/browse/FLINK-3513
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
>
> WindowOperator can have a changing name because it has the TypeSerializer 
> .toString() output in it's name. For some type serializers that don't 
> implement toString() this means that the name changes.
> This means that savepoint restore does not work for the automatically 
> generated UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1712#issuecomment-188858495
  
:+1: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167351#comment-15167351
 ] 

Robert Metzger commented on FLINK-3501:
---

I'm not able to reproduce the issue:

{code}
1 robert@robert-da /tmp/test % java -version
openjdk version "1.8.0_74"
OpenJDK Runtime Environment (build 1.8.0_74-b02)
OpenJDK 64-Bit Server VM (build 25.74-b02, mixed mode)
robert@robert-da /tmp/test % jar xf flink-runtime-0.10.2-hadoop1.jar
robert@robert-da /tmp/test %   
{code}

It could be that you are using a different JDK which has that issue. What's 
your JDK?



> flink-runtime-0.10.2-hadoop-1 jar problem
> -
>
> Key: FLINK-3501
> URL: https://issues.apache.org/jira/browse/FLINK-3501
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.10.2
>Reporter: Paul Harris
>
> Unjaring flink-runtime-0.10.2-hadoop-1 results in an error.
> Steps to reproduce
> {code:none}
> jar xf flink-runtime-0.10.2-hadoop1.jar
> {code}
> Result...
> {code:none}
> java.io.IOException: META-INF/license : could not create directory
>   at sun.tools.jar.Main.extractFile(Main.java:934)
>   at sun.tools.jar.Main.extract(Main.java:877)
>   at sun.tools.jar.Main.run(Main.java:263)
>   at sun.tools.jar.Main.main(Main.java:1177)
> {code}
> This is the result of the jar containing a file 'license' and a directory 
> license'. The file is unpacked first, preventing the directory being unpacked 
> to the same name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167350#comment-15167350
 ] 

ASF GitHub Bot commented on FLINK-3513:
---

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1712

[FLINK-3513] [FLINK-3512] Fix savepoint issues

@aljoscha spotted the following two issues (so far) while working with 
savepoints and the rocks DB state backend.

- FLINK-3513: The operator name is used in the deterministic hash 
assignment, which is actually not necessary and leads to problems with the 
WindowOperator (changing name).
- FLINK-3512: The savepoint backend was trying to be too clever and fell 
back to jobmanager when it found "wrong" config combinations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink hash_gen

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1712.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1712


commit dab3c654fafecf8caee6453b7da8965174646771
Author: Ufuk Celebi 
Date:   2016-02-25T15:55:04Z

[FLINK-3513] [runtime] Fix interplay of automatic Operator UID and Changing 
name of WindowOperator

commit 3c3a81acee8de1128a3eee7f59ca6f8803acbd5c
Author: Ufuk Celebi 
Date:   2016-02-25T15:55:38Z

[FLINK-3512] [runtime] Savepoint backend should not revert to 'jobmanager'




> Fix interplay of automatic Operator UID and Changing name of WindowOperator
> ---
>
> Key: FLINK-3513
> URL: https://issues.apache.org/jira/browse/FLINK-3513
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
>
> WindowOperator can have a changing name because it has the TypeSerializer 
> .toString() output in it's name. For some type serializers that don't 
> implement toString() this means that the name changes.
> This means that savepoint restore does not work for the automatically 
> generated UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1712

[FLINK-3513] [FLINK-3512] Fix savepoint issues

@aljoscha spotted the following two issues (so far) while working with 
savepoints and the rocks DB state backend.

- FLINK-3513: The operator name is used in the deterministic hash 
assignment, which is actually not necessary and leads to problems with the 
WindowOperator (changing name).
- FLINK-3512: The savepoint backend was trying to be too clever and fell 
back to jobmanager when it found "wrong" config combinations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink hash_gen

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1712.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1712


commit dab3c654fafecf8caee6453b7da8965174646771
Author: Ufuk Celebi 
Date:   2016-02-25T15:55:04Z

[FLINK-3513] [runtime] Fix interplay of automatic Operator UID and Changing 
name of WindowOperator

commit 3c3a81acee8de1128a3eee7f59ca6f8803acbd5c
Author: Ufuk Celebi 
Date:   2016-02-25T15:55:38Z

[FLINK-3512] [runtime] Savepoint backend should not revert to 'jobmanager'




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3512) Savepoint backend should not revert to "jobmanager"

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-3512:
---

 Summary: Savepoint backend should not revert to "jobmanager"
 Key: FLINK-3512
 URL: https://issues.apache.org/jira/browse/FLINK-3512
 Project: Flink
  Issue Type: Bug
Reporter: Aljoscha Krettek
Assignee: Ufuk Celebi
Priority: Blocker


When not setting a state backend in the config the savepoint coordinator 
defaults to use "jobmanager" as savepoint backend, ignoring the setting for the 
savepoint backend in the config.

This means that when setting a StateBackend only per job the system always 
falls back to "jobmanager" even though it could use "filesystem" to make 
savepoints persistent.

See SavepointStoreFactory.java:71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-3513:

Assignee: Ufuk Celebi

> Fix interplay of automatic Operator UID and Changing name of WindowOperator
> ---
>
> Key: FLINK-3513
> URL: https://issues.apache.org/jira/browse/FLINK-3513
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Blocker
>
> WindowOperator can have a changing name because it has the TypeSerializer 
> .toString() output in it's name. For some type serializers that don't 
> implement toString() this means that the name changes.
> This means that savepoint restore does not work for the automatically 
> generated UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-3513:
---

 Summary: Fix interplay of automatic Operator UID and Changing name 
of WindowOperator
 Key: FLINK-3513
 URL: https://issues.apache.org/jira/browse/FLINK-3513
 Project: Flink
  Issue Type: Bug
  Components: Streaming
Affects Versions: 1.0.0
Reporter: Aljoscha Krettek
Priority: Blocker


WindowOperator can have a changing name because it has the TypeSerializer 
.toString() output in it's name. For some type serializers that don't implement 
toString() this means that the name changes.

This means that savepoint restore does not work for the automatically generated 
UID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3463) Implement Calc Support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167330#comment-15167330
 ] 

ASF GitHub Bot commented on FLINK-3463:
---

Github user twalthr commented on the pull request:

https://github.com/apache/flink/pull/1696#issuecomment-188846370
  
+1 for merging


> Implement Calc Support
> --
>
> Key: FLINK-3463
> URL: https://issues.apache.org/jira/browse/FLINK-3463
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3463] implement calc translation

[twalthr]

Github user twalthr commented on the pull request:

https://github.com/apache/flink/pull/1696#issuecomment-188846370
  
+1 for merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167319#comment-15167319
 ] 

ASF GitHub Bot commented on FLINK-3496:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/1711

git push -f[FLINK-3496] Disable RocksDB tests on Windows



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3496_rocksdb_disable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1711.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1711


commit a7b965288f4a913bf0f7c37e3f7c88748e3f124a
Author: zentol 
Date:   2016-02-25T13:31:34Z

[FLINK-3496] Disable RocksDB tests on Windows




> Flink-ml tests fail on Windows
> --
>
> Key: FLINK-3496
> URL: https://issues.apache.org/jira/browse/FLINK-3496
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>
> [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 
> ---
> [INFO]
> [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 ---
> The system cannot find the file specified.
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:03 min
> [INFO] Finished at: 2016-02-24T12:47:23+01:00
> [INFO] Final Memory: 28M/506M



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: git push -f[FLINK-3496] Disable RocksDB tests ...

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/1711

git push -f[FLINK-3496] Disable RocksDB tests on Windows



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3496_rocksdb_disable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1711.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1711


commit a7b965288f4a913bf0f7c37e3f7c88748e3f124a
Author: zentol 
Date:   2016-02-25T13:31:34Z

[FLINK-3496] Disable RocksDB tests on Windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3496] Fix ML test discovery on Windows

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/1710

[FLINK-3496] Fix ML test discovery on Windows

This PR adds OS specific arguments in the flink-ml pom that are passed to 
the scala test runner.

The runner uses pattern matching to discover test files that should be run. 
The original pattern without quotes works on unix but fails on windows, with 
quotes it works on windows but skip all tests on unix.

As such I've added 2 profiles that activate based on the OS and set a 
pattern property, which is then passed to the runner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3496_ml_pattern

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1710.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1710


commit efc62c42879508450544ca72d464420bf3934724
Author: zentol 
Date:   2016-02-25T15:09:42Z

[FLINK-3496] Fix ML test discovery on Windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3508) Add more test cases to verify the rules of logical plan optimization

[Chengxiang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167301#comment-15167301
 ] 

Chengxiang Li commented on FLINK-3508:
--

Merged to tableOnCalcite branch at 72686231fd8f9fa6a1c05df48f6f29eaa3ca4f2b.

> Add more test cases to verify the rules of logical plan optimization
> 
>
> Key: FLINK-3508
> URL: https://issues.apache.org/jira/browse/FLINK-3508
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API
>Reporter: Chengxiang Li
>Assignee: Chengxiang Li
>Priority: Minor
>
> We have enabled many rules in logical plan optimization phase, more 
> complicated test cases should be added to verify whether these rules actally 
> work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3508) Add more test cases to verify the rules of logical plan optimization

[Chengxiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chengxiang Li closed FLINK-3508.


> Add more test cases to verify the rules of logical plan optimization
> 
>
> Key: FLINK-3508
> URL: https://issues.apache.org/jira/browse/FLINK-3508
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API
>Reporter: Chengxiang Li
>Assignee: Chengxiang Li
>Priority: Minor
>
> We have enabled many rules in logical plan optimization phase, more 
> complicated test cases should be added to verify whether these rules actally 
> work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3508) Add more test cases to verify the rules of logical plan optimization

[Chengxiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chengxiang Li resolved FLINK-3508.
--
Resolution: Fixed

> Add more test cases to verify the rules of logical plan optimization
> 
>
> Key: FLINK-3508
> URL: https://issues.apache.org/jira/browse/FLINK-3508
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API
>Reporter: Chengxiang Li
>Assignee: Chengxiang Li
>Priority: Minor
>
> We have enabled many rules in logical plan optimization phase, more 
> complicated test cases should be added to verify whether these rules actally 
> work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167298#comment-15167298
 ] 

ASF GitHub Bot commented on FLINK-3496:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/1710

[FLINK-3496] Fix ML test discovery on Windows

This PR adds OS specific arguments in the flink-ml pom that are passed to 
the scala test runner.

The runner uses pattern matching to discover test files that should be run. 
The original pattern without quotes works on unix but fails on windows, with 
quotes it works on windows but skip all tests on unix.

As such I've added 2 profiles that activate based on the OS and set a 
pattern property, which is then passed to the runner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3496_ml_pattern

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1710.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1710


commit efc62c42879508450544ca72d464420bf3934724
Author: zentol 
Date:   2016-02-25T15:09:42Z

[FLINK-3496] Fix ML test discovery on Windows




> Flink-ml tests fail on Windows
> --
>
> Key: FLINK-3496
> URL: https://issues.apache.org/jira/browse/FLINK-3496
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>
> [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 
> ---
> [INFO]
> [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 ---
> The system cannot find the file specified.
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:03 min
> [INFO] Finished at: 2016-02-24T12:47:23+01:00
> [INFO] Final Memory: 28M/506M



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3496) Flink-ml tests fail on Windows

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167281#comment-15167281
 ] 

Chesnay Schepler edited comment on FLINK-3496 at 2/25/16 2:56 PM:
--

using 
{code}"(?!(IT|Integration))(Test|Suite|Case)"{code} 
(essentially adding "") i could get the tests to run on windows, but now they 
no longer run on travis. One step forward, one step back.


was (Author: zentol):
using 
{code}"(?!(IT|Integration))(Test|Suite|Case)"{code} i 
could get the tests to run on windows, but now they no longer run on travis. 
One step forward, one step back.

> Flink-ml tests fail on Windows
> --
>
> Key: FLINK-3496
> URL: https://issues.apache.org/jira/browse/FLINK-3496
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>
> [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 
> ---
> [INFO]
> [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 ---
> The system cannot find the file specified.
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:03 min
> [INFO] Finished at: 2016-02-24T12:47:23+01:00
> [INFO] Final Memory: 28M/506M



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167281#comment-15167281
 ] 

Chesnay Schepler commented on FLINK-3496:
-

using 
{code}"(?!(IT|Integration))(Test|Suite|Case)"{code} i 
could get the tests to run on windows, but now they no longer run on travis. 
One step forward, one step back.

> Flink-ml tests fail on Windows
> --
>
> Key: FLINK-3496
> URL: https://issues.apache.org/jira/browse/FLINK-3496
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>
> [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 
> ---
> [INFO]
> [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 ---
> The system cannot find the file specified.
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:03 min
> [INFO] Finished at: 2016-02-24T12:47:23+01:00
> [INFO] Final Memory: 28M/506M



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3398) Flink Kafka consumer should support auto-commit opt-outs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167277#comment-15167277
 ] 

ASF GitHub Bot commented on FLINK-3398:
---

Github user shikhar commented on the pull request:

https://github.com/apache/flink/pull/1690#issuecomment-188818665
  
Just fixed the `Kafka09ITCase.testCheckpointing()` fail as well, hopefully 
CI is green with that.


> Flink Kafka consumer should support auto-commit opt-outs
> 
>
> Key: FLINK-3398
> URL: https://issues.apache.org/jira/browse/FLINK-3398
> Project: Flink
>  Issue Type: Bug
>Reporter: Shikhar Bhushan
>
> Currently the Kafka source will commit consumer offsets to Zookeeper, either 
> upon a checkpoint if checkpointing is enabled, otherwise periodically based 
> on {{auto.commit.interval.ms}}
> It should be possible to opt-out of committing consumer offsets to Zookeeper. 
> Kafka has this config as {{auto.commit.enable}} (0.8) and 
> {{enable.auto.commit}} (0.9).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3398: Allow for opting-out from Kafka of...

[shikhar]

Github user shikhar commented on the pull request:

https://github.com/apache/flink/pull/1690#issuecomment-188818665
  
Just fixed the `Kafka09ITCase.testCheckpointing()` fail as well, hopefully 
CI is green with that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3511) Flink library examples not runnable withput adding dependencies

[JIRA]

Márton Balassi created FLINK-3511:
-

 Summary: Flink library examples not runnable withput adding 
dependencies
 Key: FLINK-3511
 URL: https://issues.apache.org/jira/browse/FLINK-3511
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.0.0
Reporter: Márton Balassi


Recent changes to the build [1] where many libraries got their core 
dependencies (the ones included in the flink-dist fat jar) moved to the 
provided scope.

The reasoning was that when submitting to the Flink cluster the application 
already has these dependencies, while when a user writes a program against 
these libraries she will include the core dependencies explicitly anyway.

There is one other case of usage however, namely when someone is trying to run 
an application defined in these libraries depending on the core jars. To give 
an example if you were to run the Gelly ConnectedComponents example [2] from an 
IDE after importing Flink (or running with java -jar without including the 
flink fat jar in the classpath) you would receive the following class not found 
exception as per the current master:

Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/flink/api/common/ProgramDescription
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)

Where the class missing in runtime is in flink-core, a transitive dependency of 
the jars moved to the provided scope.

Funny thing is we have tests in place to run our examples, but those add test 
scope dependencies re-adding the missing classes, so it is never discovered.

I agree with the original purpose of PR #1683, but also think that the current 
state makes for very inconvenient user experience.

I would like to open a discussion on how and when to resolve the issue given 
the release of 1.0.0.

1. Is it a release blocker?
2. Should the change be reverted or is it sufficient to have proper 
documentation around it? Maybe a maven profile for explicitly for developing 
Flink without the provided scope?

Note that the issue was originally reported by Gábor Gévay.

[1] https://github.com/apache/flink/pull/1683
[2] 
https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3510) Pattern class class-level comment misses type argument

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-3510:
--

 Summary: Pattern class class-level comment misses type argument
 Key: FLINK-3510
 URL: https://issues.apache.org/jira/browse/FLINK-3510
 Project: Flink
  Issue Type: Bug
  Components: CEP
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi
Priority: Trivial


Class-level comment code example reads:

{code}
 * {@code
 * Pattern pattern = Pattern.begin("start")
 *   .next("middle").subtype(F.class)
 *   .followedBy("end").where(new MyFilterFunction());
 * }
 * 
{code}

The 2nd type paramter for {{Pattern}} is missing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1159) Case style anonymous functions not supported by Scala API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167209#comment-15167209
 ] 

ASF GitHub Bot commented on FLINK-1159:
---

Github user stefanobaghino commented on the pull request:

https://github.com/apache/flink/pull/1704#issuecomment-188790808
  
@StephanEwen I had to restore some of the context bounds on `ClassTag` to 
make it compile, apparently the delegated methods use them; I've rebased with 
the latest changes on the master before putting them back in place.


> Case style anonymous functions not supported by Scala API
> -
>
> Key: FLINK-1159
> URL: https://issues.apache.org/jira/browse/FLINK-1159
> Project: Flink
>  Issue Type: Bug
>  Components: Scala API
>Reporter: Till Rohrmann
>Assignee: Stefano Baghino
>
> In Scala it is very common to define anonymous functions of the following form
> {code}
> {
> case foo: Bar => foobar(foo)
> case _ => throw new RuntimeException()
> }
> {code}
> These case style anonymous functions are not supported yet by the Scala API. 
> Thus, one has to write redundant code to name the function parameter.
> What works is the following pattern, but it is not intuitive for someone 
> coming from Scala:
> {code}
> dataset.map{
>   _ match{
> case foo:Bar => ...
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1159] Case style anonymous functions no...

[stefanobaghino]

Github user stefanobaghino commented on the pull request:

https://github.com/apache/flink/pull/1704#issuecomment-188790808
  
@StephanEwen I had to restore some of the context bounds on `ClassTag` to 
make it compile, apparently the delegated methods use them; I've rebased with 
the latest changes on the master before putting them back in place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167186#comment-15167186
 ] 

Chesnay Schepler commented on FLINK-3496:
-

may a fix for this coming up, running on travis now.

> Flink-ml tests fail on Windows
> --
>
> Key: FLINK-3496
> URL: https://issues.apache.org/jira/browse/FLINK-3496
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>
> [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 
> ---
> [INFO]
> [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 ---
> The system cannot find the file specified.
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:03 min
> [INFO] Finished at: 2016-02-24T12:47:23+01:00
> [INFO] Final Memory: 28M/506M



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3332] Cassandra connector

[spdrnl]

Github user spdrnl commented on a diff in the pull request:

https://github.com/apache/flink/pull/1620#discussion_r54090368
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/GenericExactlyOnceSink.java
 ---
@@ -0,0 +1,173 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.streaming.runtime.operators;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateHandle;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTaskState;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Generic Sink that emits its input elements into an arbitrary backend. 
This sink is integrated with the checkpointing
+ * mechanism to provide exactly once semantics.
+ *
+ * Incoming records are stored within a {@link 
org.apache.flink.runtime.state.AbstractStateBackend}, and only committed if a
+ * checkpoint is completed. Should a job fail while the data is being 
committed, no exactly once guarantee can be made.
+ * @param 
+ */
+public abstract class GenericExactlyOnceSink extends 
AbstractStreamOperator implements OneInputStreamOperator {
+   private AbstractStateBackend.CheckpointStateOutputView out;
+   private TypeSerializer serializer;
+   protected TypeInformation typeInfo;
+
+   private ExactlyOnceState state = new ExactlyOnceState();
+
+   /**
+* Saves a handle in the state.
+* @param checkpointId
+* @throws IOException
+*/
+   private void saveHandleInState(final long checkpointId) throws 
IOException {
+   //only add handle if a new OperatorState was created since the 
last snapshot/notify
+   if (out != null) {
+   out.writeByte(0); //EOF-byte
+   StateHandle handle = 
out.closeAndGetHandle();
+   if (state.pendingHandles.containsKey(checkpointId)) {
+   
state.pendingHandles.get(checkpointId).add(handle);
+   } else {
+   ArrayList list = 
new ArrayList<>();
+   list.add(handle);
+   state.pendingHandles.put(checkpointId, list);
+   }
+   out = null;
+   }
+   }
+
+   @Override
+   public StreamTaskState snapshotOperatorState(final long checkpointId, 
final long timestamp) throws Exception {
+   StreamTaskState taskState = 
super.snapshotOperatorState(checkpointId, timestamp);
+   saveHandleInState(checkpointId);
+   taskState.setFunctionState(state);
+   return taskState;
+   }
+
+   @Override
+   public void restoreState(StreamTaskState state, long recoveryTimestamp) 
throws Exception {
+   super.restoreState(state, recoveryTimestamp);
+   this.state = (ExactlyOnceState) state.getFunctionState();
+   out = null;
+   }
+
+   @Override
+   public void notifyOfCompletedCheckpoint(long checkpointId) throws 
Exception {
--- End diff --

Apart from Flink specifics it is possible to get Cassandra to 

[jira] [Commented] (FLINK-3332) Provide an exactly-once Cassandra connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167166#comment-15167166
 ] 

ASF GitHub Bot commented on FLINK-3332:
---

Github user spdrnl commented on a diff in the pull request:

https://github.com/apache/flink/pull/1620#discussion_r54090368
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/GenericExactlyOnceSink.java
 ---
@@ -0,0 +1,173 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.streaming.runtime.operators;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateHandle;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTaskState;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Generic Sink that emits its input elements into an arbitrary backend. 
This sink is integrated with the checkpointing
+ * mechanism to provide exactly once semantics.
+ *
+ * Incoming records are stored within a {@link 
org.apache.flink.runtime.state.AbstractStateBackend}, and only committed if a
+ * checkpoint is completed. Should a job fail while the data is being 
committed, no exactly once guarantee can be made.
+ * @param 
+ */
+public abstract class GenericExactlyOnceSink extends 
AbstractStreamOperator implements OneInputStreamOperator {
+   private AbstractStateBackend.CheckpointStateOutputView out;
+   private TypeSerializer serializer;
+   protected TypeInformation typeInfo;
+
+   private ExactlyOnceState state = new ExactlyOnceState();
+
+   /**
+* Saves a handle in the state.
+* @param checkpointId
+* @throws IOException
+*/
+   private void saveHandleInState(final long checkpointId) throws 
IOException {
+   //only add handle if a new OperatorState was created since the 
last snapshot/notify
+   if (out != null) {
+   out.writeByte(0); //EOF-byte
+   StateHandle handle = 
out.closeAndGetHandle();
+   if (state.pendingHandles.containsKey(checkpointId)) {
+   
state.pendingHandles.get(checkpointId).add(handle);
+   } else {
+   ArrayList list = 
new ArrayList<>();
+   list.add(handle);
+   state.pendingHandles.put(checkpointId, list);
+   }
+   out = null;
+   }
+   }
+
+   @Override
+   public StreamTaskState snapshotOperatorState(final long checkpointId, 
final long timestamp) throws Exception {
+   StreamTaskState taskState = 
super.snapshotOperatorState(checkpointId, timestamp);
+   saveHandleInState(checkpointId);
+   taskState.setFunctionState(state);
+   return taskState;
+   }
+
+   @Override
+   public void restoreState(StreamTaskState state, long recoveryTimestamp) 
throws Exception {
+   super.restoreState(state, recoveryTimestamp);
+   this.state = (ExactlyOnceState) state.getFunctionState();
  

[jira] [Created] (FLINK-3509) Update Hadoop versions in release script and on travis to the latest minor version

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-3509:
-

 Summary: Update Hadoop versions in release script and on travis to 
the latest minor version
 Key: FLINK-3509
 URL: https://issues.apache.org/jira/browse/FLINK-3509
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: Robert Metzger
Assignee: Robert Metzger


The Hadoop versions in the travis.yml and the create-release-files.sh script 
are not updated to the latest Hadoop minor releases. Hadoop 2.7.2 and 2.6.4 
were released in the meantime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167115#comment-15167115
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user twalthr closed the pull request at:

https://github.com/apache/flink/pull/1679


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Translation of scalar function su...

[twalthr]

Github user twalthr closed the pull request at:

https://github.com/apache/flink/pull/1679


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1707) Add an Affinity Propagation Library Method

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167061#comment-15167061
 ] 

Josep Rubió commented on FLINK-1707:


Sorry, I did the document in sharelatex but I forgot to share it with you. 
Anyway, I updated the description adding a link to the google doc. Thanks!!

> Add an Affinity Propagation Library Method
> --
>
> Key: FLINK-1707
> URL: https://issues.apache.org/jira/browse/FLINK-1707
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Josep Rubió
>Priority: Minor
>  Labels: requires-design-doc
> Attachments: Binary_Affinity_Propagation_in_Flink_design_doc.pdf
>
>
> This issue proposes adding the an implementation of the Affinity Propagation 
> algorithm as a Gelly library method and a corresponding example.
> The algorithm is described in paper [1] and a description of a vertex-centric 
> implementation can be found is [2].
> [1]: http://www.psi.toronto.edu/affinitypropagation/FreyDueckScience07.pdf
> [2]: http://event.cwi.nl/grades2014/00-ching-slides.pdf
> Design doc:
> https://docs.google.com/document/d/1QULalzPqMVICi8jRVs3S0n39pell2ZVc7RNemz_SGA4/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)