[jira] [Assigned] (FLINK-2662) CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators."

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske reassigned FLINK-2662:


Assignee: Fabian Hueske

> CompilerException: "Bug: Plan generation for Unions picked a ship strategy 
> between binary plan operators."
> --
>
> Key: FLINK-2662
> URL: https://issues.apache.org/jira/browse/FLINK-2662
> Project: Flink
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 0.9.1, 0.10.0
>Reporter: Gabor Gevay
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: FlinkBug.scala
>
>
> I have a Flink program which throws the exception in the jira title. Full 
> text:
> Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: 
> Plan generation for Unions picked a ship strategy between binary plan 
> operators.
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.WorksetIterationPlanNode.acceptForStepFunction(WorksetIterationPlanNode.java:194)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:49)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:162)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>   at 
> org.apache.flink.client.LocalExecutor.getOptimizerPlanAsJSON(LocalExecutor.java:202)
>   at 
> org.apache.flink.api.java.LocalEnvironment.getExecutionPlan(LocalEnvironment.java:63)
>   at malom.Solver.main(Solver.java:66)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> The execution plan:
> http://compalg.inf.elte.hu/~ggevay/flink/plan_3_4_0_0_without_verif.txt
> (I obtained this by commenting out the line that throws the exception)
> The code is here:
> https://github.com/ggevay/flink/tree/plan-generation-bug
> The class to run is "Solver". It needs a command line argument, which is a 
> directory where it would write output. (On first run, it generates some 
> lookuptables for a few minutes, which are then placed to /tmp/movegen)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2662) CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators."

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497564#comment-15497564
 ] 

ASF GitHub Bot commented on FLINK-2662:
---

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/2508

[FLINK-2662] [dataSet] Translate union with multiple output into separate 
unions with single output.

Fixes FLINK-2662 by translating Union operators with two (or more) 
successors into two or more Union operators with a single successor.

In the optimizer union operators with two (or more) successors caused 
problems, when these successors had different partitioning requirements and 
some of these successors were other Union operators. In certain situations, the 
UnionMerging post pass would fail because of a non-forward shipping strategy 
between two subsequent union operators.

This fix does only adapt the program translation and does not change the 
API.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink FLINK-2662

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2508.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2508


commit 8d91e9d0074884ac430c88c4f6ad41878a8d1dff
Author: Fabian Hueske 
Date:   2016-09-16T16:40:32Z

[FLINK-2662] [dataSet] Translate union with multiple output into separate 
unions with single output.




> CompilerException: "Bug: Plan generation for Unions picked a ship strategy 
> between binary plan operators."
> --
>
> Key: FLINK-2662
> URL: https://issues.apache.org/jira/browse/FLINK-2662
> Project: Flink
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 0.9.1, 0.10.0
>Reporter: Gabor Gevay
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: FlinkBug.scala
>
>
> I have a Flink program which throws the exception in the jira title. Full 
> text:
> Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: 
> Plan generation for Unions picked a ship strategy between binary plan 
> operators.
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.WorksetIterationPlanNode.acceptForStepFunction(WorksetIterationPlanNode.java:194)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:49)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:162)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>   at 
> org.apache.flink.client.LocalExecutor.getOptimizerPlanAsJSON(LocalExecutor.java:202)
>   at 
> org.apache.flink.api.java.LocalEnvironment.getExecutionPlan(LocalEnvironment.java:63)
>   at malom.Solver.main(Solver.java:66)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> The execution plan:
> http://compalg.inf.elte.hu/~ggevay/flink/plan_3_4_0_0_without_verif.txt
> (I obtained this by commenting out the line that throws the exception)
> The code is here:
> 

[GitHub] flink pull request #2508: [FLINK-2662] [dataSet] Translate union with multip...

[fhueske]

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/2508

[FLINK-2662] [dataSet] Translate union with multiple output into separate 
unions with single output.

Fixes FLINK-2662 by translating Union operators with two (or more) 
successors into two or more Union operators with a single successor.

In the optimizer union operators with two (or more) successors caused 
problems, when these successors had different partitioning requirements and 
some of these successors were other Union operators. In certain situations, the 
UnionMerging post pass would fail because of a non-forward shipping strategy 
between two subsequent union operators.

This fix does only adapt the program translation and does not change the 
API.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink FLINK-2662

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2508.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2508


commit 8d91e9d0074884ac430c88c4f6ad41878a8d1dff
Author: Fabian Hueske 
Date:   2016-09-16T16:40:32Z

[FLINK-2662] [dataSet] Translate union with multiple output into separate 
unions with single output.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4629) Kafka v 0.10 Support

[Mariano Gonzalez (JIRA)]

Mariano Gonzalez created FLINK-4629:
---

 Summary: Kafka v 0.10 Support
 Key: FLINK-4629
 URL: https://issues.apache.org/jira/browse/FLINK-4629
 Project: Flink
  Issue Type: Wish
Reporter: Mariano Gonzalez
Priority: Minor


I couldn't find any repo or documentation about when Flink will start 
supporting Kafka v 0.10.

Is there any document that you can point me out where i can see Flink's roadmap?

Thanks in advance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496973#comment-15496973
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm I have addressed some of the review feedback and rebased to upstream 
master.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2275: FLINK-3929 Support for Kerberos Authentication with Keyta...

[vijikarthi]

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm I have addressed some of the review feedback and rebased to upstream 
master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496957#comment-15496957
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79221254
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
--- End diff --

Moved all the hard coded configuration property to static variable


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[vijikarthi]

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79221254
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
--- End diff --

Moved all the hard coded configuration property to static variable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496955#comment-15496955
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79221107
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -161,6 +161,8 @@ public CliFrontend(String configDir) throws Exception {
"filesystem scheme from configuration.", e);
}
 
+   this.config.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
configDirectory.getAbsolutePath());
--- End diff --

Good catch, though the implementation takes care of handling base directory 
and the immediate "conf" directory. Will change it to point to parent directory.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[vijikarthi]

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79221107
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -161,6 +161,8 @@ public CliFrontend(String configDir) throws Exception {
"filesystem scheme from configuration.", e);
}
 
+   this.config.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
configDirectory.getAbsolutePath());
--- End diff --

Good catch, though the implementation takes care of handling base directory 
and the immediate "conf" directory. Will change it to point to parent directory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2662) CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators."

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496768#comment-15496768
 ] 

Fabian Hueske commented on FLINK-2662:
--

I found the root cause of this problem. The plan contains a Union (U1) operator 
with two outputs of which one is another union operator (U2). One of the input 
requires a partitioning shipping strategy (U1, pushed down from a following 
distinct), the other not. In some cases, the partitioning is not pushed to the 
input of U1, such that the connection between U1 and U2 is hash-partitioned. As 
one of the last steps, the optimizer merges consecutive binary union operator 
into an n-ary union operator and checks that their connection is a simple 
forward connection, i.e., not a partitioned connection. This is where the 
translation fails.

I have a fix for this issue that translates union operators with two (or more) 
inputs into multiple internal union operators with a single output. Thereby, we 
avoid multiple competing shipping strategies for union operators and a 
potential partitioning is always pushed to the two input of a union.
Note, the fix touches only the program translation. The API does not change and 
the program does not need to be changed.

I'll open a PR with the fix soon.

> CompilerException: "Bug: Plan generation for Unions picked a ship strategy 
> between binary plan operators."
> --
>
> Key: FLINK-2662
> URL: https://issues.apache.org/jira/browse/FLINK-2662
> Project: Flink
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 0.9.1, 0.10.0
>Reporter: Gabor Gevay
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: FlinkBug.scala
>
>
> I have a Flink program which throws the exception in the jira title. Full 
> text:
> Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: 
> Plan generation for Unions picked a ship strategy between binary plan 
> operators.
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.WorksetIterationPlanNode.acceptForStepFunction(WorksetIterationPlanNode.java:194)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:49)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:162)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>   at 
> org.apache.flink.client.LocalExecutor.getOptimizerPlanAsJSON(LocalExecutor.java:202)
>   at 
> org.apache.flink.api.java.LocalEnvironment.getExecutionPlan(LocalEnvironment.java:63)
>   at malom.Solver.main(Solver.java:66)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> The execution plan:
> http://compalg.inf.elte.hu/~ggevay/flink/plan_3_4_0_0_without_verif.txt
> (I obtained this by commenting out the line that throws the exception)
> The code is here:
> https://github.com/ggevay/flink/tree/plan-generation-bug
> The class to run is "Solver". It needs a command line argument, which is a 
> directory where it would write output. (On first run, it generates some 
> lookuptables for a few minutes, which are then placed to /tmp/movegen)



--
This message was sent by Atlassian JIRA

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496555#comment-15496555
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79190433
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1233,6 +1239,9 @@
/** ZooKeeper default leader port. */
public static final int DEFAULT_ZOOKEEPER_LEADER_PORT = 3888;
 
+   /** Defaults for ZK client security **/
+   public static final boolean DEFAULT_ZOOKEEPER_SASL_DISABLE = true;
--- End diff --

I agree but it can be argued both ways. We could keep the default to false 
(enable SASL client auth if not disabled explicitly through configuration file) 
or expect an explicit ask to enable SASL through the configuration settings. I 
chose later since secure ZK is not a common deployment (mostly) and moreover we 
also have introduced new security configurations to enable security and one 
could configure/adjust ZK configuration at that time. 


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[vijikarthi]

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79190433
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1233,6 +1239,9 @@
/** ZooKeeper default leader port. */
public static final int DEFAULT_ZOOKEEPER_LEADER_PORT = 3888;
 
+   /** Defaults for ZK client security **/
+   public static final boolean DEFAULT_ZOOKEEPER_SASL_DISABLE = true;
--- End diff --

I agree but it can be argued both ways. We could keep the default to false 
(enable SASL client auth if not disabled explicitly through configuration file) 
or expect an explicit ask to enable SASL through the configuration settings. I 
chose later since secure ZK is not a common deployment (mostly) and moreover we 
also have introduced new security configurations to enable security and one 
could configure/adjust ZK configuration at that time. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496542#comment-15496542
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
>
@vijikarthi Thanks for the update. Great to see the tests are passing now. 
I'm curious, why did this issue only appear on Travis and not locally?

Kafka/ZK connection is unusually longer in Travis and I don't know the 
reason? I have noticed a similar comment in the Kafka Test code too where we 
have changed the timeout from 6 sec (default) to 30 sec. I have increased the 
timeout interval further for the secure run since occasionally it fails with 
the lower timeout settings.

The combination of longer ZK connection time and client code waiting for 
incorrect event (SysConnected) was the reason why in Travis it was failing.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2275: FLINK-3929 Support for Kerberos Authentication with Keyta...

[vijikarthi]

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
>
@vijikarthi Thanks for the update. Great to see the tests are passing now. 
I'm curious, why did this issue only appear on Travis and not locally?

Kafka/ZK connection is unusually longer in Travis and I don't know the 
reason? I have noticed a similar comment in the Kafka Test code too where we 
have changed the timeout from 6 sec (default) to 30 sec. I have increased the 
timeout interval further for the secure run since occasionally it fails with 
the lower timeout settings.

The combination of longer ZK connection time and client code waiting for 
incorrect event (SysConnected) was the reason why in Travis it was failing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4081) FieldParsers should support empty strings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496347#comment-15496347
 ] 

ASF GitHub Bot commented on FLINK-4081:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2297
  
If there are no objections, I would like to merge this PR next week. Other 
my other PRs depend on this.


> FieldParsers should support empty strings
> -
>
> Key: FLINK-4081
> URL: https://issues.apache.org/jira/browse/FLINK-4081
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Reporter: Flavio Pompermaier
>Assignee: Timo Walther
>  Labels: csvparser, table-api
>
> In order to parse CSV files using the new Table API that converts rows to Row 
> objects (that support null values), FiledParser implementations should 
> support emptry strings setting the parser state to 
> ParseErrorState.EMPTY_STRING (for example FloatParser and DoubleParser 
> doesn't respect this constraint)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2297: [FLINK-4081] [core] [table] FieldParsers should support e...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2297
  
If there are no objections, I would like to merge this PR next week. Other 
my other PRs depend on this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-4288) Make it possible to unregister tables

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther reassigned FLINK-4288:
---

Assignee: Timo Walther

> Make it possible to unregister tables
> -
>
> Key: FLINK-4288
> URL: https://issues.apache.org/jira/browse/FLINK-4288
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Table names can not be changed yet. After registration you can not modify the 
> table behind a table name. Maybe this behavior is too restrictive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4393) Failed to serialize accumulators for task

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther closed FLINK-4393.
---
Resolution: Won't Fix

This issue does not describe a bug.

> Failed to serialize accumulators for task
> -
>
> Key: FLINK-4393
> URL: https://issues.apache.org/jira/browse/FLINK-4393
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
> Environment: Redhat 6
>Reporter: Sajeev Ramakrishnan
>
> Dear Team,
>   I am getting the below exception while trying to use the Table API by 
> looping through the DataSet using collect() method.
> {code}
> 2016-08-15 07:18:52,503 WARN  
> org.apache.flink.runtime.accumulators.AccumulatorRegistry - Failed to 
> serialize accumulators for task.
> java.lang.OutOfMemoryError
> at 
> java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
> at 
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at 
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> at 
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at 
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
> at 
> org.apache.flink.util.SerializedValue.(SerializedValue.java:52)
> at 
> org.apache.flink.runtime.accumulators.AccumulatorSnapshot.(AccumulatorSnapshot.java:58)
> at 
> org.apache.flink.runtime.accumulators.AccumulatorRegistry.getSnapshot(AccumulatorRegistry.java:75)
> at 
> org.apache.flink.runtime.taskmanager.TaskManager.unregisterTaskAndNotifyFinalState(TaskManager.scala:1248)
> at 
> org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleTaskMessage(TaskManager.scala:446)
> at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:292)
> at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
> at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
> at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
> at 
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
> at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
> at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
> at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
> at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
> at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
> at 
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
> at 
> org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:124)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Suppressed: java.lang.OutOfMemoryError
> at 
> java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
> at 
> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
> at 
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at 
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at 
> 

[jira] [Commented] (FLINK-4252) Table program cannot be compiled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496227#comment-15496227
 ] 

ASF GitHub Bot commented on FLINK-4252:
---

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2507

[FLINK-4252] [table] Validate input and output classes of Table API

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR adds some pre-checking of input and output types of the Table API.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4252

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2507


commit d4f778a3a569d4e0e381c37c0472eec93338c806
Author: twalthr 
Date:   2016-09-16T12:27:47Z

[FLINK-4252] [table] Validate input and output classes of Table API




> Table program cannot be compiled
> 
>
> Key: FLINK-4252
> URL: https://issues.apache.org/jira/browse/FLINK-4252
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
> Environment: OS X EI Captain
> scala 2.11.7
> jdk 8
>Reporter: Renkai Ge
>Assignee: Timo Walther
> Attachments: TestMain.scala
>
>
> I'm trying the table apis.
> I got some errors like this
> My code is in the attachments
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed.
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:413)
>   at 
> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:92)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:389)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:376)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:61)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
>   at org.apache.flink.api.java.DataSet.collect(DataSet.java:410)
>   at org.apache.flink.api.java.DataSet.print(DataSet.java:1605)
>   at org.apache.flink.api.scala.DataSet.print(DataSet.scala:1672)
>   at TestMain$.main(TestMain.scala:31)
>   at TestMain.main(TestMain.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:331)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
>   at 
> 

[GitHub] flink pull request #2507: [FLINK-4252] [table] Validate input and output cla...

[twalthr]

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2507

[FLINK-4252] [table] Validate input and output classes of Table API

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR adds some pre-checking of input and output types of the Table API.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4252

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2507


commit d4f778a3a569d4e0e381c37c0472eec93338c806
Author: twalthr 
Date:   2016-09-16T12:27:47Z

[FLINK-4252] [table] Validate input and output classes of Table API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4084) Add configDir parameter to CliFrontend and flink shell script

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496185#comment-15496185
 ] 

ASF GitHub Bot commented on FLINK-4084:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2149#discussion_r79157764
  
--- Diff: docs/apis/cli.md ---
@@ -187,6 +187,8 @@ Action "run" compiles and runs a program.
   java.net.URLClassLoader}.
  -d,--detachedIf present, runs the job in 
detached
   mode
+--configDir The configuration directory with 
which
--- End diff --

Actually, it should be aligned and `configDir` is not part of the run 
options.


> Add configDir parameter to CliFrontend and flink shell script
> -
>
> Key: FLINK-4084
> URL: https://issues.apache.org/jira/browse/FLINK-4084
> Project: Flink
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Andrea Sella
>Priority: Minor
>
> At the moment there is no other way than the environment variable 
> FLINK_CONF_DIR to specify the configuration directory for the CliFrontend if 
> it is started via the flink shell script. In order to improve the user 
> exprience, I would propose to introduce a {{--configDir}} parameter which the 
> user can use to specify a configuration directory more easily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2149: [FLINK-4084] Add configDir parameter to CliFronten...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2149#discussion_r79157764
  
--- Diff: docs/apis/cli.md ---
@@ -187,6 +187,8 @@ Action "run" compiles and runs a program.
   java.net.URLClassLoader}.
  -d,--detachedIf present, runs the job in 
detached
   mode
+--configDir The configuration directory with 
which
--- End diff --

Actually, it should be aligned and `configDir` is not part of the run 
options.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4084) Add configDir parameter to CliFrontend and flink shell script

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496180#comment-15496180
 ] 

ASF GitHub Bot commented on FLINK-4084:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2149#discussion_r79157293
  
--- Diff: docs/apis/cli.md ---
@@ -187,6 +187,8 @@ Action "run" compiles and runs a program.
   java.net.URLClassLoader}.
  -d,--detachedIf present, runs the job in 
detached
   mode
+--configDir The configuration directory with 
which
--- End diff --

Ah ok. Looks good. Thanks!


> Add configDir parameter to CliFrontend and flink shell script
> -
>
> Key: FLINK-4084
> URL: https://issues.apache.org/jira/browse/FLINK-4084
> Project: Flink
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Andrea Sella
>Priority: Minor
>
> At the moment there is no other way than the environment variable 
> FLINK_CONF_DIR to specify the configuration directory for the CliFrontend if 
> it is started via the flink shell script. In order to improve the user 
> exprience, I would propose to introduce a {{--configDir}} parameter which the 
> user can use to specify a configuration directory more easily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4084) Add configDir parameter to CliFrontend and flink shell script

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496179#comment-15496179
 ] 

ASF GitHub Bot commented on FLINK-4084:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2149#discussion_r79156877
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java 
---
@@ -241,8 +255,11 @@ private static Options 
getSavepointOptionsWithoutDeprecatedOptions(Options optio
 * Prints the help for the client.
 */
public static void printHelp() {
-   System.out.println("./flink  [OPTIONS] [ARGUMENTS]");
+   System.out.println("./flink [CONFIGDIR]  
[ACTION-OPTIONS] [ARGUMENTS]");
--- End diff --

Shouldn't this be `./flink [--configDir ]  
[ACTION-OPTIONS] [ARGUMENTS]`?


> Add configDir parameter to CliFrontend and flink shell script
> -
>
> Key: FLINK-4084
> URL: https://issues.apache.org/jira/browse/FLINK-4084
> Project: Flink
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Andrea Sella
>Priority: Minor
>
> At the moment there is no other way than the environment variable 
> FLINK_CONF_DIR to specify the configuration directory for the CliFrontend if 
> it is started via the flink shell script. In order to improve the user 
> exprience, I would propose to introduce a {{--configDir}} parameter which the 
> user can use to specify a configuration directory more easily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2149: [FLINK-4084] Add configDir parameter to CliFronten...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2149#discussion_r79157293
  
--- Diff: docs/apis/cli.md ---
@@ -187,6 +187,8 @@ Action "run" compiles and runs a program.
   java.net.URLClassLoader}.
  -d,--detachedIf present, runs the job in 
detached
   mode
+--configDir The configuration directory with 
which
--- End diff --

Ah ok. Looks good. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2149: [FLINK-4084] Add configDir parameter to CliFronten...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2149#discussion_r79156877
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java 
---
@@ -241,8 +255,11 @@ private static Options 
getSavepointOptionsWithoutDeprecatedOptions(Options optio
 * Prints the help for the client.
 */
public static void printHelp() {
-   System.out.println("./flink  [OPTIONS] [ARGUMENTS]");
+   System.out.println("./flink [CONFIGDIR]  
[ACTION-OPTIONS] [ARGUMENTS]");
--- End diff --

Shouldn't this be `./flink [--configDir ]  
[ACTION-OPTIONS] [ARGUMENTS]`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4622) CLI help message should include 'savepoint' action

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496172#comment-15496172
 ] 

ASF GitHub Bot commented on FLINK-4622:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2501


> CLI help message should include 'savepoint' action
> --
>
> Key: FLINK-4622
> URL: https://issues.apache.org/jira/browse/FLINK-4622
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Scott Kidder
>Assignee: Scott Kidder
>Priority: Trivial
> Fix For: 1.2.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The Flink CLI help message should include the 'savepoint' action in the list 
> of available actions. It currently looks like:
> {code}
> bash-4.3# flink foo
> "foo" is not a valid action.
> Valid actions are "run", "list", "info", "stop", or "cancel".
> Specify the version option (-v or --version) to print Flink version.
> Specify the help option (-h or --help) to get help on the command.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4622) CLI help message should include 'savepoint' action

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4622.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

Implemented in d7b59d761601baba6765bb4fc407bcd9fd6a9387

> CLI help message should include 'savepoint' action
> --
>
> Key: FLINK-4622
> URL: https://issues.apache.org/jira/browse/FLINK-4622
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Scott Kidder
>Assignee: Scott Kidder
>Priority: Trivial
> Fix For: 1.2.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The Flink CLI help message should include the 'savepoint' action in the list 
> of available actions. It currently looks like:
> {code}
> bash-4.3# flink foo
> "foo" is not a valid action.
> Valid actions are "run", "list", "info", "stop", or "cancel".
> Specify the version option (-v or --version) to print Flink version.
> Specify the help option (-h or --help) to get help on the command.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2501: [FLINK-4622] CLI help message should include 'save...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2501


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2501: [FLINK-4622] CLI help message should include 'savepoint' ...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2501
  
merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4622) CLI help message should include 'savepoint' action

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496164#comment-15496164
 ] 

ASF GitHub Bot commented on FLINK-4622:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2501
  
merging


> CLI help message should include 'savepoint' action
> --
>
> Key: FLINK-4622
> URL: https://issues.apache.org/jira/browse/FLINK-4622
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Scott Kidder
>Assignee: Scott Kidder
>Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The Flink CLI help message should include the 'savepoint' action in the list 
> of available actions. It currently looks like:
> {code}
> bash-4.3# flink foo
> "foo" is not a valid action.
> Valid actions are "run", "list", "info", "stop", or "cancel".
> Specify the version option (-v or --version) to print Flink version.
> Specify the help option (-h or --help) to get help on the command.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496158#comment-15496158
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79156099
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1233,6 +1239,9 @@
/** ZooKeeper default leader port. */
public static final int DEFAULT_ZOOKEEPER_LEADER_PORT = 3888;
 
+   /** Defaults for ZK client security **/
+   public static final boolean DEFAULT_ZOOKEEPER_SASL_DISABLE = true;
--- End diff --

I'm wondering, shouldn't the default be true? If I configure Jaas, then I 
probably want to connect to a secured Zookeeper by default.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496153#comment-15496153
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153742
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -369,6 +385,11 @@ public Properties getSecureProperties() {
prop.put("security.inter.broker.protocol", 
"SASL_PLAINTEXT");
prop.put("security.protocol", "SASL_PLAINTEXT");
prop.put("sasl.kerberos.service.name", "kafka");
+
+   //add special timeout for Travis
+   prop.setProperty("zookeeper.session.timeout.ms", 
zkTimeout);
+   prop.setProperty("zookeeper.connection.timeout.ms", 
zkTimeout);
+   prop.setProperty("metadata.fetch.timeout.ms","12");
--- End diff --

That timeout seems very high.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496157#comment-15496157
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153795
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -141,9 +141,11 @@ public boolean isSecureRunSupported() {
@Override
public void prepare(int numKafkaServers, Properties 
additionalServerProperties, boolean secureMode) {
 
-   //increase the timeout since in Travis it is failing.
+   //increase the timeout since in Travis ZK connection takes long 
time for secure connection.
if(secureMode) {
-   zkTimeout = "27";
+   //run only one kafka server to avoid multiple ZK 
connections from many instances - Travis timeout
+   numKafkaServers = 1;
+   zkTimeout = String.valueOf(Integer.parseInt(zkTimeout) 
* 15);
--- End diff --

The timeout here seems arbitrary.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496154#comment-15496154
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79155885
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -158,22 +157,37 @@ public static void install(SecurityConfiguration 
config) throws Exception {
}
 
/*
-* This is a temporary fix to support both Kafka and ZK client libraries
-* that are expecting the system variable to determine secure cluster
+* This method configures some of the system property that are require 
for ZK and Kafka SASL authentication
+* See: 
https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+* See: 
https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L900
+* In this method, setting java.security.auth.login.config 
configuration is temporary hack only to support ZK and
+* Kafka current code behavior
 */
-   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+   private static void populateSystemSecurityProperties(Configuration 
configuration) {
 
-   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
-   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
-   if(null == configuration) {
-   System.setProperty("java.security.auth.login.config", 
"");
+   //required to be empty for Kafka but we will override the 
property
+   //with pseudo JAAS configuration file if SASL auth is enabled 
for ZK
+   System.setProperty("java.security.auth.login.config", "");
+
+   if(configuration == null) {
+   return;
+   }
+
+   boolean disableSaslClient = 
configuration.getBoolean(ConfigConstants.ZOOKEEPER_SASL_DISABLE,
+   ConfigConstants.DEFAULT_ZOOKEEPER_SASL_DISABLE);
+   if(disableSaslClient) {
+   LOG.info("SASL client auth for ZK will be disabled");
+   //SASL auth is disabled by default but will be enabled 
if specified in configuration
+   System.setProperty("zookeeper.sasl.client","false");
--- End diff --

This is to disable SASL with Zookeeper if a Jaas configuration is used but 
Zookeeper is not configured to use it?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496150#comment-15496150
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79154968
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
+   return;
+   }
+
+   String baseDir = 
configuration.getString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, null);
+   if(baseDir == null) {
+   System.setProperty("java.security.auth.login.config", 
"");
+   return;
+   }
+
+   File f = new File(baseDir);
+   if(!f.exists() || !f.isDirectory()) {
+   LOG.error("Invalid flink base directory {} 
configuration provided", baseDir);
+   throw new IllegalConfigurationException("Invalid flink 
base directory configuration provided");
+   }
--- End diff --

This check should really be somewhere else. I think you can use 
`ENV_FLINK_CONF_DIR` which is always set. The `FLINK_BASE_DIR_PATH_KEY` should 
be deprecated. It's not used anymore.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496155#comment-15496155
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79152195
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
--- End diff --

Could we factor out `"java.security.auth.login.config"` to a static 
variable?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496156#comment-15496156
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79151542
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -161,6 +161,8 @@ public CliFrontend(String configDir) throws Exception {
"filesystem scheme from configuration.", e);
}
 
+   this.config.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
configDirectory.getAbsolutePath());
--- End diff --

You're setting the base dir to the config directory here. Why?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496151#comment-15496151
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79154987
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
 ---
@@ -1585,6 +1585,8 @@ object TaskManager {
   }
 }
 
+conf.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
cliConfig.getConfigDir() + "/..")
--- End diff --

Here you're setting the base dir path to the root directory instead of the 
config directory.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496149#comment-15496149
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79152436
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -79,9 +82,8 @@ public static void install(SecurityConfiguration config) 
throws Exception {
JaasConfiguration jaasConfig = new 
JaasConfiguration(config.keytab, config.principal);

javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
 
-   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
-   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
-   System.setProperty("java.security.auth.login.config", "");
+   //temporary fix
+   populateJaasConfigSystemProperty(config.flinkConf);
--- End diff --

From my experience comments like "temporary fix" tend to stay forever. I 
would replace it with something that explains the reasoning. That helps to fix 
the code when it is reviewed later on.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496152#comment-15496152
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153661
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -266,23 +268,37 @@ public void createTestTopic(String topic, int 
numberOfPartitions, int replicatio
zkUtils.close();
}
 
+   LOG.info("Topic {} create request is successfully posted", 
topic);
+
// validate that the topic has been created
-   final long deadline = System.currentTimeMillis() + 3;
+   final long deadline = System.currentTimeMillis() + 
Integer.parseInt(zkTimeout);
do {
try {
-   Thread.sleep(100);
+   if(secureMode) {
+   //increase wait time since in Travis ZK 
timeout occurs frequently
+   int wait = Integer.parseInt(zkTimeout) 
/ 100;
+   LOG.info("waiting for {} msecs before 
the topic {} can be checked", wait, topic);
+   Thread.sleep(wait);
+   } else {
+   Thread.sleep(100);
--- End diff --

I would rather retry here instead of waiting.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496148#comment-15496148
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79151877
  
--- Diff: flink-dist/src/main/flink-bin/conf/flink-jaas.conf ---
@@ -0,0 +1,26 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+# We are using this file as an workaround for the Kafka and ZK SASL 
implementation
+# since they explicitly look for java.security.auth.login.config 
environment variable
--- End diff --

In the code link it looks like a Java property and not an environment 
variable: 
https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L900


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153661
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -266,23 +268,37 @@ public void createTestTopic(String topic, int 
numberOfPartitions, int replicatio
zkUtils.close();
}
 
+   LOG.info("Topic {} create request is successfully posted", 
topic);
+
// validate that the topic has been created
-   final long deadline = System.currentTimeMillis() + 3;
+   final long deadline = System.currentTimeMillis() + 
Integer.parseInt(zkTimeout);
do {
try {
-   Thread.sleep(100);
+   if(secureMode) {
+   //increase wait time since in Travis ZK 
timeout occurs frequently
+   int wait = Integer.parseInt(zkTimeout) 
/ 100;
+   LOG.info("waiting for {} msecs before 
the topic {} can be checked", wait, topic);
+   Thread.sleep(wait);
+   } else {
+   Thread.sleep(100);
--- End diff --

I would rather retry here instead of waiting.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79155885
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -158,22 +157,37 @@ public static void install(SecurityConfiguration 
config) throws Exception {
}
 
/*
-* This is a temporary fix to support both Kafka and ZK client libraries
-* that are expecting the system variable to determine secure cluster
+* This method configures some of the system property that are require 
for ZK and Kafka SASL authentication
+* See: 
https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+* See: 
https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L900
+* In this method, setting java.security.auth.login.config 
configuration is temporary hack only to support ZK and
+* Kafka current code behavior
 */
-   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+   private static void populateSystemSecurityProperties(Configuration 
configuration) {
 
-   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
-   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
-   if(null == configuration) {
-   System.setProperty("java.security.auth.login.config", 
"");
+   //required to be empty for Kafka but we will override the 
property
+   //with pseudo JAAS configuration file if SASL auth is enabled 
for ZK
+   System.setProperty("java.security.auth.login.config", "");
+
+   if(configuration == null) {
+   return;
+   }
+
+   boolean disableSaslClient = 
configuration.getBoolean(ConfigConstants.ZOOKEEPER_SASL_DISABLE,
+   ConfigConstants.DEFAULT_ZOOKEEPER_SASL_DISABLE);
+   if(disableSaslClient) {
+   LOG.info("SASL client auth for ZK will be disabled");
+   //SASL auth is disabled by default but will be enabled 
if specified in configuration
+   System.setProperty("zookeeper.sasl.client","false");
--- End diff --

This is to disable SASL with Zookeeper if a Jaas configuration is used but 
Zookeeper is not configured to use it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153742
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -369,6 +385,11 @@ public Properties getSecureProperties() {
prop.put("security.inter.broker.protocol", 
"SASL_PLAINTEXT");
prop.put("security.protocol", "SASL_PLAINTEXT");
prop.put("sasl.kerberos.service.name", "kafka");
+
+   //add special timeout for Travis
+   prop.setProperty("zookeeper.session.timeout.ms", 
zkTimeout);
+   prop.setProperty("zookeeper.connection.timeout.ms", 
zkTimeout);
+   prop.setProperty("metadata.fetch.timeout.ms","12");
--- End diff --

That timeout seems very high.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153795
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -141,9 +141,11 @@ public boolean isSecureRunSupported() {
@Override
public void prepare(int numKafkaServers, Properties 
additionalServerProperties, boolean secureMode) {
 
-   //increase the timeout since in Travis it is failing.
+   //increase the timeout since in Travis ZK connection takes long 
time for secure connection.
if(secureMode) {
-   zkTimeout = "27";
+   //run only one kafka server to avoid multiple ZK 
connections from many instances - Travis timeout
+   numKafkaServers = 1;
+   zkTimeout = String.valueOf(Integer.parseInt(zkTimeout) 
* 15);
--- End diff --

The timeout here seems arbitrary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79152436
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -79,9 +82,8 @@ public static void install(SecurityConfiguration config) 
throws Exception {
JaasConfiguration jaasConfig = new 
JaasConfiguration(config.keytab, config.principal);

javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
 
-   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
-   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
-   System.setProperty("java.security.auth.login.config", "");
+   //temporary fix
+   populateJaasConfigSystemProperty(config.flinkConf);
--- End diff --

From my experience comments like "temporary fix" tend to stay forever. I 
would replace it with something that explains the reasoning. That helps to fix 
the code when it is reviewed later on.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79152195
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
--- End diff --

Could we factor out `"java.security.auth.login.config"` to a static 
variable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79154968
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
+   return;
+   }
+
+   String baseDir = 
configuration.getString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, null);
+   if(baseDir == null) {
+   System.setProperty("java.security.auth.login.config", 
"");
+   return;
+   }
+
+   File f = new File(baseDir);
+   if(!f.exists() || !f.isDirectory()) {
+   LOG.error("Invalid flink base directory {} 
configuration provided", baseDir);
+   throw new IllegalConfigurationException("Invalid flink 
base directory configuration provided");
+   }
--- End diff --

This check should really be somewhere else. I think you can use 
`ENV_FLINK_CONF_DIR` which is always set. The `FLINK_BASE_DIR_PATH_KEY` should 
be deprecated. It's not used anymore.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79156099
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1233,6 +1239,9 @@
/** ZooKeeper default leader port. */
public static final int DEFAULT_ZOOKEEPER_LEADER_PORT = 3888;
 
+   /** Defaults for ZK client security **/
+   public static final boolean DEFAULT_ZOOKEEPER_SASL_DISABLE = true;
--- End diff --

I'm wondering, shouldn't the default be true? If I configure Jaas, then I 
probably want to connect to a secured Zookeeper by default.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79151542
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -161,6 +161,8 @@ public CliFrontend(String configDir) throws Exception {
"filesystem scheme from configuration.", e);
}
 
+   this.config.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
configDirectory.getAbsolutePath());
--- End diff --

You're setting the base dir to the config directory here. Why?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79151877
  
--- Diff: flink-dist/src/main/flink-bin/conf/flink-jaas.conf ---
@@ -0,0 +1,26 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+# We are using this file as an workaround for the Kafka and ZK SASL 
implementation
+# since they explicitly look for java.security.auth.login.config 
environment variable
--- End diff --

In the code link it looks like a Java property and not an environment 
variable: 
https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L900


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2275: FLINK-3929 Support for Kerberos Authentication wit...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79154987
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
 ---
@@ -1585,6 +1585,8 @@ object TaskManager {
   }
 }
 
+conf.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
cliConfig.getConfigDir() + "/..")
--- End diff --

Here you're setting the base dir path to the root directory instead of the 
config directory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496069#comment-15496069
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
@vijikarthi Thanks for the update. Great to see the tests are passing now. 
I'm curious, why did this issue only appear on Travis and not locally?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2275: FLINK-3929 Support for Kerberos Authentication with Keyta...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
@vijikarthi Thanks for the update. Great to see the tests are passing now. 
I'm curious, why did this issue only appear on Travis and not locally?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-4252) Table program cannot be compiled

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther reassigned FLINK-4252:
---

Assignee: Timo Walther

> Table program cannot be compiled
> 
>
> Key: FLINK-4252
> URL: https://issues.apache.org/jira/browse/FLINK-4252
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
> Environment: OS X EI Captain
> scala 2.11.7
> jdk 8
>Reporter: Renkai Ge
>Assignee: Timo Walther
> Attachments: TestMain.scala
>
>
> I'm trying the table apis.
> I got some errors like this
> My code is in the attachments
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed.
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:413)
>   at 
> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:92)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:389)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:376)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:61)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
>   at org.apache.flink.api.java.DataSet.collect(DataSet.java:410)
>   at org.apache.flink.api.java.DataSet.print(DataSet.java:1605)
>   at org.apache.flink.api.scala.DataSet.print(DataSet.scala:1672)
>   at TestMain$.main(TestMain.scala:31)
>   at TestMain.main(TestMain.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:331)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:853)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:799)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:799)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.Exception: The user defined 'open(Configuration)' method 
> in class org.apache.flink.api.table.runtime.FlatMapRunner caused an 
> exception: Table program cannot be compiled. This is a bug. Please file an 
> issue.
>   at 
> org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1337)
>   at 
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.openTask(ChainedFlatMapDriver.java:47)
>   at 
> org.apache.flink.runtime.operators.BatchTask.openChainedTasks(BatchTask.java:1377)
>   at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:471)
>   at 
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:351)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>

[jira] [Commented] (FLINK-4581) Table API throws "No suitable driver found for jdbc:calcite"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496007#comment-15496007
 ] 

ASF GitHub Bot commented on FLINK-4581:
---

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2506

[FLINK-4581] [table] Table API throws "No suitable driver found for 
jdbc:calcite"

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR simplifies the planner creation. A dummy JDBC connection which also 
caused problems for a user is not necessary anymore.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4581

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2506.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2506


commit d3b0a7a0209526ece85bd27618a6400829b253d0
Author: twalthr 
Date:   2016-09-16T09:41:15Z

[FLINK-4581] [table] Table API throws "No suitable driver found for 
jdbc:calcite"




> Table API throws "No suitable driver found for jdbc:calcite"
> 
>
> Key: FLINK-4581
> URL: https://issues.apache.org/jira/browse/FLINK-4581
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> It seems that in certain cases the internal Calcite JDBC driver cannot be 
> found. We should either try to get rid of the entire JDBC invocation or fix 
> this bug.
> From ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Stream-sql-query-in-Flink-tp8928.html
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
>   at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:524)
>   at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:331)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253)
>   at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
> Caused by: java.lang.RuntimeException: java.sql.SQLException: No suitable
> driver found for jdbc:calcite:
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:151)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:127)
>   at
> org.apache.flink.api.table.FlinkRelBuilder$.create(FlinkRelBuilder.scala:56)
>   at
> org.apache.flink.api.table.TableEnvironment.(TableEnvironment.scala:73)
>   at
> org.apache.flink.api.table.StreamTableEnvironment.(StreamTableEnvironment.scala:58)
>   at
> org.apache.flink.api.java.table.StreamTableEnvironment.(StreamTableEnvironment.scala:45)
>   at
> org.apache.flink.api.table.TableEnvironment$.getTableEnvironment(TableEnvironment.scala:376)
>   at
> org.apache.flink.api.table.TableEnvironment.getTableEnvironment(TableEnvironment.scala)
>   at 
> org.myorg.quickstart.ReadingFromKafka2.main(ReadingFromKafka2.java:48)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at
> 

[GitHub] flink pull request #2506: [FLINK-4581] [table] Table API throws "No suitable...

[twalthr]

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2506

[FLINK-4581] [table] Table API throws "No suitable driver found for 
jdbc:calcite"

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR simplifies the planner creation. A dummy JDBC connection which also 
caused problems for a user is not necessary anymore.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4581

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2506.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2506


commit d3b0a7a0209526ece85bd27618a6400829b253d0
Author: twalthr 
Date:   2016-09-16T09:41:15Z

[FLINK-4581] [table] Table API throws "No suitable driver found for 
jdbc:calcite"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-4608) Use short-circuit AND in Max/Min AggregationFunction

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4608.
---
Resolution: Fixed

> Use short-circuit AND in Max/Min AggregationFunction
> 
>
> Key: FLINK-4608
> URL: https://issues.apache.org/jira/browse/FLINK-4608
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> Max/Min AggregationFunction use & instead of &&. Usually we use short-circuit 
> logic in if operators in java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (FLINK-4608) Use short-circuit AND in Max/Min AggregationFunction

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reopened FLINK-4608:
-

> Use short-circuit AND in Max/Min AggregationFunction
> 
>
> Key: FLINK-4608
> URL: https://issues.apache.org/jira/browse/FLINK-4608
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> Max/Min AggregationFunction use & instead of &&. Usually we use short-circuit 
> logic in if operators in java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4607) Close FileInputStream in ParameterTool and other

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4607.
---
Resolution: Fixed

Fixed in db90580ffdb93a55a6318b18b5d50ec3666b001b

> Close FileInputStream in ParameterTool and other
> 
>
> Key: FLINK-4607
> URL: https://issues.apache.org/jira/browse/FLINK-4607
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> ParameterTool and some tests do not close FileInputStream
> {code}
> flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java
> flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java
> flink-java/src/test/java/org/apache/flink/api/java/utils/ParameterToolTest.java
> flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java
> flink-runtime/src/test/java/org/apache/flink/runtime/util/JarFileCreatorTest.java
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4608) Use short-circuit AND in Max/Min AggregationFunction

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495983#comment-15495983
 ] 

Chesnay Schepler edited comment on FLINK-4608 at 9/16/16 10:38 AM:
---

Fixed in 9046374c6b5b35589d90416725c1e8eb09843bea


was (Author: zentol):
Fixed in ceb3aa05bae3d05b9f6c5a7a55e6e43fc91a9450

> Use short-circuit AND in Max/Min AggregationFunction
> 
>
> Key: FLINK-4608
> URL: https://issues.apache.org/jira/browse/FLINK-4608
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> Max/Min AggregationFunction use & instead of &&. Usually we use short-circuit 
> logic in if operators in java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4607) Close FileInputStream in ParameterTool and other

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495980#comment-15495980
 ] 

ASF GitHub Bot commented on FLINK-4607:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2488


> Close FileInputStream in ParameterTool and other
> 
>
> Key: FLINK-4607
> URL: https://issues.apache.org/jira/browse/FLINK-4607
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> ParameterTool and some tests do not close FileInputStream
> {code}
> flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java
> flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java
> flink-java/src/test/java/org/apache/flink/api/java/utils/ParameterToolTest.java
> flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java
> flink-runtime/src/test/java/org/apache/flink/runtime/util/JarFileCreatorTest.java
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4612) Close FileWriter using try with resources

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4612.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

Fixed in f06930bcf0d3c2a840cdc5a2e6e5f4b1d03f45d2

> Close FileWriter using try with resources
> -
>
> Key: FLINK-4612
> URL: https://issues.apache.org/jira/browse/FLINK-4612
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
> Fix For: 1.2.0
>
>
> FileWriter is not closed properly in many places in the project modules



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4608) Use short-circuit AND in Max/Min AggregationFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495979#comment-15495979
 ] 

ASF GitHub Bot commented on FLINK-4608:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2489


> Use short-circuit AND in Max/Min AggregationFunction
> 
>
> Key: FLINK-4608
> URL: https://issues.apache.org/jira/browse/FLINK-4608
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> Max/Min AggregationFunction use & instead of &&. Usually we use short-circuit 
> logic in if operators in java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4608) Use short-circuit AND in Max/Min AggregationFunction

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4608.
---
Resolution: Fixed

Fixed in ceb3aa05bae3d05b9f6c5a7a55e6e43fc91a9450

> Use short-circuit AND in Max/Min AggregationFunction
> 
>
> Key: FLINK-4608
> URL: https://issues.apache.org/jira/browse/FLINK-4608
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> Max/Min AggregationFunction use & instead of &&. Usually we use short-circuit 
> logic in if operators in java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4612) Close FileWriter using try with resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495978#comment-15495978
 ] 

ASF GitHub Bot commented on FLINK-4612:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2492


> Close FileWriter using try with resources
> -
>
> Key: FLINK-4612
> URL: https://issues.apache.org/jira/browse/FLINK-4612
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
> Fix For: 1.2.0
>
>
> FileWriter is not closed properly in many places in the project modules



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2488: [FLINK-4607] Close FileInputStream in ParameterToo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2488


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-4626) Missing break in MetricStore#add()

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4626.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

Fixed in 1586fc8cfc6f41244e46b27ad25dcb6b08b7ce43

> Missing break in MetricStore#add()
> --
>
> Key: FLINK-4626
> URL: https://issues.apache.org/jira/browse/FLINK-4626
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
> Fix For: 1.2.0
>
>
> {code}
>   switch (info.getCategory()) {
> case INFO_CATEGORY_JM:
>   addMetric(jobManager.metrics, name, metric);
> case INFO_CATEGORY_TM:
>   String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) 
> info).taskManagerID;
> {code}
> Looks like a break is missing following addMetric(jobManager.metrics) call



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2489: [FLINK-4608] Use short-circuit AND in Max/Min Aggr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2489


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2492: [FLINK-4612] Close FileWriter using try with resou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2492


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4612) Close FileWriter using try with resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495962#comment-15495962
 ] 

ASF GitHub Bot commented on FLINK-4612:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2492
  
merging


> Close FileWriter using try with resources
> -
>
> Key: FLINK-4612
> URL: https://issues.apache.org/jira/browse/FLINK-4612
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> FileWriter is not closed properly in many places in the project modules



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2492: [FLINK-4612] Close FileWriter using try with resources

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2492
  
merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4628) User class loader unavailable during input split assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495956#comment-15495956
 ] 

ASF GitHub Bot commented on FLINK-4628:
---

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2505

[FLINK-4628] provide user class loader during input split assignment

In analogy to the configure() method, this also sets a context class loader 
during input split assignment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-4628

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2505


commit 62d4075f89f96c1d85ca951c7705fd15e3e4
Author: Maximilian Michels 
Date:   2016-09-16T10:21:54Z

[FLINK-4628] provide user class loader during input split assignment

In analogy to the configure() method, this also sets a context class
loader during input split assignment.




> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the 
> executing thread.
> In 2 we only have the system class loader available. If any classes need to 
> be loaded then, we have a problem. Some InputFormats rely on code which 
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the 
> latest master.
> We should make the user class loader available when creating input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4626) Missing break in MetricStore#add()

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-4626:

Affects Version/s: 1.2.0

> Missing break in MetricStore#add()
> --
>
> Key: FLINK-4626
> URL: https://issues.apache.org/jira/browse/FLINK-4626
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>
> {code}
>   switch (info.getCategory()) {
> case INFO_CATEGORY_JM:
>   addMetric(jobManager.metrics, name, metric);
> case INFO_CATEGORY_TM:
>   String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) 
> info).taskManagerID;
> {code}
> Looks like a break is missing following addMetric(jobManager.metrics) call



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4626) Missing break in MetricStore#add()

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-4626:
---

Assignee: Chesnay Schepler

> Missing break in MetricStore#add()
> --
>
> Key: FLINK-4626
> URL: https://issues.apache.org/jira/browse/FLINK-4626
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>
> {code}
>   switch (info.getCategory()) {
> case INFO_CATEGORY_JM:
>   addMetric(jobManager.metrics, name, metric);
> case INFO_CATEGORY_TM:
>   String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) 
> info).taskManagerID;
> {code}
> Looks like a break is missing following addMetric(jobManager.metrics) call



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4626) Missing break in MetricStore#add()

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-4626:

Component/s: Metrics

> Missing break in MetricStore#add()
> --
>
> Key: FLINK-4626
> URL: https://issues.apache.org/jira/browse/FLINK-4626
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>
> {code}
>   switch (info.getCategory()) {
> case INFO_CATEGORY_JM:
>   addMetric(jobManager.metrics, name, metric);
> case INFO_CATEGORY_TM:
>   String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) 
> info).taskManagerID;
> {code}
> Looks like a break is missing following addMetric(jobManager.metrics) call



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2505: [FLINK-4628] provide user class loader during inpu...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2505

[FLINK-4628] provide user class loader during input split assignment

In analogy to the configure() method, this also sets a context class loader 
during input split assignment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-4628

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2505


commit 62d4075f89f96c1d85ca951c7705fd15e3e4
Author: Maximilian Michels 
Date:   2016-09-16T10:21:54Z

[FLINK-4628] provide user class loader during input split assignment

In analogy to the configure() method, this also sets a context class
loader during input split assignment.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4608) Use short-circuit AND in Max/Min AggregationFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495952#comment-15495952
 ] 

ASF GitHub Bot commented on FLINK-4608:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2489
  
merging


> Use short-circuit AND in Max/Min AggregationFunction
> 
>
> Key: FLINK-4608
> URL: https://issues.apache.org/jira/browse/FLINK-4608
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> Max/Min AggregationFunction use & instead of &&. Usually we use short-circuit 
> logic in if operators in java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2488: [FLINK-4607] Close FileInputStream in ParameterTool and o...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2488
  
merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4607) Close FileInputStream in ParameterTool and other

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495953#comment-15495953
 ] 

ASF GitHub Bot commented on FLINK-4607:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2488
  
merging


> Close FileInputStream in ParameterTool and other
> 
>
> Key: FLINK-4607
> URL: https://issues.apache.org/jira/browse/FLINK-4607
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> ParameterTool and some tests do not close FileInputStream
> {code}
> flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java
> flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java
> flink-java/src/test/java/org/apache/flink/api/java/utils/ParameterToolTest.java
> flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java
> flink-runtime/src/test/java/org/apache/flink/runtime/util/JarFileCreatorTest.java
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2489: [FLINK-4608] Use short-circuit AND in Max/Min Aggregation...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2489
  
merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4628) User class loader unavailable during input split assignment

[Flavio Pompermaier (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495945#comment-15495945
 ] 

Flavio Pompermaier commented on FLINK-4628:
---

Could this be also the cause of 
https://issues.apache.org/jira/browse/FLINK-4061?

> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the 
> executing thread.
> In 2 we only have the system class loader available. If any classes need to 
> be loaded then, we have a problem. Some InputFormats rely on code which 
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the 
> latest master.
> We should make the user class loader available when creating input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495940#comment-15495940
 ] 

ASF GitHub Bot commented on FLINK-4485:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
@StephanEwen Yes, it is simple. I just pushed a commit. This now releases 
all temp files after job completion.


> Finished jobs in yarn session fill /tmp filesystem
> --
>
> Key: FLINK-4485
> URL: https://issues.apache.org/jira/browse/FLINK-4485
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>Priority: Blocker
>
> On a Yarn cluster I start a yarn-session with a few containers and task slots.
> Then I fire a 'large' number of Flink batch jobs in sequence against this 
> yarn session. It is the exact same job (java code) yet it gets different 
> parameters.
> In this scenario it is exporting HBase tables to files in HDFS and the 
> parameters are about which data from which tables and the name of the target 
> directory.
> After running several dozen jobs the jobs submission started to fail and we 
> investigated.
> We found that the cause was that on the Yarn node which was hosting the 
> jobmanager the /tmp file system was full (4GB was 100% full).
> How ever the output of {{du -hcs /tmp}} showed only 200MB in use.
> We found that a very large file (we guess it is the jar of the job) was put 
> in /tmp , used, deleted yet the file handle was not closed by the jobmanager.
> As soon as we killed the jobmanager the disk space was freed.
> The summary of the impact of this is that a yarn-session that receives enough 
> jobs brings down the Yarn node for all users.
> See parts of the output we got from {{lsof}} below.
> {code}
> COMMAND PID  USER   FD  TYPE DEVICE  SIZE   
> NODE NAME
> java  15034   nbasjes  550r  REG 253,17  66219695
> 245 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0003 
> (deleted)
> java  15034   nbasjes  551r  REG 253,17  66219695
> 252 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0007 
> (deleted)
> java  15034   nbasjes  552r  REG 253,17  66219695
> 267 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0012 
> (deleted)
> java  15034   nbasjes  553r  REG 253,17  66219695
> 250 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0005 
> (deleted)
> java  15034   nbasjes  554r  REG 253,17  66219695
> 288 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0018 
> (deleted)
> java  15034   nbasjes  555r  REG 253,17  66219695
> 298 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0025 
> (deleted)
> java  15034   nbasjes  557r  REG 253,17  66219695
> 254 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0008 
> (deleted)
> java  15034   nbasjes  558r  REG 253,17  66219695
> 292 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0019 
> (deleted)
> java  15034   nbasjes  559r  REG 253,17  66219695
> 275 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0013 
> (deleted)
> java  15034   nbasjes  560r  REG 253,17  66219695
> 159 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0002 
> (deleted)
> java  15034   nbasjes  562r  REG 253,17  66219695
> 238 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0001 
> (deleted)
> java  15034   nbasjes  568r  REG 253,17  66219695
> 246 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0004 
> (deleted)
> java  15034   nbasjes  569r  REG 253,17  66219695
> 255 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0009 
> (deleted)
> java  15034   nbasjes  571r  REG 253,17  66219695
> 299 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0026 
> (deleted)
> java  15034   nbasjes  572r  REG 253,17  66219695
> 293 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0020 
> (deleted)
> java  15034   nbasjes  574r  REG 253,17  66219695
> 256 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0010 
> (deleted)
> java  15034   nbasjes  575r  REG   

[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
@StephanEwen Yes, it is simple. I just pushed a commit. This now releases 
all temp files after job completion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4628) User class loader unavailable during input split assignment

[Maximilian Michels (JIRA)]

Maximilian Michels created FLINK-4628:
-

 Summary: User class loader unavailable during input split 
assignment
 Key: FLINK-4628
 URL: https://issues.apache.org/jira/browse/FLINK-4628
 Project: Flink
  Issue Type: Bug
  Components: JobManager
Affects Versions: 1.1.2, 1.2.0
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Priority: Minor


{{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.

1. initializeOnMaster which runs the configure() method on the InputFormat
2. createInputSplits when the ExecutionJobVertex is created

In 1 we set the user class loader as the context class loader of the executing 
thread.

In 2 we only have the system class loader available. If any classes need to be 
loaded then, we have a problem. Some InputFormats rely on code which lazily 
load classes at different points in time.

In particular, this is a problem with the HBase TableInputFormat in the latest 
master.

We should make the user class loader available when creating input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495910#comment-15495910
 ] 

ASF GitHub Bot commented on FLINK-4485:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2499
  
What would the fix for the TaskManager look like? Simply explicitly closing 
the UserCodeClassloader, or does it need more?


> Finished jobs in yarn session fill /tmp filesystem
> --
>
> Key: FLINK-4485
> URL: https://issues.apache.org/jira/browse/FLINK-4485
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>Priority: Blocker
>
> On a Yarn cluster I start a yarn-session with a few containers and task slots.
> Then I fire a 'large' number of Flink batch jobs in sequence against this 
> yarn session. It is the exact same job (java code) yet it gets different 
> parameters.
> In this scenario it is exporting HBase tables to files in HDFS and the 
> parameters are about which data from which tables and the name of the target 
> directory.
> After running several dozen jobs the jobs submission started to fail and we 
> investigated.
> We found that the cause was that on the Yarn node which was hosting the 
> jobmanager the /tmp file system was full (4GB was 100% full).
> How ever the output of {{du -hcs /tmp}} showed only 200MB in use.
> We found that a very large file (we guess it is the jar of the job) was put 
> in /tmp , used, deleted yet the file handle was not closed by the jobmanager.
> As soon as we killed the jobmanager the disk space was freed.
> The summary of the impact of this is that a yarn-session that receives enough 
> jobs brings down the Yarn node for all users.
> See parts of the output we got from {{lsof}} below.
> {code}
> COMMAND PID  USER   FD  TYPE DEVICE  SIZE   
> NODE NAME
> java  15034   nbasjes  550r  REG 253,17  66219695
> 245 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0003 
> (deleted)
> java  15034   nbasjes  551r  REG 253,17  66219695
> 252 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0007 
> (deleted)
> java  15034   nbasjes  552r  REG 253,17  66219695
> 267 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0012 
> (deleted)
> java  15034   nbasjes  553r  REG 253,17  66219695
> 250 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0005 
> (deleted)
> java  15034   nbasjes  554r  REG 253,17  66219695
> 288 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0018 
> (deleted)
> java  15034   nbasjes  555r  REG 253,17  66219695
> 298 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0025 
> (deleted)
> java  15034   nbasjes  557r  REG 253,17  66219695
> 254 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0008 
> (deleted)
> java  15034   nbasjes  558r  REG 253,17  66219695
> 292 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0019 
> (deleted)
> java  15034   nbasjes  559r  REG 253,17  66219695
> 275 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0013 
> (deleted)
> java  15034   nbasjes  560r  REG 253,17  66219695
> 159 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0002 
> (deleted)
> java  15034   nbasjes  562r  REG 253,17  66219695
> 238 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0001 
> (deleted)
> java  15034   nbasjes  568r  REG 253,17  66219695
> 246 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0004 
> (deleted)
> java  15034   nbasjes  569r  REG 253,17  66219695
> 255 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0009 
> (deleted)
> java  15034   nbasjes  571r  REG 253,17  66219695
> 299 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0026 
> (deleted)
> java  15034   nbasjes  572r  REG 253,17  66219695
> 293 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0020 
> (deleted)
> java  15034   nbasjes  574r  REG 253,17  66219695
> 256 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0010 
> (deleted)
> java  15034   nbasjes  

[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2499
  
What would the fix for the TaskManager look like? Simply explicitly closing 
the UserCodeClassloader, or does it need more?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495900#comment-15495900
 ] 

ASF GitHub Bot commented on FLINK-4485:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
Thanks! Just a few words to @nielsbasjes who reported the issue. I've 
tested the fix using the test instructions you provided. Even before this fix, 
I could get rid of the temp files by forcing a manual garbage collection on the 
JVM, using `jcmd  GC.run`. However, that only worked once the job meta 
data had been removed from the archive, i.e. it doesn't show up in the web 
interface anymore. With this fix, the class loader is cleared upon job 
completion and the files are immediately removed. `lsof | fgrep blob_` didn't 
show any of these files anymore.

Note, that we don't perform any cleanup on the TaskManager side. There we 
also wind up with some left over files but they don't seem to pile up. It must 
be that the garbage collector can figure out when to clean much earlier. Plus, 
we don't keep a reference to old Task instances like we do for the web 
interface on the JobManager side.

@StephanEwen I'm thinking about adding a similar fix for the TaskManager 
side. What do you think?


> Finished jobs in yarn session fill /tmp filesystem
> --
>
> Key: FLINK-4485
> URL: https://issues.apache.org/jira/browse/FLINK-4485
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>Priority: Blocker
>
> On a Yarn cluster I start a yarn-session with a few containers and task slots.
> Then I fire a 'large' number of Flink batch jobs in sequence against this 
> yarn session. It is the exact same job (java code) yet it gets different 
> parameters.
> In this scenario it is exporting HBase tables to files in HDFS and the 
> parameters are about which data from which tables and the name of the target 
> directory.
> After running several dozen jobs the jobs submission started to fail and we 
> investigated.
> We found that the cause was that on the Yarn node which was hosting the 
> jobmanager the /tmp file system was full (4GB was 100% full).
> How ever the output of {{du -hcs /tmp}} showed only 200MB in use.
> We found that a very large file (we guess it is the jar of the job) was put 
> in /tmp , used, deleted yet the file handle was not closed by the jobmanager.
> As soon as we killed the jobmanager the disk space was freed.
> The summary of the impact of this is that a yarn-session that receives enough 
> jobs brings down the Yarn node for all users.
> See parts of the output we got from {{lsof}} below.
> {code}
> COMMAND PID  USER   FD  TYPE DEVICE  SIZE   
> NODE NAME
> java  15034   nbasjes  550r  REG 253,17  66219695
> 245 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0003 
> (deleted)
> java  15034   nbasjes  551r  REG 253,17  66219695
> 252 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0007 
> (deleted)
> java  15034   nbasjes  552r  REG 253,17  66219695
> 267 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0012 
> (deleted)
> java  15034   nbasjes  553r  REG 253,17  66219695
> 250 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0005 
> (deleted)
> java  15034   nbasjes  554r  REG 253,17  66219695
> 288 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0018 
> (deleted)
> java  15034   nbasjes  555r  REG 253,17  66219695
> 298 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0025 
> (deleted)
> java  15034   nbasjes  557r  REG 253,17  66219695
> 254 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0008 
> (deleted)
> java  15034   nbasjes  558r  REG 253,17  66219695
> 292 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0019 
> (deleted)
> java  15034   nbasjes  559r  REG 253,17  66219695
> 275 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0013 
> (deleted)
> java  15034   nbasjes  560r  REG 253,17  66219695
> 159 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0002 
> (deleted)
> java  15034   nbasjes  562r  REG 253,17  66219695
> 238 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0001 
> (deleted)
> java  15034   nbasjes 

[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
Thanks! Just a few words to @nielsbasjes who reported the issue. I've 
tested the fix using the test instructions you provided. Even before this fix, 
I could get rid of the temp files by forcing a manual garbage collection on the 
JVM, using `jcmd  GC.run`. However, that only worked once the job meta 
data had been removed from the archive, i.e. it doesn't show up in the web 
interface anymore. With this fix, the class loader is cleared upon job 
completion and the files are immediately removed. `lsof | fgrep blob_` didn't 
show any of these files anymore.

Note, that we don't perform any cleanup on the TaskManager side. There we 
also wind up with some left over files but they don't seem to pile up. It must 
be that the garbage collector can figure out when to clean much earlier. Plus, 
we don't keep a reference to old Task instances like we do for the web 
interface on the JobManager side.

@StephanEwen I'm thinking about adding a similar fix for the TaskManager 
side. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2476: [FLINK-4589] Fix Merging of Covering Window in Mer...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2476


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4589) Fix Merging of Covering Window in MergingWindowSet

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495883#comment-15495883
 ] 

ASF GitHub Bot commented on FLINK-4589:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2476


> Fix Merging of Covering Window in MergingWindowSet
> --
>
> Key: FLINK-4589
> URL: https://issues.apache.org/jira/browse/FLINK-4589
> Project: Flink
>  Issue Type: Bug
>  Components: Windowing Operators
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.0.4, 1.2.0, 1.1.3
>
>
> Right now, when a new window gets merged that covers all of the existing 
> window {{MergingWindowSet}} does not correctly set the state window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4581) Table API throws "No suitable driver found for jdbc:calcite"

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther reassigned FLINK-4581:
---

Assignee: Timo Walther

> Table API throws "No suitable driver found for jdbc:calcite"
> 
>
> Key: FLINK-4581
> URL: https://issues.apache.org/jira/browse/FLINK-4581
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> It seems that in certain cases the internal Calcite JDBC driver cannot be 
> found. We should either try to get rid of the entire JDBC invocation or fix 
> this bug.
> From ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Stream-sql-query-in-Flink-tp8928.html
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
>   at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:524)
>   at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:331)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253)
>   at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
> Caused by: java.lang.RuntimeException: java.sql.SQLException: No suitable
> driver found for jdbc:calcite:
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:151)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:127)
>   at
> org.apache.flink.api.table.FlinkRelBuilder$.create(FlinkRelBuilder.scala:56)
>   at
> org.apache.flink.api.table.TableEnvironment.(TableEnvironment.scala:73)
>   at
> org.apache.flink.api.table.StreamTableEnvironment.(StreamTableEnvironment.scala:58)
>   at
> org.apache.flink.api.java.table.StreamTableEnvironment.(StreamTableEnvironment.scala:45)
>   at
> org.apache.flink.api.table.TableEnvironment$.getTableEnvironment(TableEnvironment.scala:376)
>   at
> org.apache.flink.api.table.TableEnvironment.getTableEnvironment(TableEnvironment.scala)
>   at 
> org.myorg.quickstart.ReadingFromKafka2.main(ReadingFromKafka2.java:48)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>   ... 6 more
> Caused by: java.sql.SQLException: No suitable driver found for jdbc:calcite:
>   at java.sql.DriverManager.getConnection(DriverManager.java:689)
>   at java.sql.DriverManager.getConnection(DriverManager.java:208)
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:144)
>   ... 20 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4590) Some Table API tests are failing when debug lvl is set to DEBUG

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495808#comment-15495808
 ] 

ASF GitHub Bot commented on FLINK-4590:
---

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2504

[FLINK-4590] [table] Some Table API tests are failing when debug lvl is set 
to DEBUG

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR fixes the estimated costs for minus and union DataSet RelNodes. I 
went through all RelNodes and also fixed multiple potential null pointers, 
bugs, and warnings.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4590

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2504.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2504


commit a13e30799cb4a5d59de813b96d3519a337461aa6
Author: twalthr 
Date:   2016-09-16T08:52:28Z

[FLINK-4590] [table] Some Table API tests are failing when debug lvl is set 
to DEBUG




> Some Table API tests are failing when debug lvl is set to DEBUG
> ---
>
> Key: FLINK-4590
> URL: https://issues.apache.org/jira/browse/FLINK-4590
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Timo Walther
>
> For debugging another issue, I've set the log level on travis to DEBUG.
> After that, the Table API tests started failing
> {code}
> Failed tests: 
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
> {code}
> Probably Calcite is executing additional assertions depending on the debug 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2504: [FLINK-4590] [table] Some Table API tests are fail...

[twalthr]

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/2504

[FLINK-4590] [table] Some Table API tests are failing when debug lvl is set 
to DEBUG

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

This PR fixes the estimated costs for minus and union DataSet RelNodes. I 
went through all RelNodes and also fixed multiple potential null pointers, 
bugs, and warnings.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink FLINK-4590

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2504.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2504


commit a13e30799cb4a5d59de813b96d3519a337461aa6
Author: twalthr 
Date:   2016-09-16T08:52:28Z

[FLINK-4590] [table] Some Table API tests are failing when debug lvl is set 
to DEBUG




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3869) WindowedStream.apply with FoldFunction is too restrictive

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495761#comment-15495761
 ] 

Aljoscha Krettek commented on FLINK-3869:
-

I have another proposal for solving this: I started to dislike how the 
reduce/fold with window function is called {{apply()}}. Yes, You can see the 
apply with incremental aggregation as either an apply that has an additional 
incremental aggregation function or as an incremental aggregation (reduce/fold) 
that has an apply added to it. Right now, I think these two methods should be:
{code}
reduce(ReduceFunction, WindowFunction)
fold(Initial, FoldFunction, WindowFunction)
{code}

this would make them more discoverable, i.e. you do a reduce and then you see 
that you can also give a {{WindowFunction}} to get all the meta data for the 
window.

If we changed to that new naming scheme we could fix the fold restriction and 
deprecate the old methods, thereby not breaking any existing code.

[~StephanEwen] what do you think?

> WindowedStream.apply with FoldFunction is too restrictive
> -
>
> Key: FLINK-3869
> URL: https://issues.apache.org/jira/browse/FLINK-3869
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> Right now we have this signature:
> {code}
> public  SingleOutputStreamOperator apply(R initialValue, 
> FoldFunction foldFunction, WindowFunction function) {
> {code}
> but we should have this signature to allow users to return a type other than 
> the fold accumulator type from their window function:
> {code}
> public  SingleOutputStreamOperator apply(ACC initialValue, 
> FoldFunction foldFunction, WindowFunction function) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4620) Automatically creating savepoints

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495665#comment-15495665
 ] 

Aljoscha Krettek commented on FLINK-4620:
-

I think FLINK-4511 is the specific issue for that part of the refactoring. 
[~nielsbasjes], maybe you could move your comments and description there, 
they're very good.

> Automatically creating savepoints
> -
>
> Key: FLINK-4620
> URL: https://issues.apache.org/jira/browse/FLINK-4620
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.1.2
>Reporter: Niels Basjes
>
> In the current versions of Flink you can run an external command and then a 
> savepoint is persisted in a durable location.
> Feature request: Make this a lot more automatic and easy to use.
> _Proposed workflow_
> # In my application I do something like this:
> {code}
> StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setStateBackend(new FsStateBackend("hdfs:///tmp/applicationState"));
> env.enableCheckpointing(5000, CheckpointingMode.EXACTLY_ONCE);
> env.enableAutomaticSavePoints(30);
> env.enableAutomaticSavePointCleaner(10);
> {code}
> # When I start the application for the first time the state backend is 
> 'empty'. 
> I expect the system to start in a clean state.
> After 10 minutes (30ms) a savepoint is created and stored.
> # When I stop and start the topology again it will automatically restore the 
> last available savepoint.
> Things to think about:
> * Note that this feature still means the manual version is useful!!
> * What to do on startup if the state is incompatible with the topology? Fail 
> the startup?
> * How many automatic savepoints to we keep? Only the last one?
> * Perhaps the API should allow multiple automatic savepoints at different 
> intervals in different locations.
> {code}
> // Make every 10 minutes and keep the last 10
> env.enableAutomaticSavePoints(30, new 
> FsStateBackend("hdfs:///tmp/applicationState"), 10);
> // Make every 24 hours and keep the last 30
> // Useful for being able to reproduce a problem a few days later
> env.enableAutomaticSavePoints(8640, new 
> FsStateBackend("hdfs:///tmp/applicationDailyStateSnapshot"), 30);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)