[GitHub] [flink] flinkbot edited a comment on issue #9610: [FLINK-13936][licensing] Update NOTICE-binary

[GitBox]

flinkbot edited a comment on issue #9610: [FLINK-13936][licensing] Update 
NOTICE-binary
URL: https://github.com/apache/flink/pull/9610#issuecomment-527838177
 
 
   
   ## CI report:
   
   * 755a869c93208b7d95a85d3fe4f0027ca55eff2e : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/125869215)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9614: [FLINK-13623][Tests] Update StatefulJobSavepointMigrationITCase to restore from 1.9 savepoint

[GitBox]

flinkbot edited a comment on issue #9614: [FLINK-13623][Tests] Update 
StatefulJobSavepointMigrationITCase to restore from 1.9 savepoint
URL: https://github.com/apache/flink/pull/9614#issuecomment-527883601
 
 
   
   ## CI report:
   
   * e96c273e30e6a6fa2a55541668b5dd5cda5a34a0 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125888732)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-13955) Integrate ContinuousFileReaderOperator with StreamTask mailbox execution model

[Alex (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex updated FLINK-13955:
-
Description: 
The {{ContinuousFileReaderOperator}} spawns an additional ({{reader}}) thread 
and coordinates actions with it via {{checkpointLock}}. The operator may block, 
waiting for a notification from the {{reader}} thread, potentially preventing 
progress in mailbox loop.

This is similar to {{AsyncWaitOperator}} situation described in FLINK-12958.

  was:
The {{ContinuousFileReaderOperator}} spawns an additional ({{reader}}) thread 
and coordinates actions with it via {{checkpointLock}}. The operator may block, 
waiting for a notification from the {{reader}} thread, preventing progress in 
mailbox loop.

This is similar to {{AsyncWaitOperator}} situation described in FLINK-12958.


> Integrate ContinuousFileReaderOperator with StreamTask mailbox execution model
> --
>
> Key: FLINK-13955
> URL: https://issues.apache.org/jira/browse/FLINK-13955
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Task
>Reporter: Alex
>Priority: Major
>
> The {{ContinuousFileReaderOperator}} spawns an additional ({{reader}}) thread 
> and coordinates actions with it via {{checkpointLock}}. The operator may 
> block, waiting for a notification from the {{reader}} thread, potentially 
> preventing progress in mailbox loop.
> This is similar to {{AsyncWaitOperator}} situation described in FLINK-12958.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput

[GitBox]

AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate 
isFinished method from AvailabilityListener to AsyncDataInput
URL: https://github.com/apache/flink/pull/9483#discussion_r320741075
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/AvailabilityListener.java
 ##
 @@ -33,11 +33,6 @@
 */
CompletableFuture AVAILABLE = 
CompletableFuture.completedFuture(null);
 
-   /**
 
 Review comment:
   Not sure if this is relevant to this PR, but isn't `AvailabilityListener` a 
misnomer? I'd expect a listener to be the one to be notified and not the one to 
notify. I also only know listeners as callbacks.
   Implementators of this interface would rather be `AvailabilityReporter` or 
`AvailabilityProvider`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput

[GitBox]

AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate 
isFinished method from AvailabilityListener to AsyncDataInput
URL: https://github.com/apache/flink/pull/9483#discussion_r320748449
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/AvailabilityListener.java
 ##
 @@ -55,7 +50,7 @@
 * @return a future that is completed if there are more records 
available. If there are more
 * records available immediately, {@link #AVAILABLE} should be 
returned. Previously returned
 * not completed futures should become completed once there is more 
input available or if
-* the input {@link #isFinished()}.
+* the input is finished.
 
 Review comment:
   Should we also have a constant for `FINISHED`? As is the 
`AvailabilityListener` depends on its implementations to provide some 
`isFinished()`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput

[GitBox]

AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate 
isFinished method from AvailabilityListener to AsyncDataInput
URL: https://github.com/apache/flink/pull/9483#discussion_r320743013
 
 

 ##
 File path: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessor.java
 ##
 @@ -174,30 +173,30 @@ public boolean isFinished() {
}
 
@Override
-   public boolean processInput() throws Exception {
+   public InputStatus processInput() throws Exception {
// the preparations here are not placed in the constructor 
because all work in it
// must be executed after all operators are opened.
prepareForProcessing();
 
int readingInputIndex = selectNextReadingInputIndex();
if (readingInputIndex == -1) {
-   return false;
+   return InputStatus.NOTHING_AVAILABLE;
}
lastReadInputIndex = readingInputIndex;
 
-   InputStatus status;
+   InputStatus status, anotherStatus;
 
 Review comment:
   `anotherStatus` is not really expressive. How about `otherInputStatus`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput

[GitBox]

AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate 
isFinished method from AvailabilityListener to AsyncDataInput
URL: https://github.com/apache/flink/pull/9483#discussion_r320736635
 
 

 ##
 File path: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessor.java
 ##
 @@ -121,18 +115,46 @@ public StreamTwoInputProcessor(
taskManagerConfig,
taskName);
checkState(checkpointedInputGates.length == 2);
-   this.input1 = new 
StreamTaskNetworkInput(checkpointedInputGates[0], inputSerializer1, ioManager, 
0);
-   this.input2 = new 
StreamTaskNetworkInput(checkpointedInputGates[1], inputSerializer2, ioManager, 
1);
 
-   this.statusWatermarkValve1 = new StatusWatermarkValve(
-   unionedInputGate1.getNumberOfInputChannels(),
-   new ForwardingValveOutputHandler(streamOperator, lock, 
streamStatusMaintainer, input1WatermarkGauge, 0));
-   this.statusWatermarkValve2 = new StatusWatermarkValve(
-   unionedInputGate2.getNumberOfInputChannels(),
-   new ForwardingValveOutputHandler(streamOperator, lock, 
streamStatusMaintainer, input2WatermarkGauge, 1));
+   this.output1 = new StreamTaskNetworkOutput<>(
+   streamOperator,
+   (StreamRecord record) -> {
 
 Review comment:
   I'd extract it to a function and reference it as a method lambda


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput

[GitBox]

AHeise commented on a change in pull request #9483: [FLINK-13767][task] Migrate 
isFinished method from AvailabilityListener to AsyncDataInput
URL: https://github.com/apache/flink/pull/9483#discussion_r320738355
 
 

 ##
 File path: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessor.java
 ##
 @@ -72,10 +72,11 @@
 * Stream status for the two inputs. We need to keep track for 
determining when
 * to forward stream status changes downstream.
 */
-   private StreamStatus firstStatus;
-   private StreamStatus secondStatus;
+   private StreamStatus firstStatus = StreamStatus.ACTIVE;
 
 Review comment:
   +1 to have initialized fields outside of ctor


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9404: [FLINK-13667][ml] Add the utility class for the Table

[GitBox]

flinkbot edited a comment on issue #9404: [FLINK-13667][ml] Add the utility 
class for the Table
URL: https://github.com/apache/flink/pull/9404#issuecomment-519872127
 
 
   
   ## CI report:
   
   * 4a726770daa19b1a587c5f8d9221a95e08387592 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/122573104)
   * cd4cfbe81b4d511ec985bc4ad55a5dad82722863 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124368074)
   * 6efbedf230f6fdeec96b9802d296629269a230ab : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125253936)
   * 51f02e22e7372fa20ab2dd679aaa1099b37d0b1e : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9405: [FLINK-13668][ml] Add abstract classes for three typical scenarios of (Flat)Mapper

[GitBox]

flinkbot edited a comment on issue #9405: [FLINK-13668][ml] Add abstract 
classes for three typical scenarios of (Flat)Mapper
URL: https://github.com/apache/flink/pull/9405#issuecomment-519901201
 
 
   
   ## CI report:
   
   * c1fedc4de6a92bc2c5f84d0e0c3038f29fd8944f : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122585906)
   * 7f235ecf56417da32c78bb5d064c9d833e1d5261 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9406: [FLINK-13669][ml] Add class for BinarizerMapper

[GitBox]

flinkbot edited a comment on issue #9406: [FLINK-13669][ml] Add class for 
BinarizerMapper
URL: https://github.com/apache/flink/pull/9406#issuecomment-519954683
 
 
   
   ## CI report:
   
   * eb27a4f02fee6d3d96a102ef7b4734d06442596a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122608982)
   * 8a8a364e5b392e9251dfcec4241dd162bf62d089 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9407: [FLINK-13670][ml] Add class for VectorAssemblerMapper

[GitBox]

flinkbot edited a comment on issue #9407: [FLINK-13670][ml] Add class for 
VectorAssemblerMapper
URL: https://github.com/apache/flink/pull/9407#issuecomment-51995
 
 
   
   ## CI report:
   
   * bf69f9f25f8a62a494f530e7cba75879ac002412 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122610736)
   * e92b935e874524c967a9b0296833872c602d0a33 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9408: [FLINK-13671][ml] Add class for VectorEleWiseProductMapper

[GitBox]

flinkbot edited a comment on issue #9408: [FLINK-13671][ml] Add class for 
VectorEleWiseProductMapper
URL: https://github.com/apache/flink/pull/9408#issuecomment-519961164
 
 
   
   ## CI report:
   
   * fbf9704fe2812d28d7a6b9ca6272a260e91089b3 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122611669)
   * a2091168929daf552e6e839c9ef179c983192fcf : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9411: [FLINK-13674][ml] Add class of Vector Size Hint Mapper

[GitBox]

flinkbot edited a comment on issue #9411: [FLINK-13674][ml] Add class of Vector 
Size Hint Mapper
URL: https://github.com/apache/flink/pull/9411#issuecomment-519965855
 
 
   
   ## CI report:
   
   * 4af38a946a55d1d6b68d7d249f87a65239c947f7 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122613586)
   * 5e9e66fcf02dc111a807661502761e1a67a043eb : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9409: [FLINK-13672][ml] Add class for VectorInteractionMapper

[GitBox]

flinkbot edited a comment on issue #9409: [FLINK-13672][ml] Add class for 
VectorInteractionMapper
URL: https://github.com/apache/flink/pull/9409#issuecomment-519961173
 
 
   
   ## CI report:
   
   * 9b5e1620eb70dd95996ec3e8b7bebe1f38ac89a5 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122611642)
   * 1d402f4ed866db5e9bbd35496949f8fdc8197c5e : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9412: [FLINK-13675][ml] Add class of Vector Slice Mapper

[GitBox]

flinkbot edited a comment on issue #9412: [FLINK-13675][ml] Add class of Vector 
Slice Mapper
URL: https://github.com/apache/flink/pull/9412#issuecomment-519965864
 
 
   
   ## CI report:
   
   * e9ff64826d561117cbd984df4201d15e0523b091 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122613554)
   * 8e513594aaea05df91f6d2dd17088c4adb2fd1e9 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9410: [FLINK-13673][ml] Add class of Vector Normalize Mapper

[GitBox]

flinkbot edited a comment on issue #9410: [FLINK-13673][ml] Add class of Vector 
Normalize Mapper
URL: https://github.com/apache/flink/pull/9410#issuecomment-519963431
 
 
   
   ## CI report:
   
   * 06b26ce69e490794f3bcda79766f8edc9d6e48ce : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122612649)
   * a5e7c30d865ea6ab35644743c9477a940ec9de65 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9413: [FLINK-13676][ml] Add class of Vector to Columns mapper

[GitBox]

flinkbot edited a comment on issue #9413: [FLINK-13676][ml] Add class of Vector 
to Columns mapper
URL: https://github.com/apache/flink/pull/9413#issuecomment-519968250
 
 
   
   ## CI report:
   
   * a585d0f8fef391d4da791b0cd1506e4f8b1acdc1 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122614562)
   * e399e94cb093d210dc9183bef48db93d60c30ca7 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (FLINK-13382) Port DecimalITCase to unit tests

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922478#comment-16922478
 ] 

Leonard Xu edited comment on FLINK-13382 at 9/4/19 1:24 PM:


[~jark] I am glad to fix this


was (Author: leonard xu):
[~jark]I am glad to fix this

> Port DecimalITCase to unit tests
> 
>
> Key: FLINK-13382
> URL: https://issues.apache.org/jira/browse/FLINK-13382
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Minor
> Fix For: 1.10.0
>
>
> There are 81 query tests in 
> {{org.apache.flink.table.runtime.batch.sql.DecimalITCase}} and 67 query tests 
> in {{org.apache.flink.table.runtime.batch.table.DecimalITCase}}. 
> Note that each query will be executed on a MiniCluster which will consume a 
> lot of testing time. However, decimals can be tested as unit tests similar to 
> other Scalar functions tests.
> So I would suggest to move Decimal IT cases to unit tests 
> {{org.apache.flink.table.expressions.DecimalTypeTest}}. This should save much 
> testing time. 
> In the meanwhile, IMO, this is not required by 1.9 release. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (FLINK-13382) Port DecimalITCase to unit tests

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922478#comment-16922478
 ] 

Leonard Xu commented on FLINK-13382:


[~jark]I am glad to fix this

> Port DecimalITCase to unit tests
> 
>
> Key: FLINK-13382
> URL: https://issues.apache.org/jira/browse/FLINK-13382
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Minor
> Fix For: 1.10.0
>
>
> There are 81 query tests in 
> {{org.apache.flink.table.runtime.batch.sql.DecimalITCase}} and 67 query tests 
> in {{org.apache.flink.table.runtime.batch.table.DecimalITCase}}. 
> Note that each query will be executed on a MiniCluster which will consume a 
> lot of testing time. However, decimals can be tested as unit tests similar to 
> other Scalar functions tests.
> So I would suggest to move Decimal IT cases to unit tests 
> {{org.apache.flink.table.expressions.DecimalTypeTest}}. This should save much 
> testing time. 
> In the meanwhile, IMO, this is not required by 1.9 release. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (FLINK-13382) Port DecimalITCase to unit tests

[Jark Wu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu reassigned FLINK-13382:
---

Assignee: Leonard Xu

> Port DecimalITCase to unit tests
> 
>
> Key: FLINK-13382
> URL: https://issues.apache.org/jira/browse/FLINK-13382
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Minor
> Fix For: 1.10.0
>
>
> There are 81 query tests in 
> {{org.apache.flink.table.runtime.batch.sql.DecimalITCase}} and 67 query tests 
> in {{org.apache.flink.table.runtime.batch.table.DecimalITCase}}. 
> Note that each query will be executed on a MiniCluster which will consume a 
> lot of testing time. However, decimals can be tested as unit tests similar to 
> other Scalar functions tests.
> So I would suggest to move Decimal IT cases to unit tests 
> {{org.apache.flink.table.expressions.DecimalTypeTest}}. This should save much 
> testing time. 
> In the meanwhile, IMO, this is not required by 1.9 release. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] lirui-apache opened a new pull request #9615: [FLINK-13952][table-planner][hive] PartitionableTableSink can not wor…

[GitBox]

lirui-apache opened a new pull request #9615: 
[FLINK-13952][table-planner][hive] PartitionableTableSink can not wor…
URL: https://github.com/apache/flink/pull/9615
 
 
   …k with OverwritableTableSink
   
   
   
   ## What is the purpose of the change
   
   To support insert overwrite partition.
   
   
   ## Brief change log
   
 - Make sure planner supports sinks that are both PartitionableTableSink 
and OverwritableTableSink
 - Added tests for insert overwrite partitions.
   
   
   ## Verifying this change
   
   Added new test cases.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? NA
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-13952) PartitionableTableSink can not work with OverwritableTableSink

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-13952:
---
Labels: pull-request-available  (was: )

> PartitionableTableSink can not work with OverwritableTableSink
> --
>
> Key: FLINK-13952
> URL: https://issues.apache.org/jira/browse/FLINK-13952
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Reporter: Jingsong Lee
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0
>
>
> {code:java}
> tableSink match {
>   case partitionableSink: PartitionableTableSink
> if partitionableSink.getPartitionFieldNames != null
>   && partitionableSink.getPartitionFieldNames.nonEmpty =>
> partitionableSink.setStaticPartition(insertOptions.staticPartitions)
>   case overwritableTableSink: OverwritableTableSink =>
> overwritableTableSink.setOverwrite(insertOptions.overwrite)
> {code}
> Code in TableEnvImpl and PlannerBase
> overwrite will not be invoked when there are static partition columns.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (FLINK-13957) Redact passwords from dynamic properties logged on job submission

[Jira]

 [ 
https://issues.apache.org/jira/browse/FLINK-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi closed FLINK-13957.
--
Resolution: Invalid

The issue fortunately did not make it to the upstream implementation, was only 
present in the Cludera fork of Flink. We will commit the original contribution 
without the issue. 

> Redact passwords from dynamic properties logged on job submission
> -
>
> Key: FLINK-13957
> URL: https://issues.apache.org/jira/browse/FLINK-13957
> Project: Flink
>  Issue Type: Improvement
>  Components: Client / Job Submission
>Affects Versions: 1.9.0
>Reporter: Matyas Orhidi
>Assignee: Matyas Orhidi
>Priority: Major
>  Labels: log, security, sensitivity
> Fix For: 1.9.1
>
>
> SSL related passwords specified by dynamic properties
> {{flink run -m yarn-cluster -sae -p 2 -ynm HeapMonitor}}
>  {{...}}
> -yD security.ssl.internal.key-password=changeit
>  {{-yD security.ssl.internal.keystore-password=}}{{changeit}}
>  {{-yD security.ssl.internal.truststore-password=}}{{changeit}}
> {{...}}
> are showing up in {{FlinkYarnSessionCli}} logs in plain text:
> {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore-password=changeit}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.keystore-password=changeit}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.key-password=changeit}}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] flinkbot commented on issue #9615: [FLINK-13952][table-planner][hive] PartitionableTableSink can not wor…

[GitBox]

flinkbot commented on issue #9615: [FLINK-13952][table-planner][hive] 
PartitionableTableSink can not wor…
URL: https://github.com/apache/flink/pull/9615#issuecomment-527902937
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit b9cf01cfcebcc894450a7dd9d8c013108faccd6c (Wed Sep 04 
13:33:59 UTC 2019)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **This pull request references an unassigned [Jira 
ticket](https://issues.apache.org/jira/browse/FLINK-13952).** According to the 
[code contribution 
guide](https://flink.apache.org/contributing/contribute-code.html), tickets 
need to be assigned before starting with the implementation work.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9404: [FLINK-13667][ml] Add the utility class for the Table

[GitBox]

flinkbot edited a comment on issue #9404: [FLINK-13667][ml] Add the utility 
class for the Table
URL: https://github.com/apache/flink/pull/9404#issuecomment-519872127
 
 
   
   ## CI report:
   
   * 4a726770daa19b1a587c5f8d9221a95e08387592 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/122573104)
   * cd4cfbe81b4d511ec985bc4ad55a5dad82722863 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124368074)
   * 6efbedf230f6fdeec96b9802d296629269a230ab : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125253936)
   * 51f02e22e7372fa20ab2dd679aaa1099b37d0b1e : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893622)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9405: [FLINK-13668][ml] Add abstract classes for three typical scenarios of (Flat)Mapper

[GitBox]

flinkbot edited a comment on issue #9405: [FLINK-13668][ml] Add abstract 
classes for three typical scenarios of (Flat)Mapper
URL: https://github.com/apache/flink/pull/9405#issuecomment-519901201
 
 
   
   ## CI report:
   
   * c1fedc4de6a92bc2c5f84d0e0c3038f29fd8944f : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122585906)
   * 7f235ecf56417da32c78bb5d064c9d833e1d5261 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893682)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-13957) Log dynamic properties on job submission

[Jira]

 [ 
https://issues.apache.org/jira/browse/FLINK-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi updated FLINK-13957:
---
Summary: Log dynamic properties on job submission  (was: Redact passwords 
from dynamic properties logged on job submission)

> Log dynamic properties on job submission
> 
>
> Key: FLINK-13957
> URL: https://issues.apache.org/jira/browse/FLINK-13957
> Project: Flink
>  Issue Type: Improvement
>  Components: Client / Job Submission
>Affects Versions: 1.9.0
>Reporter: Matyas Orhidi
>Assignee: Matyas Orhidi
>Priority: Major
>  Labels: log, security, sensitivity
> Fix For: 1.9.1
>
>
> SSL related passwords specified by dynamic properties
> {{flink run -m yarn-cluster -sae -p 2 -ynm HeapMonitor}}
>  {{...}}
> -yD security.ssl.internal.key-password=changeit
>  {{-yD security.ssl.internal.keystore-password=}}{{changeit}}
>  {{-yD security.ssl.internal.truststore-password=}}{{changeit}}
> {{...}}
> are showing up in {{FlinkYarnSessionCli}} logs in plain text:
> {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore-password=changeit}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.keystore-password=changeit}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.key-password=changeit}}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] flinkbot edited a comment on issue #9406: [FLINK-13669][ml] Add class for BinarizerMapper

[GitBox]

flinkbot edited a comment on issue #9406: [FLINK-13669][ml] Add class for 
BinarizerMapper
URL: https://github.com/apache/flink/pull/9406#issuecomment-519954683
 
 
   
   ## CI report:
   
   * eb27a4f02fee6d3d96a102ef7b4734d06442596a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122608982)
   * 8a8a364e5b392e9251dfcec4241dd162bf62d089 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893714)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9407: [FLINK-13670][ml] Add class for VectorAssemblerMapper

[GitBox]

flinkbot edited a comment on issue #9407: [FLINK-13670][ml] Add class for 
VectorAssemblerMapper
URL: https://github.com/apache/flink/pull/9407#issuecomment-51995
 
 
   
   ## CI report:
   
   * bf69f9f25f8a62a494f530e7cba75879ac002412 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122610736)
   * e92b935e874524c967a9b0296833872c602d0a33 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893754)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9410: [FLINK-13673][ml] Add class of Vector Normalize Mapper

[GitBox]

flinkbot edited a comment on issue #9410: [FLINK-13673][ml] Add class of Vector 
Normalize Mapper
URL: https://github.com/apache/flink/pull/9410#issuecomment-519963431
 
 
   
   ## CI report:
   
   * 06b26ce69e490794f3bcda79766f8edc9d6e48ce : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122612649)
   * a5e7c30d865ea6ab35644743c9477a940ec9de65 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893914)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9409: [FLINK-13672][ml] Add class for VectorInteractionMapper

[GitBox]

flinkbot edited a comment on issue #9409: [FLINK-13672][ml] Add class for 
VectorInteractionMapper
URL: https://github.com/apache/flink/pull/9409#issuecomment-519961173
 
 
   
   ## CI report:
   
   * 9b5e1620eb70dd95996ec3e8b7bebe1f38ac89a5 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122611642)
   * 1d402f4ed866db5e9bbd35496949f8fdc8197c5e : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893871)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9413: [FLINK-13676][ml] Add class of Vector to Columns mapper

[GitBox]

flinkbot edited a comment on issue #9413: [FLINK-13676][ml] Add class of Vector 
to Columns mapper
URL: https://github.com/apache/flink/pull/9413#issuecomment-519968250
 
 
   
   ## CI report:
   
   * a585d0f8fef391d4da791b0cd1506e4f8b1acdc1 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122614562)
   * e399e94cb093d210dc9183bef48db93d60c30ca7 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125894099)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9408: [FLINK-13671][ml] Add class for VectorEleWiseProductMapper

[GitBox]

flinkbot edited a comment on issue #9408: [FLINK-13671][ml] Add class for 
VectorEleWiseProductMapper
URL: https://github.com/apache/flink/pull/9408#issuecomment-519961164
 
 
   
   ## CI report:
   
   * fbf9704fe2812d28d7a6b9ca6272a260e91089b3 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122611669)
   * a2091168929daf552e6e839c9ef179c983192fcf : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893825)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9411: [FLINK-13674][ml] Add class of Vector Size Hint Mapper

[GitBox]

flinkbot edited a comment on issue #9411: [FLINK-13674][ml] Add class of Vector 
Size Hint Mapper
URL: https://github.com/apache/flink/pull/9411#issuecomment-519965855
 
 
   
   ## CI report:
   
   * 4af38a946a55d1d6b68d7d249f87a65239c947f7 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122613586)
   * 5e9e66fcf02dc111a807661502761e1a67a043eb : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893971)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9412: [FLINK-13675][ml] Add class of Vector Slice Mapper

[GitBox]

flinkbot edited a comment on issue #9412: [FLINK-13675][ml] Add class of Vector 
Slice Mapper
URL: https://github.com/apache/flink/pull/9412#issuecomment-519965864
 
 
   
   ## CI report:
   
   * e9ff64826d561117cbd984df4201d15e0523b091 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122613554)
   * 8e513594aaea05df91f6d2dd17088c4adb2fd1e9 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125894095)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Reopened] (FLINK-13957) Log dynamic properties on job submission

[Jira]

 [ 
https://issues.apache.org/jira/browse/FLINK-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi reopened FLINK-13957:


> Log dynamic properties on job submission
> 
>
> Key: FLINK-13957
> URL: https://issues.apache.org/jira/browse/FLINK-13957
> Project: Flink
>  Issue Type: Improvement
>  Components: Client / Job Submission
>Affects Versions: 1.9.0
>Reporter: Matyas Orhidi
>Assignee: Matyas Orhidi
>Priority: Major
>  Labels: log, security, sensitivity
> Fix For: 1.9.1
>
>
> This would let users understand their job submissions better, expected output 
> after suitable redactions from the {{FlinkYarnSessionCli:}} 
> {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore-password=*}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore=/opt/security/jks/truststore.jks}}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Issue Comment Deleted] (FLINK-13957) Log dynamic properties on job submission

[Jira]

 [ 
https://issues.apache.org/jira/browse/FLINK-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi updated FLINK-13957:
---
Comment: was deleted

(was: The issue fortunately did not make it to the upstream implementation, was 
only present in the Cludera fork of Flink. We will commit the original 
contribution without the issue. )

> Log dynamic properties on job submission
> 
>
> Key: FLINK-13957
> URL: https://issues.apache.org/jira/browse/FLINK-13957
> Project: Flink
>  Issue Type: Improvement
>  Components: Client / Job Submission
>Affects Versions: 1.9.0
>Reporter: Matyas Orhidi
>Assignee: Matyas Orhidi
>Priority: Major
>  Labels: log, security, sensitivity
> Fix For: 1.9.1
>
>
> This would let users understand their job submissions better, expected output 
> after suitable redactions from the {{FlinkYarnSessionCli:}} 
> {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore-password=*}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore=/opt/security/jks/truststore.jks}}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (FLINK-13957) Log dynamic properties on job submission

[Jira]

 [ 
https://issues.apache.org/jira/browse/FLINK-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi updated FLINK-13957:
---
Description: 

This would let users understand their job submissions better, expected output 
after suitable redactions from the {{FlinkYarnSessionCli:}} 

{{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
security.ssl.internal.truststore-password=*}}
 {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
security.ssl.internal.truststore=/opt/security/jks/truststore.jks}}


  was:
SSL related passwords specified by dynamic properties

{{flink run -m yarn-cluster -sae -p 2 -ynm HeapMonitor}}
 {{...}}
-yD security.ssl.internal.key-password=changeit
 {{-yD security.ssl.internal.keystore-password=}}{{changeit}}
 {{-yD security.ssl.internal.truststore-password=}}{{changeit}}

{{...}}

are showing up in {{FlinkYarnSessionCli}} logs in plain text:

{{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
security.ssl.internal.truststore-password=changeit}}
 {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
security.ssl.internal.keystore-password=changeit}}
 {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
security.ssl.internal.key-password=changeit}}


> Log dynamic properties on job submission
> 
>
> Key: FLINK-13957
> URL: https://issues.apache.org/jira/browse/FLINK-13957
> Project: Flink
>  Issue Type: Improvement
>  Components: Client / Job Submission
>Affects Versions: 1.9.0
>Reporter: Matyas Orhidi
>Assignee: Matyas Orhidi
>Priority: Major
>  Labels: log, security, sensitivity
> Fix For: 1.9.1
>
>
> This would let users understand their job submissions better, expected output 
> after suitable redactions from the {{FlinkYarnSessionCli:}} 
> {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore-password=*}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore=/opt/security/jks/truststore.jks}}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] YngwieWang commented on issue #9548: [hotfix] [docs] Fix typo

[GitBox]

YngwieWang commented on issue #9548: [hotfix] [docs] Fix typo
URL: https://github.com/apache/flink/pull/9548#issuecomment-527907617
 
 
   @1u0 , you can open a new PR with my changes, it's OK to me,  appreciate for 
your help.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] lirui-apache commented on issue #9615: [FLINK-13952][table-planner][hive] PartitionableTableSink can not wor…

[GitBox]

lirui-apache commented on issue #9615: [FLINK-13952][table-planner][hive] 
PartitionableTableSink can not wor…
URL: https://github.com/apache/flink/pull/9615#issuecomment-527908571
 
 
   cc @JingsongLi @bowenli86 @zjuwangg @xuefuz 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

tillrohrmann commented on a change in pull request #9607: [FLINK-13946] Remove 
job session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320768594
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
 ##
 @@ -146,21 +129,9 @@ private JobExecutorService 
createJobExecutorService(Configuration configuration)
return miniCluster;
}
 
-   @Override
-   public void stop() throws Exception {
-   synchronized (lock) {
-   if (jobExecutorService != null) {
-   jobExecutorService.close();
-   jobExecutorService = null;
-   }
-   }
-   }
-
-   @Override
-   public boolean isRunning() {
-   synchronized (lock) {
-   return jobExecutorService != null;
-   }
+   private void stopExecutorService(final JobExecutorService 
executorService) throws Exception {
 
 Review comment:
   The `JobExecutor` and the `JobExecutorService` have been introduced to 
bridge between the Flip-6 `MiniCluster` and the legacy `FlinkMiniCluster`. They 
should be obsolete now and could be removed if needed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9609: [FLINK-13750][client][coordination] Separate HA services between client-side and server-side

[GitBox]

flinkbot edited a comment on issue #9609: [FLINK-13750][client][coordination] 
Separate HA services between client-side and server-side
URL: https://github.com/apache/flink/pull/9609#issuecomment-527816485
 
 
   
   ## CI report:
   
   * a658808b883045ccb7d6c5124be2130de8a79fbd : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/125860328)
   * dad2525702bf24e0f0ecaf6f5f69a55f8da0b8f6 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125876658)
   * 1cb00f6d0ec01d7e2b5fd291eb4a503920888545 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/125878438)
   * e60ab79d65e04d597a7fd731c0fa08054ec5ac1d : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/12588)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot commented on issue #9615: [FLINK-13952][table-planner][hive] PartitionableTableSink can not wor…

[GitBox]

flinkbot commented on issue #9615: [FLINK-13952][table-planner][hive] 
PartitionableTableSink can not wor…
URL: https://github.com/apache/flink/pull/9615#issuecomment-527909466
 
 
   
   ## CI report:
   
   * b9cf01cfcebcc894450a7dd9d8c013108faccd6c : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320769875
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
 ##
 @@ -77,11 +70,11 @@
// 

 
public LocalExecutor() {
-   this(null);
+   this(new Configuration());
}
 
public LocalExecutor(Configuration conf) {
-   this.baseConfiguration = conf != null ? conf : new 
Configuration();
+   this.baseConfiguration = checkNotNull(conf);
 
 Review comment:
   I think you can only omit the `checkNotNull()` calls when your build system 
verifies that the `@Nonnull` contract is actually followed. Outside code, that 
is calling your function might still pass in a `null`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320773446
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
 ##
 @@ -178,65 +139,65 @@ public boolean isRunning() {
 */
@Override
public JobExecutionResult executePlan(Plan plan) throws Exception {
-   if (plan == null) {
-   throw new IllegalArgumentException("The plan may not be 
null.");
-   }
+   checkNotNull(plan);
 
-   synchronized (this.lock) {
+   final Configuration jobExecutorServiceConfiguration = 
configureExecution(plan);
 
-   // check if we start a session dedicated for this 
execution
-   final boolean shutDownAtEnd;
+   try (final JobExecutorService executorService = 
createJobExecutorService(jobExecutorServiceConfiguration)) {
 
-   if (jobExecutorService == null) {
-   shutDownAtEnd = true;
+   Optimizer pc = new Optimizer(new DataStatistics(), 
jobExecutorServiceConfiguration);
+   OptimizedPlan op = pc.compile(plan);
 
-   // configure the number of local slots equal to 
the parallelism of the local plan
-   if (this.taskManagerNumSlots == 
DEFAULT_TASK_MANAGER_NUM_SLOTS) {
-   int maxParallelism = 
plan.getMaximumParallelism();
-   if (maxParallelism > 0) {
-   this.taskManagerNumSlots = 
maxParallelism;
-   }
-   }
+   JobGraphGenerator jgg = new 
JobGraphGenerator(jobExecutorServiceConfiguration);
+   JobGraph jobGraph = jgg.compileJobGraph(op, 
plan.getJobId());
 
-   // start the cluster for us
-   start();
-   }
-   else {
-   // we use the existing session
-   shutDownAtEnd = false;
+   return executorService.executeJobBlocking(jobGraph);
+   }
+   }
+
+   private Configuration configureExecution(final Plan plan) {
+   setNumberOfTaskSlots(plan);
+   final Configuration executorConfiguration = 
createExecutorServiceConfig();
+   setPlanParallelism(plan, executorConfiguration);
+   return executorConfiguration;
+   }
+
+   private void setNumberOfTaskSlots(final Plan plan) {
 
 Review comment:
   This seems superfluous. We set `taskManagerNumSlots` only to use it in 
`createExecutorServiceConfig` to set the configuration. I think this can be 
replaced by a local variable and an argument to `createExecutorServiceConfig()`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320774791
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/RemoteExecutor.java
 ##
 @@ -32,13 +32,14 @@
 import org.apache.flink.optimizer.costs.DefaultCostEstimator;
 import org.apache.flink.optimizer.plan.OptimizedPlan;
 import org.apache.flink.optimizer.plandump.PlanJSONDumpGenerator;
-import org.apache.flink.util.NetUtils;
 
 import java.net.InetSocketAddress;
 import java.net.URL;
 import java.util.Collections;
 import java.util.List;
 
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
 
 Review comment:
   Nice simplifications in this file. 😊 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320774343
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
 ##
 @@ -178,65 +139,65 @@ public boolean isRunning() {
 */
@Override
public JobExecutionResult executePlan(Plan plan) throws Exception {
-   if (plan == null) {
-   throw new IllegalArgumentException("The plan may not be 
null.");
-   }
+   checkNotNull(plan);
 
-   synchronized (this.lock) {
+   final Configuration jobExecutorServiceConfiguration = 
configureExecution(plan);
 
-   // check if we start a session dedicated for this 
execution
-   final boolean shutDownAtEnd;
+   try (final JobExecutorService executorService = 
createJobExecutorService(jobExecutorServiceConfiguration)) {
 
-   if (jobExecutorService == null) {
-   shutDownAtEnd = true;
+   Optimizer pc = new Optimizer(new DataStatistics(), 
jobExecutorServiceConfiguration);
+   OptimizedPlan op = pc.compile(plan);
 
-   // configure the number of local slots equal to 
the parallelism of the local plan
-   if (this.taskManagerNumSlots == 
DEFAULT_TASK_MANAGER_NUM_SLOTS) {
-   int maxParallelism = 
plan.getMaximumParallelism();
-   if (maxParallelism > 0) {
-   this.taskManagerNumSlots = 
maxParallelism;
-   }
-   }
+   JobGraphGenerator jgg = new 
JobGraphGenerator(jobExecutorServiceConfiguration);
+   JobGraph jobGraph = jgg.compileJobGraph(op, 
plan.getJobId());
 
-   // start the cluster for us
-   start();
-   }
-   else {
-   // we use the existing session
-   shutDownAtEnd = false;
+   return executorService.executeJobBlocking(jobGraph);
+   }
+   }
+
+   private Configuration configureExecution(final Plan plan) {
+   setNumberOfTaskSlots(plan);
+   final Configuration executorConfiguration = 
createExecutorServiceConfig();
+   setPlanParallelism(plan, executorConfiguration);
+   return executorConfiguration;
+   }
+
+   private void setNumberOfTaskSlots(final Plan plan) {
+   if (this.taskManagerNumSlots == DEFAULT_TASK_MANAGER_NUM_SLOTS) 
{
+   int maxParallelism = plan.getMaximumParallelism();
+   if (maxParallelism > 0) {
+   this.taskManagerNumSlots = maxParallelism;
}
+   }
+   }
 
-   try {
-   // TODO: Set job's default parallelism to max 
number of slots
-   final int slotsPerTaskManager = 
jobExecutorServiceConfiguration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, 
taskManagerNumSlots);
-   final int numTaskManagers = 
jobExecutorServiceConfiguration.getInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER,
 1);
-   plan.setDefaultParallelism(slotsPerTaskManager 
* numTaskManagers);
+   private Configuration createExecutorServiceConfig() {
+   final Configuration newConfiguration = new Configuration();
+   newConfiguration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 
taskManagerNumSlots);
+   
newConfiguration.setBoolean(CoreOptions.FILESYTEM_DEFAULT_OVERRIDE, 
defaultOverwriteFiles);
 
-   Optimizer pc = new Optimizer(new 
DataStatistics(), jobExecutorServiceConfiguration);
-   OptimizedPlan op = pc.compile(plan);
+   newConfiguration.addAll(baseConfiguration);
 
-   JobGraphGenerator jgg = new 
JobGraphGenerator(jobExecutorServiceConfiguration);
-   JobGraph jobGraph = jgg.compileJobGraph(op, 
plan.getJobId());
+   return newConfiguration;
+   }
 
-   return 
jobExecutorService.executeJobBlocking(jobGraph);
-   }
-   finally {
-   if (shutDownAtEnd) {
-   stop();
-   }
-   }
-   }
+   private void setPlanParallelism(final Plan plan, final Configuration 
ex

[GitHub] [flink] aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320773696
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
 ##
 @@ -178,65 +139,65 @@ public boolean isRunning() {
 */
@Override
public JobExecutionResult executePlan(Plan plan) throws Exception {
-   if (plan == null) {
-   throw new IllegalArgumentException("The plan may not be 
null.");
-   }
+   checkNotNull(plan);
 
-   synchronized (this.lock) {
+   final Configuration jobExecutorServiceConfiguration = 
configureExecution(plan);
 
-   // check if we start a session dedicated for this 
execution
-   final boolean shutDownAtEnd;
+   try (final JobExecutorService executorService = 
createJobExecutorService(jobExecutorServiceConfiguration)) {
 
-   if (jobExecutorService == null) {
-   shutDownAtEnd = true;
+   Optimizer pc = new Optimizer(new DataStatistics(), 
jobExecutorServiceConfiguration);
+   OptimizedPlan op = pc.compile(plan);
 
-   // configure the number of local slots equal to 
the parallelism of the local plan
-   if (this.taskManagerNumSlots == 
DEFAULT_TASK_MANAGER_NUM_SLOTS) {
-   int maxParallelism = 
plan.getMaximumParallelism();
-   if (maxParallelism > 0) {
-   this.taskManagerNumSlots = 
maxParallelism;
-   }
-   }
+   JobGraphGenerator jgg = new 
JobGraphGenerator(jobExecutorServiceConfiguration);
+   JobGraph jobGraph = jgg.compileJobGraph(op, 
plan.getJobId());
 
-   // start the cluster for us
-   start();
-   }
-   else {
-   // we use the existing session
-   shutDownAtEnd = false;
+   return executorService.executeJobBlocking(jobGraph);
+   }
+   }
+
+   private Configuration configureExecution(final Plan plan) {
+   setNumberOfTaskSlots(plan);
+   final Configuration executorConfiguration = 
createExecutorServiceConfig();
+   setPlanParallelism(plan, executorConfiguration);
+   return executorConfiguration;
+   }
+
+   private void setNumberOfTaskSlots(final Plan plan) {
+   if (this.taskManagerNumSlots == DEFAULT_TASK_MANAGER_NUM_SLOTS) 
{
+   int maxParallelism = plan.getMaximumParallelism();
+   if (maxParallelism > 0) {
+   this.taskManagerNumSlots = maxParallelism;
}
+   }
+   }
 
-   try {
-   // TODO: Set job's default parallelism to max 
number of slots
-   final int slotsPerTaskManager = 
jobExecutorServiceConfiguration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, 
taskManagerNumSlots);
-   final int numTaskManagers = 
jobExecutorServiceConfiguration.getInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER,
 1);
-   plan.setDefaultParallelism(slotsPerTaskManager 
* numTaskManagers);
+   private Configuration createExecutorServiceConfig() {
+   final Configuration newConfiguration = new Configuration();
+   newConfiguration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 
taskManagerNumSlots);
+   
newConfiguration.setBoolean(CoreOptions.FILESYTEM_DEFAULT_OVERRIDE, 
defaultOverwriteFiles);
 
 Review comment:
   I only see usage of this setter in one single test, maybe we can also remove 
that or at least not use a field for this anymore.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

aljoscha commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320775256
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
 ##
 @@ -72,15 +71,9 @@ public String getExecutionPlan() throws Exception {
return gen.getOptimizerPlanAsJSON(op);
}
 
-   @Override
-   public void startNewSession() throws Exception {
-   jobID = JobID.generate();
-   }
-
@Override
public String toString() {
-   return "Context Environment (parallelism = " + 
(getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT ? "default" : 
getParallelism())
-   + ") : " + getIdString();
+   return "Context Environment (parallelism = " + 
(getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT ? "default" : 
getParallelism()) + ").";
 
 Review comment:
   I wouldn't put a `.` here, I don't think it's a sentence


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9615: [FLINK-13952][table-planner][hive] PartitionableTableSink can not wor…

[GitBox]

flinkbot edited a comment on issue #9615: [FLINK-13952][table-planner][hive] 
PartitionableTableSink can not wor…
URL: https://github.com/apache/flink/pull/9615#issuecomment-527909466
 
 
   
   ## CI report:
   
   * b9cf01cfcebcc894450a7dd9d8c013108faccd6c : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125899825)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann closed pull request #9609: [FLINK-13750][client][coordination] Separate HA services between client-side and server-side

[GitBox]

tillrohrmann closed pull request #9609: [FLINK-13750][client][coordination] 
Separate HA services between client-side and server-side
URL: https://github.com/apache/flink/pull/9609
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9409: [FLINK-13672][ml] Add class for VectorInteractionMapper

[GitBox]

flinkbot edited a comment on issue #9409: [FLINK-13672][ml] Add class for 
VectorInteractionMapper
URL: https://github.com/apache/flink/pull/9409#issuecomment-519961173
 
 
   
   ## CI report:
   
   * 9b5e1620eb70dd95996ec3e8b7bebe1f38ac89a5 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122611642)
   * 1d402f4ed866db5e9bbd35496949f8fdc8197c5e : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893871)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9411: [FLINK-13674][ml] Add class of Vector Size Hint Mapper

[GitBox]

flinkbot edited a comment on issue #9411: [FLINK-13674][ml] Add class of Vector 
Size Hint Mapper
URL: https://github.com/apache/flink/pull/9411#issuecomment-519965855
 
 
   
   ## CI report:
   
   * 4af38a946a55d1d6b68d7d249f87a65239c947f7 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122613586)
   * 5e9e66fcf02dc111a807661502761e1a67a043eb : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893971)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on issue #9609: [FLINK-13750][client][coordination] Separate HA services between client-side and server-side

[GitBox]

TisonKun commented on issue #9609: [FLINK-13750][client][coordination] Separate 
HA services between client-side and server-side
URL: https://github.com/apache/flink/pull/9609#issuecomment-527917668
 
 
   Thanks for your review @tillrohrmann!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator

[GitBox]

flinkbot edited a comment on issue #9564: 
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox 
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#issuecomment-526487220
 
 
   
   ## CI report:
   
   * 3c6c3d69983136fb8bfa11b28a7ba783dfe61e52 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125188905)
   * a9d65d8fabd938d47abd801a4686cb185f2fcf68 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125206763)
   * 1bbbce7f1f81598cb5a454f0e4d25adae6ed75b4 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/125425831)
   * d68aa63cea98320f14ff9bbe483776c8873a77ca : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125431268)
   * 097e1ec9e9e64ad6e9e1de16bc2937acb84ab42e : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/125726843)
   * 9bc0b14a50cf03c23eb4203aa2637130572047f9 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125739773)
   * ee6669575eb9481fd7181c3a9be52e615f613e8f : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125755445)
   * 712fdb428e78fc0d47262c6c755429a37a6be3eb : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125860362)
   * 16bac68a7fd82876d0ad2d56c8e7c199dc816d04 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/125884245)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (FLINK-13750) Separate HA services between client-/ and server-side

[Till Rohrmann (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-13750.
---
Fix Version/s: 1.10.0
 Release Note: The `HighAvailabilityServices` have been split up into 
client side `ClientHighAvailabilityServices` and cluster side 
`HighAvailabilityServices`. When implementing your own high availability 
services, you should follow this separation by overriding the factory method 
`HighAvailabilityServicesFactory.createClientHAServices`.
   Resolution: Done

Done via e0759c3776e7292e4a4937c6fb76d31ce7bf702d

> Separate HA services between client-/ and server-side
> -
>
> Key: FLINK-13750
> URL: https://issues.apache.org/jira/browse/FLINK-13750
> Project: Flink
>  Issue Type: Improvement
>  Components: Command Line Client, Runtime / Coordination
>Reporter: Chesnay Schepler
>Assignee: TisonKun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, we use the same {{HighAvailabilityServices}} on the client and 
> server. However, the client does not need several of the features that the 
> services currently provide (access to the blobstore or checkpoint metadata).
> Additionally, due to how these services are setup they also require the 
> client to have access to the blob storage, despite it never actually being 
> used, which can cause issues, like FLINK-13500.
> [~Tison] Would be be interested in this issue?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-13960) Provide default implementation for deprecated HighAvailabilityServices.getWebMonitorLeaderRetriever

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-13960:
-

 Summary: Provide default implementation for deprecated 
HighAvailabilityServices.getWebMonitorLeaderRetriever
 Key: FLINK-13960
 URL: https://issues.apache.org/jira/browse/FLINK-13960
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.10.0


With FLINK-13750 we split the {{HighAvailabilityServices}} up into a client 
side {{ClientHighAvailabilitySerivces}} and cluster side 
{{HighAvailabilityServices}}. As part of this split, the method 
{{HighAvailabilityServices.getWebMonitorLeaderRetriever}} was deprecated. In 
order to not having to implement this method anymore, I suggest to provide a 
default implementation which throws an {{UnsupportedOperationException}} 
referring to the {{ClientHighAvailabilityServices}}.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] morhidi opened a new pull request #9616: [FLINK-13957] [flink-yarn] Log dynamic properties on job submission

[GitBox]

morhidi opened a new pull request #9616: [FLINK-13957] [flink-yarn] Log dynamic 
properties on job submission
URL: https://github.com/apache/flink/pull/9616
 
 
   
   
   ## What is the purpose of the change
   
   This would let users understand their job submissions better, expected 
output after suitable redactions from the FlinkYarnSessionCli
   
   ```
   19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
security.ssl.internal.truststore-password=*
   19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
security.ssl.internal.truststore=/opt/security/jks/truststore.jks
   ```
   
   ## Brief change log
   
 - FlinkYarnSessionCli dumps dynamic properties using 
*GlobalConfiguration.isSensitive(key)* logic for redaction
 - Added a new password property to 
*FlinkYarnSessionCliTest.testDynamicProperties *
   
   
   ## Verifying this change
   
   This change is already covered by existing tests, 
*FlinkYarnSessionCliTest.testDynamicProperties*.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-13957) Log dynamic properties on job submission

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-13957:
---
Labels: log pull-request-available security sensitivity  (was: log security 
sensitivity)

> Log dynamic properties on job submission
> 
>
> Key: FLINK-13957
> URL: https://issues.apache.org/jira/browse/FLINK-13957
> Project: Flink
>  Issue Type: Improvement
>  Components: Client / Job Submission
>Affects Versions: 1.9.0
>Reporter: Matyas Orhidi
>Assignee: Matyas Orhidi
>Priority: Major
>  Labels: log, pull-request-available, security, sensitivity
> Fix For: 1.9.1
>
>
> This would let users understand their job submissions better, expected output 
> after suitable redactions from the {{FlinkYarnSessionCli:}} 
> {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore-password=*}}
>  {{19/09/04 04:57:43 INFO cli.FlinkYarnSessionCli: Dynamic Property set: 
> security.ssl.internal.truststore=/opt/security/jks/truststore.jks}}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] flinkbot commented on issue #9616: [FLINK-13957] [flink-yarn] Log dynamic properties on job submission

[GitBox]

flinkbot commented on issue #9616: [FLINK-13957] [flink-yarn] Log dynamic 
properties on job submission
URL: https://github.com/apache/flink/pull/9616#issuecomment-527922323
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit b9114df609d2f21117e431ed36ae647a9f3a93a5 (Wed Sep 04 
14:19:37 UTC 2019)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-13958) Job class loader may not be reused after batch job recovery

[Alex (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922545#comment-16922545
 ] 

Alex commented on FLINK-13958:
--

I think this has the the same root cause as FLINK-11402. Specifically, that we 
cannot load a native library more than once in the same JVM process.

> Job class loader may not be reused after batch job recovery
> ---
>
> Key: FLINK-13958
> URL: https://issues.apache.org/jira/browse/FLINK-13958
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Task
>Affects Versions: 1.9.0
>Reporter: David Moravek
>Priority: Major
>
> [https://lists.apache.org/thread.html/e241be9a1a10810a1203786dff3b7386d265fbe8702815a77bad42eb@%3Cdev.flink.apache.org%3E|http://example.com]
> 1) We have a per-job flink cluster
> 2) We use BATCH execution mode + region failover strategy
> Point 1) should imply single user code class loader per task manager (because 
> there is only single pipeline, that reuses class loader cached in 
> BlobLibraryCacheManager). We need this property, because we have UDFs that 
> access C libraries using JNI (I think this may be fairly common use-case when 
> dealing with legacy code). [JDK 
> internals|https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/ClassLoader.java#L2466]
>  make sure that single library can be only loaded by a single class loader 
> per JVM.
> When region recovery is triggered, vertices that need recover are first reset 
> back to CREATED stated and then rescheduled. In case all tasks in a task 
> manager are reset, this results in [cached class loader being 
> released|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java#L338].
>  This unfortunately causes job failure, because we try to reload a native 
> library in a newly created class loader.
> I believe the correct approach would be not to release cached class loader if 
> the job is recovering, even though there are no tasks currently registered 
> with TM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320787256
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
 ##
 @@ -77,11 +70,11 @@
// 

 
public LocalExecutor() {
-   this(null);
+   this(new Configuration());
}
 
public LocalExecutor(Configuration conf) {
-   this.baseConfiguration = conf != null ? conf : new 
Configuration();
+   this.baseConfiguration = checkNotNull(conf);
 
 Review comment:
   @aljoscha thanks for your input. I agree that we could both make use of 
compile time constraint for internal development and place runtime checker when 
input may be from outside.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9614: [FLINK-13623][Tests] Update StatefulJobSavepointMigrationITCase to restore from 1.9 savepoint

[GitBox]

flinkbot edited a comment on issue #9614: [FLINK-13623][Tests] Update 
StatefulJobSavepointMigrationITCase to restore from 1.9 savepoint
URL: https://github.com/apache/flink/pull/9614#issuecomment-527883601
 
 
   
   ## CI report:
   
   * e96c273e30e6a6fa2a55541668b5dd5cda5a34a0 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/125888732)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot commented on issue #9616: [FLINK-13957] [flink-yarn] Log dynamic properties on job submission

[GitBox]

flinkbot commented on issue #9616: [FLINK-13957] [flink-yarn] Log dynamic 
properties on job submission
URL: https://github.com/apache/flink/pull/9616#issuecomment-527924104
 
 
   
   ## CI report:
   
   * b9114df609d2f21117e431ed36ae647a9f3a93a5 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9405: [FLINK-13668][ml] Add abstract classes for three typical scenarios of (Flat)Mapper

[GitBox]

flinkbot edited a comment on issue #9405: [FLINK-13668][ml] Add abstract 
classes for three typical scenarios of (Flat)Mapper
URL: https://github.com/apache/flink/pull/9405#issuecomment-519901201
 
 
   
   ## CI report:
   
   * c1fedc4de6a92bc2c5f84d0e0c3038f29fd8944f : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122585906)
   * 7f235ecf56417da32c78bb5d064c9d833e1d5261 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893682)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9406: [FLINK-13669][ml] Add class for BinarizerMapper

[GitBox]

flinkbot edited a comment on issue #9406: [FLINK-13669][ml] Add class for 
BinarizerMapper
URL: https://github.com/apache/flink/pull/9406#issuecomment-519954683
 
 
   
   ## CI report:
   
   * eb27a4f02fee6d3d96a102ef7b4734d06442596a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122608982)
   * 8a8a364e5b392e9251dfcec4241dd162bf62d089 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893714)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9407: [FLINK-13670][ml] Add class for VectorAssemblerMapper

[GitBox]

flinkbot edited a comment on issue #9407: [FLINK-13670][ml] Add class for 
VectorAssemblerMapper
URL: https://github.com/apache/flink/pull/9407#issuecomment-51995
 
 
   
   ## CI report:
   
   * bf69f9f25f8a62a494f530e7cba75879ac002412 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122610736)
   * e92b935e874524c967a9b0296833872c602d0a33 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893754)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9408: [FLINK-13671][ml] Add class for VectorEleWiseProductMapper

[GitBox]

flinkbot edited a comment on issue #9408: [FLINK-13671][ml] Add class for 
VectorEleWiseProductMapper
URL: https://github.com/apache/flink/pull/9408#issuecomment-519961164
 
 
   
   ## CI report:
   
   * fbf9704fe2812d28d7a6b9ca6272a260e91089b3 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122611669)
   * a2091168929daf552e6e839c9ef179c983192fcf : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893825)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320791574
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
 ##
 @@ -146,21 +129,9 @@ private JobExecutorService 
createJobExecutorService(Configuration configuration)
return miniCluster;
}
 
-   @Override
-   public void stop() throws Exception {
-   synchronized (lock) {
-   if (jobExecutorService != null) {
-   jobExecutorService.close();
-   jobExecutorService = null;
-   }
-   }
-   }
-
-   @Override
-   public boolean isRunning() {
-   synchronized (lock) {
-   return jobExecutorService != null;
-   }
+   private void stopExecutorService(final JobExecutorService 
executorService) throws Exception {
 
 Review comment:
   Thanks for you information @tillrohrmann. @kl0u I'd like to make this 
removal a separated issue, fyi FLINK-13961.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-13958) Job class loader may not be reused after batch job recovery

[David Moravek (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922554#comment-16922554
 ] 

David Moravek commented on FLINK-13958:
---

[~1u0] It's unrelated issue. The behavior you are describing is expected when 
you submit multiple jobs into the same cluster. Only option in that case is to 
workaround using system class loader (it's technically not a workaround).

> Job class loader may not be reused after batch job recovery
> ---
>
> Key: FLINK-13958
> URL: https://issues.apache.org/jira/browse/FLINK-13958
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Task
>Affects Versions: 1.9.0
>Reporter: David Moravek
>Priority: Major
>
> [https://lists.apache.org/thread.html/e241be9a1a10810a1203786dff3b7386d265fbe8702815a77bad42eb@%3Cdev.flink.apache.org%3E|http://example.com]
> 1) We have a per-job flink cluster
> 2) We use BATCH execution mode + region failover strategy
> Point 1) should imply single user code class loader per task manager (because 
> there is only single pipeline, that reuses class loader cached in 
> BlobLibraryCacheManager). We need this property, because we have UDFs that 
> access C libraries using JNI (I think this may be fairly common use-case when 
> dealing with legacy code). [JDK 
> internals|https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/ClassLoader.java#L2466]
>  make sure that single library can be only loaded by a single class loader 
> per JVM.
> When region recovery is triggered, vertices that need recover are first reset 
> back to CREATED stated and then rescheduled. In case all tasks in a task 
> manager are reset, this results in [cached class loader being 
> released|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java#L338].
>  This unfortunately causes job failure, because we try to reload a native 
> library in a newly created class loader.
> I believe the correct approach would be not to release cached class loader if 
> the job is recovering, even though there are no tasks currently registered 
> with TM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-13961) Remove obsolete abstraction JobExecutor(Service)

[TisonKun (Jira)]

TisonKun created FLINK-13961:


 Summary: Remove obsolete abstraction JobExecutor(Service) 
 Key: FLINK-13961
 URL: https://issues.apache.org/jira/browse/FLINK-13961
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Affects Versions: 1.10.0
Reporter: TisonKun
 Fix For: 1.10.0


Refer to Till's comment

The JobExecutor and the JobExecutorService have been introduced to bridge 
between the Flip-6 MiniCluster and the legacy FlinkMiniCluster. They should be 
obsolete now and could be removed if needed.

Actually we should make used of {{MiniClusterClient}} for submission ideally 
but we have some tests based on MiniCluster in flink-runtime or somewhere that 
doesn't have a dependency to flink-client; while move {{MiniClusterClient}} to 
flink-runtime is unclear whether reasonable or not. Thus I'd prefer keep 
{{executeJobBlocking}} for now and defer the possible refactor.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320792153
 
 

 ##
 File path: 
flink-clients/src/main/java/org/apache/flink/client/RemoteExecutor.java
 ##
 @@ -178,49 +127,30 @@ public boolean isRunning() {
 
@Override
public JobExecutionResult executePlan(Plan plan) throws Exception {
-   if (plan == null) {
-   throw new IllegalArgumentException("The plan may not be 
null.");
-   }
+   checkNotNull(plan);
 
JobWithJars p = new JobWithJars(plan, this.jarFiles, 
this.globalClasspaths);
return executePlanWithJars(p);
}
 
-   public JobExecutionResult executePlanWithJars(JobWithJars program) 
throws Exception {
-   if (program == null) {
-   throw new IllegalArgumentException("The job may not be 
null.");
-   }
-
-   synchronized (this.lock) {
-   // check if we start a session dedicated for this 
execution
-   final boolean shutDownAtEnd;
-
-   if (client == null) {
-   shutDownAtEnd = true;
-   // start the executor for us
-   start();
-   }
-   else {
-   // we use the existing session
-   shutDownAtEnd = false;
-   }
+   private JobExecutionResult executePlanWithJars(JobWithJars program) 
throws Exception {
+   checkNotNull(program);
 
-   try {
-   return client.run(program, 
defaultParallelism).getJobExecutionResult();
-   }
-   finally {
-   if (shutDownAtEnd) {
-   stop();
-   }
+   ClusterClient  client = null;
+   try {
+   client = startClusterClient();
+   return client.run(program, 
defaultParallelism).getJobExecutionResult();
+   } finally {
+   if (client != null) {
+   stopClusterClient(client);
 
 Review comment:
   I think the latest CI fails on uncompleted refactor
   
   ```
   12:40:29.667 [ERROR] 
/home/travis/build/flink-ci/flink/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ProgramDeployer.java:[134,54]
 cannot find symbol
 symbol:   method shutdown()
 location: variable clusterClient of type 
org.apache.flink.client.program.ClusterClient
   12:40:29.668 [ERROR] 
/home/travis/build/flink-ci/flink/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ProgramDeployer.java:[175,54]
 cannot find symbol
 symbol:   method shutdown()
 location: variable clusterClient of type 
org.apache.flink.client.program.ClusterClient
   12:40:29.669 [ERROR] 
/home/travis/build/flink-ci/flink/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java:[434,62]
 cannot find symbol
 symbol:   method shutdown()
 location: variable clusterClient of type 
org.apache.flink.client.program.ClusterClient
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (FLINK-13962) Execution#taskRestore leaks if task fails before deploying

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-13962:
---

 Summary: Execution#taskRestore leaks if task fails before deploying
 Key: FLINK-13962
 URL: https://issues.apache.org/jira/browse/FLINK-13962
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.9.0, 1.10.0
Reporter: Zhu Zhu


Currently Execution#taskRestore is reset to null in task deployment stage.

The purpose of it is "allows the JobManagerTaskRestore instance to be garbage 
collected. Furthermore, it won't be archived along with the Execution in the 
ExecutionVertex in case of a restart. This is especially important when setting 
state.backend.fs.memory-threshold to larger values because every state below 
this threshold will be stored in the meta state files and, thus, also the 
JobManagerTaskRestore instances." (From FLINK-9693)

 

However, if a task fails before it comes to the deployment stage, the 
Execution#taskRestore will remain non-null and will be archived in prior 
executions. 

This may result in large JM heap cost in certain cases.

 

I think we should check the Execution#taskRestore and make sure it is null when 
moving a execution to prior executions.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job session related code from ExecutionEnvironment

[GitBox]

TisonKun commented on a change in pull request #9607: [FLINK-13946] Remove job 
session related code from ExecutionEnvironment
URL: https://github.com/apache/flink/pull/9607#discussion_r320792694
 
 

 ##
 File path: 
flink-java/src/main/java/org/apache/flink/api/java/LocalEnvironment.java
 ##
 @@ -71,150 +63,34 @@ public LocalEnvironment(Configuration config) {
"The LocalEnvironment cannot be 
instantiated when running in a pre-defined context " +
"(such as Command Line 
Client, Scala Shell, or TestEnvironment)");
}
-   this.configuration = config == null ? new Configuration() : 
config;
+   this.configuration = checkNotNull(config);
}
 
// 

 
+   // TODO: 31.08.19 make sure that start and stop are called in the 
execute.
+   // the other place would be here, but this can complicate code, as the
+   // lifecycle management would be outside the executor itself.
+
@Override
public JobExecutionResult execute(String jobName) throws Exception {
-   if (executor == null) {
-   startNewSession();
-   }
-
-   Plan p = createProgramPlan(jobName);
+   final Plan p = createProgramPlan(jobName);
 
-   // Session management is disabled, revert this commit to enable
-   //p.setJobId(jobID);
-   //p.setSessionTimeout(sessionTimeout);
-
-   JobExecutionResult result = executor.executePlan(p);
-
-   this.lastJobExecutionResult = result;
-   return result;
+   // TODO: 31.08.19 make the executor autocloseable
 
 Review comment:
   Thanks for the explanation. Sounds reasonable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (FLINK-13963) Consolidate Hadoop file systems usage and Hadoop integration docs

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-13963:
---

 Summary: Consolidate Hadoop file systems usage and Hadoop 
integration docs
 Key: FLINK-13963
 URL: https://issues.apache.org/jira/browse/FLINK-13963
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem, Connectors / Hadoop 
Compatibility, Documentation, FileSystems
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.10.0


We have hadoop related docs in several places at the moment:
 * *dev/batch/connectors.md* (Hadoop FS implementations and setup)
 * *dev/batch/hadoop_compatibility.md* (not valid any more that Flink always 
has Hadoop types out of the box as we do not build and provide Flink with 
Hadoop by default)
 * *ops/filesystems/index.md* (plugins, Hadoop FS implementations and setup 
revisited)
 * *ops/deployment/hadoop.md* (Hadoop classpath)
 * *ops/config.md* (deprecated way to provide Hadoop configuration in Flink 
conf)

We could consolidate all these pieces of docs into a consistent structure to 
help users to navigate through the docs to well-defined spots depending on 
which feature they are trying to use.

The places in docs which should contain the information about Hadoop:
 * *dev/batch/hadoop_compatibility.md* (only Dataset API specific stuff about 
integration with Hadoop)
 * *ops/filesystems/index.md* (Flink FS plugins and Hadoop FS implementations)
 * *ops/deployment/hadoop.md* (Hadoop configuration and classpath)

How to setup Hadoop itself should be only in *ops/deployment/hadoop.md*. All 
other places dealing with Hadoop/HDFS should contain only their related things 
and just reference it 'how to configure Hadoop'. Like all chapters about 
writing to file systems (batch connectors and streaming file sinks) should just 
reference file systems.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (FLINK-11402) User code can fail with an UnsatisfiedLinkError in the presence of multiple classloaders

[David Moravek (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922555#comment-16922555
 ] 

David Moravek commented on FLINK-11402:
---

This is a [JVM 
limitation|https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/ClassLoader.java#L2466]
 and you have to make sure to load native libraries using system class loader 
in case you're submitting multiple jobs into the same cluster.

> User code can fail with an UnsatisfiedLinkError in the presence of multiple 
> classloaders
> 
>
> Key: FLINK-11402
> URL: https://issues.apache.org/jira/browse/FLINK-11402
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Coordination, Runtime / Task
>Affects Versions: 1.7.0
>Reporter: Ufuk Celebi
>Priority: Major
> Attachments: hello-snappy-1.0-SNAPSHOT.jar, hello-snappy.tgz
>
>
> As reported on the user mailing list thread "[`env.java.opts` not persisting 
> after job canceled or failed and then 
> restarted|https://lists.apache.org/thread.html/37cc1b628e16ca6c0bacced5e825de8057f88a8d601b90a355b6a291@%3Cuser.flink.apache.org%3E]";,
>  there can be issues with using native libraries and user code class loading.
> h2. Steps to reproduce
> I was able to reproduce the issue reported on the mailing list using 
> [snappy-java|https://github.com/xerial/snappy-java] in a user program. 
> Running the attached user program works fine on initial submission, but 
> results in a failure when re-executed.
> I'm using Flink 1.7.0 using a standalone cluster started via 
> {{bin/start-cluster.sh}}.
> 0. Unpack attached Maven project and build using {{mvn clean package}} *or* 
> directly use attached {{hello-snappy-1.0-SNAPSHOT.jar}}
> 1. Download 
> [snappy-java-1.1.7.2.jar|http://central.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.7.2/snappy-java-1.1.7.2.jar]
>  and unpack libsnappyjava for your system:
> {code}
> jar tf snappy-java-1.1.7.2.jar | grep libsnappy
> ...
> org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so
> ...
> org/xerial/snappy/native/Mac/x86_64/libsnappyjava.jnilib
> ...
> {code}
> 2. Configure system library path to {{libsnappyjava}} in {{flink-conf.yaml}} 
> (path needs to be adjusted for your system):
> {code}
> env.java.opts: -Djava.library.path=/.../org/xerial/snappy/native/Mac/x86_64
> {code}
> 3. Run attached {{hello-snappy-1.0-SNAPSHOT.jar}}
> {code}
> bin/flink run hello-snappy-1.0-SNAPSHOT.jar
> Starting execution of program
> Program execution finished
> Job with JobID ae815b918dd7bc64ac8959e4e224f2b4 has finished.
> Job Runtime: 359 ms
> {code}
> 4. Rerun attached {{hello-snappy-1.0-SNAPSHOT.jar}}
> {code}
> bin/flink run hello-snappy-1.0-SNAPSHOT.jar
> Starting execution of program
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Job failed. 
> (JobID: 7d69baca58f33180cb9251449ddcd396)
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>   at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>   at 
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>   at com.github.uce.HelloSnappy.main(HelloSnappy.java:18)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>   at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>   at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>   at 
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
>   at 
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>   at 
> org.apache.flink.cl

[jira] [Updated] (FLINK-13962) Execution#taskRestore leaks if task fails before deploying

[Zhu Zhu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhu Zhu updated FLINK-13962:

Description: 
Currently Execution#taskRestore is reset to null in task deployment stage.

The purpose of it is "allows the JobManagerTaskRestore instance to be garbage 
collected. Furthermore, it won't be archived along with the Execution in the 
ExecutionVertex in case of a restart. This is especially important when setting 
state.backend.fs.memory-threshold to larger values because every state below 
this threshold will be stored in the meta state files and, thus, also the 
JobManagerTaskRestore instances." (From FLINK-9693)

 

However, if a task fails before it comes to the deployment stage(e.g. fails due 
to slot allocation timeout), the Execution#taskRestore will remain non-null and 
will be archived in prior executions. 

This may result in large JM heap cost in certain cases.

 

I think we should check the Execution#taskRestore and make sure it is null when 
moving a execution to prior executions.

  was:
Currently Execution#taskRestore is reset to null in task deployment stage.

The purpose of it is "allows the JobManagerTaskRestore instance to be garbage 
collected. Furthermore, it won't be archived along with the Execution in the 
ExecutionVertex in case of a restart. This is especially important when setting 
state.backend.fs.memory-threshold to larger values because every state below 
this threshold will be stored in the meta state files and, thus, also the 
JobManagerTaskRestore instances." (From FLINK-9693)

 

However, if a task fails before it comes to the deployment stage, the 
Execution#taskRestore will remain non-null and will be archived in prior 
executions. 

This may result in large JM heap cost in certain cases.

 

I think we should check the Execution#taskRestore and make sure it is null when 
moving a execution to prior executions.


> Execution#taskRestore leaks if task fails before deploying
> --
>
> Key: FLINK-13962
> URL: https://issues.apache.org/jira/browse/FLINK-13962
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
>
> Currently Execution#taskRestore is reset to null in task deployment stage.
> The purpose of it is "allows the JobManagerTaskRestore instance to be garbage 
> collected. Furthermore, it won't be archived along with the Execution in the 
> ExecutionVertex in case of a restart. This is especially important when 
> setting state.backend.fs.memory-threshold to larger values because every 
> state below this threshold will be stored in the meta state files and, thus, 
> also the JobManagerTaskRestore instances." (From FLINK-9693)
>  
> However, if a task fails before it comes to the deployment stage(e.g. fails 
> due to slot allocation timeout), the Execution#taskRestore will remain 
> non-null and will be archived in prior executions. 
> This may result in large JM heap cost in certain cases.
>  
> I think we should check the Execution#taskRestore and make sure it is null 
> when moving a execution to prior executions.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (FLINK-13958) Job class loader may not be reused after batch job recovery

[Alex (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922545#comment-16922545
 ] 

Alex edited comment on FLINK-13958 at 9/4/19 2:35 PM:
--

I think this has the same root cause as FLINK-11402. Specifically, that we 
cannot load a native library more than once in the same JVM process.


was (Author: 1u0):
I think this has the the same root cause as FLINK-11402. Specifically, that we 
cannot load a native library more than once in the same JVM process.

> Job class loader may not be reused after batch job recovery
> ---
>
> Key: FLINK-13958
> URL: https://issues.apache.org/jira/browse/FLINK-13958
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Task
>Affects Versions: 1.9.0
>Reporter: David Moravek
>Priority: Major
>
> [https://lists.apache.org/thread.html/e241be9a1a10810a1203786dff3b7386d265fbe8702815a77bad42eb@%3Cdev.flink.apache.org%3E|http://example.com]
> 1) We have a per-job flink cluster
> 2) We use BATCH execution mode + region failover strategy
> Point 1) should imply single user code class loader per task manager (because 
> there is only single pipeline, that reuses class loader cached in 
> BlobLibraryCacheManager). We need this property, because we have UDFs that 
> access C libraries using JNI (I think this may be fairly common use-case when 
> dealing with legacy code). [JDK 
> internals|https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/ClassLoader.java#L2466]
>  make sure that single library can be only loaded by a single class loader 
> per JVM.
> When region recovery is triggered, vertices that need recover are first reset 
> back to CREATED stated and then rescheduled. In case all tasks in a task 
> manager are reset, this results in [cached class loader being 
> released|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java#L338].
>  This unfortunately causes job failure, because we try to reload a native 
> library in a newly created class loader.
> I believe the correct approach would be not to release cached class loader if 
> the job is recovering, even though there are no tasks currently registered 
> with TM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (FLINK-13953) Facilitate enabling new Scheduler in MiniCluster Tests

[Gary Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao reassigned FLINK-13953:


Assignee: Andy  (was: Gary Yao)

> Facilitate enabling new Scheduler in MiniCluster Tests
> --
>
> Key: FLINK-13953
> URL: https://issues.apache.org/jira/browse/FLINK-13953
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination, Tests
>Reporter: Gary Yao
>Assignee: Andy
>Priority: Major
>
> Currently, tests using the {{MiniCluster}} use the legacy scheduler by 
> default. Once the new scheduler is implemented, we should run tests with the 
> new scheduler enabled. However, it is not expected that all tests will pass 
> immediately. Therefore, it should be possible to enable the new scheduler for 
> a subset of tests. 
> *Acceptance Criteria*
> * Subset of tests using {{MiniCluster}} can be run on a per-commit basis (on 
> Travis) against new scheduler and also legacy scheduler 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (FLINK-13953) Facilitate enabling new Scheduler in MiniCluster Tests

[Gary Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao reassigned FLINK-13953:


Assignee: Zhu Zhu

> Facilitate enabling new Scheduler in MiniCluster Tests
> --
>
> Key: FLINK-13953
> URL: https://issues.apache.org/jira/browse/FLINK-13953
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination, Tests
>Reporter: Gary Yao
>Assignee: Zhu Zhu
>Priority: Major
>
> Currently, tests using the {{MiniCluster}} use the legacy scheduler by 
> default. Once the new scheduler is implemented, we should run tests with the 
> new scheduler enabled. However, it is not expected that all tests will pass 
> immediately. Therefore, it should be possible to enable the new scheduler for 
> a subset of tests. 
> *Acceptance Criteria*
> * Subset of tests using {{MiniCluster}} can be run on a per-commit basis (on 
> Travis) against new scheduler and also legacy scheduler 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (FLINK-13953) Facilitate enabling new Scheduler in MiniCluster Tests

[Gary Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao reassigned FLINK-13953:


Assignee: (was: Andy)

> Facilitate enabling new Scheduler in MiniCluster Tests
> --
>
> Key: FLINK-13953
> URL: https://issues.apache.org/jira/browse/FLINK-13953
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination, Tests
>Reporter: Gary Yao
>Priority: Major
>
> Currently, tests using the {{MiniCluster}} use the legacy scheduler by 
> default. Once the new scheduler is implemented, we should run tests with the 
> new scheduler enabled. However, it is not expected that all tests will pass 
> immediately. Therefore, it should be possible to enable the new scheduler for 
> a subset of tests. 
> *Acceptance Criteria*
> * Subset of tests using {{MiniCluster}} can be run on a per-commit basis (on 
> Travis) against new scheduler and also legacy scheduler 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] flinkbot edited a comment on issue #9616: [FLINK-13957] [flink-yarn] Log dynamic properties on job submission

[GitBox]

flinkbot edited a comment on issue #9616: [FLINK-13957] [flink-yarn] Log 
dynamic properties on job submission
URL: https://github.com/apache/flink/pull/9616#issuecomment-527924104
 
 
   
   ## CI report:
   
   * b9114df609d2f21117e431ed36ae647a9f3a93a5 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125906000)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (FLINK-13919) Remove 1.6.4 from downloads page and SVN

[Chesnay Schepler (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-13919.

Resolution: Fixed

asf-site: a00510f26e1065b4f0c05524caf735ce8dba372a

> Remove 1.6.4 from downloads page and SVN
> 
>
> Key: FLINK-13919
> URL: https://issues.apache.org/jira/browse/FLINK-13919
> Project: Flink
>  Issue Type: Improvement
>  Components: Project Website
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
>
> 1.6.4 is still listed on the download page, and the release artifacts are 
> still present on dist.apache.org .
> Given that we no longer support 1.6 and the last release (1.6.4) was 6 months 
> ago I would say it is time to remove these.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] flinkbot edited a comment on issue #9410: [FLINK-13673][ml] Add class of Vector Normalize Mapper

[GitBox]

flinkbot edited a comment on issue #9410: [FLINK-13673][ml] Add class of Vector 
Normalize Mapper
URL: https://github.com/apache/flink/pull/9410#issuecomment-519963431
 
 
   
   ## CI report:
   
   * 06b26ce69e490794f3bcda79766f8edc9d6e48ce : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122612649)
   * a5e7c30d865ea6ab35644743c9477a940ec9de65 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893914)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9413: [FLINK-13676][ml] Add class of Vector to Columns mapper

[GitBox]

flinkbot edited a comment on issue #9413: [FLINK-13676][ml] Add class of Vector 
to Columns mapper
URL: https://github.com/apache/flink/pull/9413#issuecomment-519968250
 
 
   
   ## CI report:
   
   * a585d0f8fef391d4da791b0cd1506e4f8b1acdc1 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122614562)
   * e399e94cb093d210dc9183bef48db93d60c30ca7 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125894099)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9412: [FLINK-13675][ml] Add class of Vector Slice Mapper

[GitBox]

flinkbot edited a comment on issue #9412: [FLINK-13675][ml] Add class of Vector 
Slice Mapper
URL: https://github.com/apache/flink/pull/9412#issuecomment-519965864
 
 
   
   ## CI report:
   
   * e9ff64826d561117cbd984df4201d15e0523b091 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/122613554)
   * 8e513594aaea05df91f6d2dd17088c4adb2fd1e9 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125894095)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (FLINK-13964) Remove usage of deprecated methods from MiniCluster

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-13964:
-

 Summary: Remove usage of deprecated methods from MiniCluster
 Key: FLINK-13964
 URL: https://issues.apache.org/jira/browse/FLINK-13964
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.10.0


With FLINK-13750 we deprecated 
{{HighAvailabilityServices#getWebMonitorRetrieverService}}. This method is 
still actively used by the {{MiniCluster}}. We should remove the usage in order 
to also support custom {{HighAvailabilityService}} implementations which no 
longer implement {{HighAvailabilityServices#getWebMonitorRetrieverService}}.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] Aitozi commented on issue #8479: [FLINK-11193][State Backends]Use user passed configuration overriding default configuration loading from file

[GitBox]

Aitozi commented on issue #8479: [FLINK-11193][State Backends]Use user passed 
configuration overriding default configuration loading from file
URL: https://github.com/apache/flink/pull/8479#issuecomment-527940974
 
 
   Hi @azagrebin , I have done the fix following your comments, please help 
review when you are free.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8479: [FLINK-11193][State Backends]Use user passed configuration overriding default configuration loading from file

[GitBox]

flinkbot edited a comment on issue #8479: [FLINK-11193][State Backends]Use user 
passed configuration overriding default configuration loading from file
URL: https://github.com/apache/flink/pull/8479#issuecomment-526521132
 
 
   
   ## CI report:
   
   * fdd6f0e24a689b70d15b2dc190fc2fc3b0600bfa : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125200442)
   * 341702cb226b8c34f077e9350fa656e9e2b120d4 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9404: [FLINK-13667][ml] Add the utility class for the Table

[GitBox]

flinkbot edited a comment on issue #9404: [FLINK-13667][ml] Add the utility 
class for the Table
URL: https://github.com/apache/flink/pull/9404#issuecomment-519872127
 
 
   
   ## CI report:
   
   * 4a726770daa19b1a587c5f8d9221a95e08387592 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/122573104)
   * cd4cfbe81b4d511ec985bc4ad55a5dad82722863 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124368074)
   * 6efbedf230f6fdeec96b9802d296629269a230ab : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125253936)
   * 51f02e22e7372fa20ab2dd679aaa1099b37d0b1e : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125893622)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8479: [FLINK-11193][State Backends]Use user passed configuration overriding default configuration loading from file

[GitBox]

flinkbot edited a comment on issue #8479: [FLINK-11193][State Backends]Use user 
passed configuration overriding default configuration loading from file
URL: https://github.com/apache/flink/pull/8479#issuecomment-526521132
 
 
   
   ## CI report:
   
   * fdd6f0e24a689b70d15b2dc190fc2fc3b0600bfa : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/125200442)
   * 341702cb226b8c34f077e9350fa656e9e2b120d4 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/125915214)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #9615: [FLINK-13952][table-planner][hive] PartitionableTableSink can not wor…

[GitBox]

flinkbot edited a comment on issue #9615: [FLINK-13952][table-planner][hive] 
PartitionableTableSink can not wor…
URL: https://github.com/apache/flink/pull/9615#issuecomment-527909466
 
 
   
   ## CI report:
   
   * b9cf01cfcebcc894450a7dd9d8c013108faccd6c : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/125899825)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin opened a new pull request #9617: [FLINK-13963] Consolidate Hadoop file systems usage and Hadoop integration docs

[GitBox]

azagrebin opened a new pull request #9617: [FLINK-13963] Consolidate Hadoop 
file systems usage and Hadoop integration docs
URL: https://github.com/apache/flink/pull/9617
 
 
   ## What is the purpose of the change
   
   We have hadoop related docs in several places at the moment:
   
   - **dev/batch/connectors.md** (Hadoop FS implementations and setup)
   - **dev/batch/hadoop_compatibility.md** (not valid any more that Flink 
always has Hadoop types out of the box as we do not build and provide Flink 
with Hadoop by default)
   - **ops/filesystems/index.md** (plugins, Hadoop FS implementations and setup 
revisited)
   - **ops/deployment/hadoop.md** (Hadoop classpath)
   - **ops/config.md** (deprecated way to provide Hadoop configuration in Flink 
conf)
   
   We could consolidate all these pieces of docs into a consistent structure to 
help users to navigate through the docs to well-defined spots depending on 
which feature they are trying to use.
   
   The places in docs which should contain the information about Hadoop:
   
   - **dev/batch/hadoop_compatibility.md** (only Dataset API specific stuff 
about integration with Hadoop)
   - **ops/filesystems/index.md** (Flink FS plugins and Hadoop FS 
implementations)
   - **ops/deployment/hadoop.md** (Hadoop configuration and classpath)
   
   How to setup Hadoop itself should be only in ops/deployment/hadoop.md. All 
other places dealing with Hadoop/HDFS should contain only their related things 
and just reference it 'how to configure Hadoop'. Like all chapters about 
writing to file systems (batch connectors and streaming file sinks) should just 
reference file systems.
   
   ## Brief change log
   
   See previous section.
   
   ## Verifying this change
   
   Run ./docs/build_docs.sh -i and open http://localhost:4000 in browser to 
check the doc changes.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
 - The S3 file system connector: (yes)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (docs)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-13963) Consolidate Hadoop file systems usage and Hadoop integration docs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-13963:
---
Labels: pull-request-available  (was: )

> Consolidate Hadoop file systems usage and Hadoop integration docs
> -
>
> Key: FLINK-13963
> URL: https://issues.apache.org/jira/browse/FLINK-13963
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem, Connectors / Hadoop 
> Compatibility, Documentation, FileSystems
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0
>
>
> We have hadoop related docs in several places at the moment:
>  * *dev/batch/connectors.md* (Hadoop FS implementations and setup)
>  * *dev/batch/hadoop_compatibility.md* (not valid any more that Flink always 
> has Hadoop types out of the box as we do not build and provide Flink with 
> Hadoop by default)
>  * *ops/filesystems/index.md* (plugins, Hadoop FS implementations and setup 
> revisited)
>  * *ops/deployment/hadoop.md* (Hadoop classpath)
>  * *ops/config.md* (deprecated way to provide Hadoop configuration in Flink 
> conf)
> We could consolidate all these pieces of docs into a consistent structure to 
> help users to navigate through the docs to well-defined spots depending on 
> which feature they are trying to use.
> The places in docs which should contain the information about Hadoop:
>  * *dev/batch/hadoop_compatibility.md* (only Dataset API specific stuff about 
> integration with Hadoop)
>  * *ops/filesystems/index.md* (Flink FS plugins and Hadoop FS implementations)
>  * *ops/deployment/hadoop.md* (Hadoop configuration and classpath)
> How to setup Hadoop itself should be only in *ops/deployment/hadoop.md*. All 
> other places dealing with Hadoop/HDFS should contain only their related 
> things and just reference it 'how to configure Hadoop'. Like all chapters 
> about writing to file systems (batch connectors and streaming file sinks) 
> should just reference file systems.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (FLINK-13927) Add note about hadoop dependencies for local debug

[Andrey Zagrebin (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Zagrebin updated FLINK-13927:

Description: 
Currently if user tries to run the job locally (e.g. from IDE) and uses Hadoop 
fs, it will not work if hadoop dependencies are not on the class path which is 
the case for the example from the quick start.

We can add a hint about adding provided hadoop dependencies in this case (+ 
Idea hint about provided included checkbox in run/debug app setup) to:
 
[https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/hadoop.html]

  was:
Currently if user tries to run the job locally (e.g. from IDE) and uses Hadoop 
fs, it will not work if hadoop dependencies are not on the class path which is 
the case for the example from the quick start.

We can add a hint about adding provided hadoop dependencies in this case (+ 
Idea hint about provided included checkbox in run/debug app setup) to:
 
[https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/hadoop.html]

and cross reference with:
 
[https://ci.apache.org/projects/flink/flink-docs-master/ops/filesystems/index.html#hadoop-configuration]


> Add note about hadoop dependencies for local debug
> --
>
> Key: FLINK-13927
> URL: https://issues.apache.org/jira/browse/FLINK-13927
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem, Documentation, FileSystems
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
>
> Currently if user tries to run the job locally (e.g. from IDE) and uses 
> Hadoop fs, it will not work if hadoop dependencies are not on the class path 
> which is the case for the example from the quick start.
> We can add a hint about adding provided hadoop dependencies in this case (+ 
> Idea hint about provided included checkbox in run/debug app setup) to:
>  
> [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/hadoop.html]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] flinkbot commented on issue #9617: [FLINK-13963] Consolidate Hadoop file systems usage and Hadoop integration docs

[GitBox]

flinkbot commented on issue #9617: [FLINK-13963] Consolidate Hadoop file 
systems usage and Hadoop integration docs
URL: https://github.com/apache/flink/pull/9617#issuecomment-527967715
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit ecf3897b7c61bca5a65d3ecc66a009d6ab91c596 (Wed Sep 04 
15:59:44 UTC 2019)
   
✅no warnings
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin opened a new pull request #9618: [FLINK-13927] Add note about hadoop dependencies for local debug

[GitBox]

azagrebin opened a new pull request #9618: [FLINK-13927] Add note about hadoop 
dependencies for local debug
URL: https://github.com/apache/flink/pull/9618
 
 
   ## What is the purpose of the change
   
   Currently if user tries to run the job locally (e.g. from IDE) and uses 
Hadoop fs, it will not work if hadoop dependencies are not on the class path 
which is the case for the example from the quick start.
   
   ## Brief change log
   
   We can add a hint about adding provided hadoop dependencies in this case (+ 
Idea hint about provided included checkbox in run/debug app setup) to 
`docs/ops/deployment/hadoop.md`.
   
   ## Verifying this change
   
   Run `./docs/build_docs.sh` -i and 
[open](http://localhost:4000/ops/deployment/hadoop.html) in browser to check 
the doc changes.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (docs)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-13927) Add note about hadoop dependencies for local debug

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-13927:
---
Labels: pull-request-available  (was: )

> Add note about hadoop dependencies for local debug
> --
>
> Key: FLINK-13927
> URL: https://issues.apache.org/jira/browse/FLINK-13927
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem, Documentation, FileSystems
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
>  Labels: pull-request-available
>
> Currently if user tries to run the job locally (e.g. from IDE) and uses 
> Hadoop fs, it will not work if hadoop dependencies are not on the class path 
> which is the case for the example from the quick start.
> We can add a hint about adding provided hadoop dependencies in this case (+ 
> Idea hint about provided included checkbox in run/debug app setup) to:
>  
> [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/hadoop.html]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [flink] flinkbot edited a comment on issue #9616: [FLINK-13957] [flink-yarn] Log dynamic properties on job submission

[GitBox]

flinkbot edited a comment on issue #9616: [FLINK-13957] [flink-yarn] Log 
dynamic properties on job submission
URL: https://github.com/apache/flink/pull/9616#issuecomment-527924104
 
 
   
   ## CI report:
   
   * b9114df609d2f21117e431ed36ae647a9f3a93a5 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/125906000)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services