[jira] [Commented] (FLINK-10386) Remove legacy class TaskExecutionStateListener
[
https://issues.apache.org/jira/browse/FLINK-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625341#comment-16625341
]
ASF GitHub Bot commented on FLINK-10386:
TisonKun commented on issue #6729: [FLINK-10386] [taskmanager] Remove legacy
class TaskExecutionStateListener
URL: https://github.com/apache/flink/pull/6729#issuecomment-423867964
cc @Clark @GJL @tillrohrmann
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove legacy class TaskExecutionStateListener
> --
>
> Key: FLINK-10386
> URL: https://issues.apache.org/jira/browse/FLINK-10386
> Project: Flink
> Issue Type: Improvement
> Components: TaskManager
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> After a discussion
> [here|https://github.com/apache/flink/commit/0735b5b935b0c0757943e2d58047afcfb9949560#commitcomment-30584257]
> with [~trohrm...@apache.org]. I start to analyze the usage of
> {{ActorGatewayTaskExecutionStateListener}} and {{TaskExecutionStateListener}}.
> In conclusion, we abort {{TaskExecutionStateListener}} strategy and no any
> component rely on it. Instead, we introduce {{TaskManagerActions}} to take
> the role for the communication of {{Task}} with {{TaskManager}}. No one
> except {{TaskManager}} should directly communicate with {{Task}}. So it can
> be safely remove legacy class {{TaskExecutionStateListener}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun commented on issue #6729: [FLINK-10386] [taskmanager] Remove legacy class TaskExecutionStateListener
TisonKun commented on issue #6729: [FLINK-10386] [taskmanager] Remove legacy class TaskExecutionStateListener URL: https://github.com/apache/flink/pull/6729#issuecomment-423867964 cc @Clark @GJL @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-10256) Port legacy jobmanager test to FILP-6
[ https://issues.apache.org/jira/browse/FLINK-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tison closed FLINK-10256. - Resolution: Duplicate Release Note: The purpose of this issue would be covered by FLINK-10392 > Port legacy jobmanager test to FILP-6 > - > > Key: FLINK-10256 > URL: https://issues.apache.org/jira/browse/FLINK-10256 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.7.0 >Reporter: tison >Assignee: tison >Priority: Major > Fix For: 1.7.0 > > > I am planning to rework JobManagerFailsITCase and JobManagerTest into > JobMasterITCase and JobMasterHAITCase. That is, reorganize the legacy tests, > make them neat and cover cases explicitly. The PR would follow before this > weekend. > While reworking, I'd like to add more jm failover test cases list below, for > the further implement of jm failover with RECONCILING state. For "jm > failover", I mean a real world failover(like low power or process exit), > without calling Flink internal postStop logic or something like it. > 1. Streaming task with jm failover. > 2. Streaming task with jm failover concurrent to task fail. > 3. Batch task with jm failover. > 4. Batch task with jm failover concurrent to task fail. > 5. Batch task with jm failover when some vertex has already been FINISHED. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] TisonKun closed pull request #6716: [hotfix] [yarn-test] Clean up inactive test
TisonKun closed pull request #6716: [hotfix] [yarn-test] Clean up inactive test
URL: https://github.com/apache/flink/pull/6716
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingApplicationMaster.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingApplicationMaster.java
deleted file mode 100644
index 785dff9c0c7..000
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingApplicationMaster.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.yarn;
-
-import org.apache.flink.runtime.jobmanager.JobManager;
-import org.apache.flink.runtime.jobmanager.MemoryArchivist;
-import org.apache.flink.runtime.taskmanager.TaskManager;
-import org.apache.flink.runtime.testingUtils.TestingMemoryArchivist;
-import org.apache.flink.runtime.testutils.TestingResourceManager;
-import org.apache.flink.runtime.util.EnvironmentInformation;
-import org.apache.flink.runtime.util.JvmShutdownSafeguard;
-import org.apache.flink.runtime.util.SignalHandler;
-
-/**
- * Yarn application master which starts the {@link TestingYarnJobManager},
- * {@link TestingResourceManager}, and the {@link TestingMemoryArchivist}.
- */
-public class TestingApplicationMaster extends YarnApplicationMasterRunner {
-
- @Override
- public Class getJobManagerClass() {
- return TestingYarnJobManager.class;
- }
-
- @Override
- public Class getArchivistClass() {
- return TestingMemoryArchivist.class;
- }
-
- @Override
- protected Class getTaskManagerClass() {
- return TestingYarnTaskManager.class;
- }
-
- @Override
- public Class
getResourceManagerClass() {
- return TestingYarnFlinkResourceManager.class;
- }
-
- public static void main(String[] args) {
- EnvironmentInformation.logEnvironmentInfo(LOG, "YARN
ApplicationMaster / JobManager", args);
- SignalHandler.register(LOG);
- JvmShutdownSafeguard.installAsShutdownHook(LOG);
-
- // run and exit with the proper return code
- int returnCode = new TestingApplicationMaster().run(args);
- System.exit(returnCode);
- }
-
-}
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
deleted file mode 100644
index 37b8d410a5d..000
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
+++ /dev/null
@@ -1,106 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.yarn;
-
-import org.apache.flink.client.deployment.ClusterSpecification;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.runtime.jobgraph.JobGraph;
-import org.apache.flink.util.Preconditions;
-
-import org.apache.hadoop.yarn.client.api.YarnClient;
-import org.apache.hadoop.yarn.conf.YarnConfiguration;
-
-import java.io.File;
-import java.io.FilenameFilter;
-import java.util.ArrayList;
-import
[GitHub] TisonKun commented on issue #6716: [hotfix] [yarn-test] Clean up inactive test
TisonKun commented on issue #6716: [hotfix] [yarn-test] Clean up inactive test URL: https://github.com/apache/flink/pull/6716#issuecomment-423867059 ... close as it would be covered by FLINK-10392 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10289) Classify Exceptions to different category for apply different failover strategy
[ https://issues.apache.org/jira/browse/FLINK-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-10289: --- Labels: pull-request-available (was: ) > Classify Exceptions to different category for apply different failover > strategy > --- > > Key: FLINK-10289 > URL: https://issues.apache.org/jira/browse/FLINK-10289 > Project: Flink > Issue Type: Sub-task > Components: JobManager >Reporter: JIN SUN >Assignee: JIN SUN >Priority: Major > Labels: pull-request-available > > We need to classify exceptions and treat them with different strategies. To > do this, we propose to introduce the following Throwable Types, and the > corresponding exceptions: > * NonRecoverable > ** We shouldn’t retry if an exception was classified as NonRecoverable > ** For example, NoResouceAvailiableException is a NonRecoverable Exception > ** Introduce a new Exception UserCodeException to wrap all exceptions that > throw from user code > * PartitionDataMissingError > ** In certain scenarios producer data was transferred in blocking mode or > data was saved in persistent store. If the partition was missing, we need to > revoke/rerun the produce task to regenerate the data. > ** Introduce a new exception PartitionDataMissingException to wrap all those > kinds of issues. > * EnvironmentError > ** It happened due to hardware, or software issues that were related to > specific environments. The assumption is that a task will succeed if we run > it in a different environment, and other task run in this bad environment > will very likely fail. If multiple task failures in the same machine due to > EnvironmentError, we need to consider adding the bad machine to blacklist, > and avoiding schedule task on it. > ** Introduce a new exception EnvironmentException to wrap all those kind of > issues. > * Recoverable > ** We assume other issues are recoverable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10289) Classify Exceptions to different category for apply different failover strategy
[ https://issues.apache.org/jira/browse/FLINK-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625289#comment-16625289 ] ASF GitHub Bot commented on FLINK-10289: isunjin opened a new pull request #6739: [FLINK-10289] [JobManager] Classify Exceptions to different category for apply different failover strategy URL: https://github.com/apache/flink/pull/6739 ## What is the purpose of the change We need to classify exceptions and treat them with different strategies. To do this, we propose to introduce the following Throwable Types, and the corresponding exceptions: - NonRecoverable - We shouldn’t retry if an exception was classified as NonRecoverable - For example, NoResouceAvailiableException is a NonRecoverable Exception - Introduce a new Exception UserCodeException to wrap all exceptions that throw from user code - PartitionDataMissingError - In certain scenarios producer data was transferred in blocking mode or data was saved in persistent store. If the partition was missing, we need to revoke/rerun the produce task to regenerate the data. - Introduce a new exception PartitionDataMissingException to wrap all those kinds of issues. - EnvironmentError - It happened due to hardware, or software issues that were related to specific environments. The assumption is that a task will succeed if we run it in a different environment, and other task run in this bad environment will very likely fail. If multiple task failures in the same machine due to EnvironmentError, we need to consider adding the bad machine to blacklist, and avoiding schedule task on it. - Introduce a new exception EnvironmentException to wrap all those kind of issues. - Recoverable - We assume other issues are recoverable. ## Brief change log - *Add exception types* - *Add a class to classify exceptions* - *Unittests* ## Verifying this change This change added tests and can be verified as follows: - *Added test that validates SuppressRestartsException was a NonRecoverable Exception* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (*no*) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? YES, Document is [(Here)](https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?spm=a1zcr.8293797.0.0.3b116385Btb5sf) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Classify Exceptions to different category for apply different failover > strategy > --- > > Key: FLINK-10289 > URL: https://issues.apache.org/jira/browse/FLINK-10289 > Project: Flink > Issue Type: Sub-task > Components: JobManager >Reporter: JIN SUN >Assignee: JIN SUN >Priority: Major > Labels: pull-request-available > > We need to classify exceptions and treat them with different strategies. To > do this, we propose to introduce the following Throwable Types, and the > corresponding exceptions: > * NonRecoverable > ** We shouldn’t retry if an exception was classified as NonRecoverable > ** For example, NoResouceAvailiableException is a NonRecoverable Exception > ** Introduce a new Exception UserCodeException to wrap all exceptions that > throw from user code > * PartitionDataMissingError > ** In certain scenarios producer data was transferred in blocking mode or > data was saved in persistent store. If the partition was missing, we need to > revoke/rerun the produce task to regenerate the data. > ** Introduce a new exception PartitionDataMissingException to wrap all those > kinds of issues. > * EnvironmentError > ** It happened due to hardware, or software issues that were related to > specific environments. The assumption is that a task will succeed if we run > it in a different environment, and other task run in this bad environment > will very likely fail. If multiple task failures in the same machine due to > EnvironmentError, we
[GitHub] isunjin opened a new pull request #6739: [FLINK-10289] [JobManager] Classify Exceptions to different category for apply different failover strategy
isunjin opened a new pull request #6739: [FLINK-10289] [JobManager] Classify Exceptions to different category for apply different failover strategy URL: https://github.com/apache/flink/pull/6739 ## What is the purpose of the change We need to classify exceptions and treat them with different strategies. To do this, we propose to introduce the following Throwable Types, and the corresponding exceptions: - NonRecoverable - We shouldn’t retry if an exception was classified as NonRecoverable - For example, NoResouceAvailiableException is a NonRecoverable Exception - Introduce a new Exception UserCodeException to wrap all exceptions that throw from user code - PartitionDataMissingError - In certain scenarios producer data was transferred in blocking mode or data was saved in persistent store. If the partition was missing, we need to revoke/rerun the produce task to regenerate the data. - Introduce a new exception PartitionDataMissingException to wrap all those kinds of issues. - EnvironmentError - It happened due to hardware, or software issues that were related to specific environments. The assumption is that a task will succeed if we run it in a different environment, and other task run in this bad environment will very likely fail. If multiple task failures in the same machine due to EnvironmentError, we need to consider adding the bad machine to blacklist, and avoiding schedule task on it. - Introduce a new exception EnvironmentException to wrap all those kind of issues. - Recoverable - We assume other issues are recoverable. ## Brief change log - *Add exception types* - *Add a class to classify exceptions* - *Unittests* ## Verifying this change This change added tests and can be verified as follows: - *Added test that validates SuppressRestartsException was a NonRecoverable Exception* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (*no*) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? YES, Document is [(Here)](https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?spm=a1zcr.8293797.0.0.3b116385Btb5sf) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-10402) Port AbstractTaskManagerProcessFailureRecoveryTest to new code base
Till Rohrmann created FLINK-10402:
-
Summary: Port AbstractTaskManagerProcessFailureRecoveryTest to new
code base
Key: FLINK-10402
URL: https://issues.apache.org/jira/browse/FLINK-10402
Project: Flink
Issue Type: Sub-task
Components: Tests
Affects Versions: 1.7.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.7.0
Port {{AbstractTaskManagerProcessFailureRecoveryTest}} to new code base.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10401) Port ProcessFailureCancelingITCase to new code base
Till Rohrmann created FLINK-10401:
-
Summary: Port ProcessFailureCancelingITCase to new code base
Key: FLINK-10401
URL: https://issues.apache.org/jira/browse/FLINK-10401
Project: Flink
Issue Type: Sub-task
Components: Tests
Affects Versions: 1.7.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.7.0
Port {{ProcessFailureCancelingITCase}} to new code base.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10369) Enable YARNITCase
[
https://issues.apache.org/jira/browse/FLINK-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625178#comment-16625178
]
ASF GitHub Bot commented on FLINK-10369:
tillrohrmann closed pull request #6717: [FLINK-10369][tests] Enable YARNITCase
to test per job mode deployment
URL: https://github.com/apache/flink/pull/6717
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
index 37b8d410a5d..a16cb0b752c 100644
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
+++
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
@@ -22,12 +22,12 @@
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.yarn.util.YarnTestUtils;
import org.apache.hadoop.yarn.client.api.YarnClient;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import java.io.File;
-import java.io.FilenameFilter;
import java.util.ArrayList;
import java.util.List;
@@ -52,15 +52,15 @@ public TestingYarnClusterDescriptor(
sharedYarnClient);
List filesToShip = new ArrayList<>();
- File testingJar = YarnTestBase.findFile("..", new
TestJarFinder("flink-yarn-tests"));
+ File testingJar = YarnTestBase.findFile("..", new
YarnTestUtils.TestJarFinder("flink-yarn-tests"));
Preconditions.checkNotNull(testingJar, "Could not find the
flink-yarn-tests tests jar. " +
"Make sure to package the flink-yarn-tests module.");
- File testingRuntimeJar = YarnTestBase.findFile("..", new
TestJarFinder("flink-runtime"));
+ File testingRuntimeJar = YarnTestBase.findFile("..", new
YarnTestUtils.TestJarFinder("flink-runtime"));
Preconditions.checkNotNull(testingRuntimeJar, "Could not find
the flink-runtime tests " +
"jar. Make sure to package the flink-runtime module.");
- File testingYarnJar = YarnTestBase.findFile("..", new
TestJarFinder("flink-yarn"));
+ File testingYarnJar = YarnTestBase.findFile("..", new
YarnTestUtils.TestJarFinder("flink-yarn"));
Preconditions.checkNotNull(testingRuntimeJar, "Could not find
the flink-yarn tests " +
"jar. Make sure to package the flink-yarn module.");
@@ -89,18 +89,4 @@ public YarnClusterClient deployJobCluster(
throw new UnsupportedOperationException("Cannot deploy a
per-job cluster yet.");
}
- static class TestJarFinder implements FilenameFilter {
-
- private final String jarName;
-
- TestJarFinder(final String jarName) {
- this.jarName = jarName;
- }
-
- @Override
- public boolean accept(File dir, String name) {
- return name.startsWith(jarName) &&
name.endsWith("-tests.jar") &&
- dir.getAbsolutePath().contains(dir.separator +
jarName + dir.separator);
- }
- }
}
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
index 9a8f5033f3f..3625a90076f 100644
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
+++
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
@@ -60,7 +60,7 @@
/**
* Tests that verify correct HA behavior.
*/
-public class YARNHighAvailabilityITCase extends YarnTestBase {
+ public class YARNHighAvailabilityITCase extends YarnTestBase {
private static TestingServer zkServer;
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
index 758a09866d0..814e8082f70 100644
--- a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
+++ b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
@@ -20,24 +20,31 @@
import org.apache.flink.client.deployment.ClusterSpecification;
import org.apache.flink.client.program.ClusterClient;
+import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.AkkaOptions;
import org.apache.flink.configuration.ConfigConstants;
import
[jira] [Closed] (FLINK-10369) Enable YARNITCase
[
https://issues.apache.org/jira/browse/FLINK-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-10369.
-
Resolution: Fixed
Fixed via f6a14c02d898b881903e62ed847bf595beaea7cd
> Enable YARNITCase
> -
>
> Key: FLINK-10369
> URL: https://issues.apache.org/jira/browse/FLINK-10369
> Project: Flink
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> The {{YARNITCase}} is ignored because when it was added it was not possible
> to terminate the Flink cluster. This has changed now and consequently, we
> should enable this test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann closed pull request #6717: [FLINK-10369][tests] Enable YARNITCase to test per job mode deployment
tillrohrmann closed pull request #6717: [FLINK-10369][tests] Enable YARNITCase
to test per job mode deployment
URL: https://github.com/apache/flink/pull/6717
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
index 37b8d410a5d..a16cb0b752c 100644
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
+++
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/TestingYarnClusterDescriptor.java
@@ -22,12 +22,12 @@
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.yarn.util.YarnTestUtils;
import org.apache.hadoop.yarn.client.api.YarnClient;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import java.io.File;
-import java.io.FilenameFilter;
import java.util.ArrayList;
import java.util.List;
@@ -52,15 +52,15 @@ public TestingYarnClusterDescriptor(
sharedYarnClient);
List filesToShip = new ArrayList<>();
- File testingJar = YarnTestBase.findFile("..", new
TestJarFinder("flink-yarn-tests"));
+ File testingJar = YarnTestBase.findFile("..", new
YarnTestUtils.TestJarFinder("flink-yarn-tests"));
Preconditions.checkNotNull(testingJar, "Could not find the
flink-yarn-tests tests jar. " +
"Make sure to package the flink-yarn-tests module.");
- File testingRuntimeJar = YarnTestBase.findFile("..", new
TestJarFinder("flink-runtime"));
+ File testingRuntimeJar = YarnTestBase.findFile("..", new
YarnTestUtils.TestJarFinder("flink-runtime"));
Preconditions.checkNotNull(testingRuntimeJar, "Could not find
the flink-runtime tests " +
"jar. Make sure to package the flink-runtime module.");
- File testingYarnJar = YarnTestBase.findFile("..", new
TestJarFinder("flink-yarn"));
+ File testingYarnJar = YarnTestBase.findFile("..", new
YarnTestUtils.TestJarFinder("flink-yarn"));
Preconditions.checkNotNull(testingRuntimeJar, "Could not find
the flink-yarn tests " +
"jar. Make sure to package the flink-yarn module.");
@@ -89,18 +89,4 @@ public YarnClusterClient deployJobCluster(
throw new UnsupportedOperationException("Cannot deploy a
per-job cluster yet.");
}
- static class TestJarFinder implements FilenameFilter {
-
- private final String jarName;
-
- TestJarFinder(final String jarName) {
- this.jarName = jarName;
- }
-
- @Override
- public boolean accept(File dir, String name) {
- return name.startsWith(jarName) &&
name.endsWith("-tests.jar") &&
- dir.getAbsolutePath().contains(dir.separator +
jarName + dir.separator);
- }
- }
}
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
index 9a8f5033f3f..3625a90076f 100644
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
+++
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
@@ -60,7 +60,7 @@
/**
* Tests that verify correct HA behavior.
*/
-public class YARNHighAvailabilityITCase extends YarnTestBase {
+ public class YARNHighAvailabilityITCase extends YarnTestBase {
private static TestingServer zkServer;
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
index 758a09866d0..814e8082f70 100644
--- a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
+++ b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
@@ -20,24 +20,31 @@
import org.apache.flink.client.deployment.ClusterSpecification;
import org.apache.flink.client.program.ClusterClient;
+import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.AkkaOptions;
import org.apache.flink.configuration.ConfigConstants;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobmaster.JobResult;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import
[jira] [Commented] (FLINK-10369) Enable YARNITCase
[
https://issues.apache.org/jira/browse/FLINK-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625176#comment-16625176
]
ASF GitHub Bot commented on FLINK-10369:
tillrohrmann commented on a change in pull request #6717: [FLINK-10369][tests]
Enable YARNITCase to test per job mode deployment
URL: https://github.com/apache/flink/pull/6717#discussion_r219704301
##
File path:
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
##
@@ -60,7 +60,7 @@
/**
* Tests that verify correct HA behavior.
*/
-public class YARNHighAvailabilityITCase extends YarnTestBase {
+ public class YARNHighAvailabilityITCase extends YarnTestBase {
Review comment:
Good catch. Will remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Enable YARNITCase
> -
>
> Key: FLINK-10369
> URL: https://issues.apache.org/jira/browse/FLINK-10369
> Project: Flink
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> The {{YARNITCase}} is ignored because when it was added it was not possible
> to terminate the Flink cluster. This has changed now and consequently, we
> should enable this test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann commented on a change in pull request #6717: [FLINK-10369][tests] Enable YARNITCase to test per job mode deployment
tillrohrmann commented on a change in pull request #6717: [FLINK-10369][tests]
Enable YARNITCase to test per job mode deployment
URL: https://github.com/apache/flink/pull/6717#discussion_r219704301
##
File path:
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNHighAvailabilityITCase.java
##
@@ -60,7 +60,7 @@
/**
* Tests that verify correct HA behavior.
*/
-public class YARNHighAvailabilityITCase extends YarnTestBase {
+ public class YARNHighAvailabilityITCase extends YarnTestBase {
Review comment:
Good catch. Will remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-10126) There should be a Scala DataSource
[ https://issues.apache.org/jira/browse/FLINK-10126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-10126: --- Labels: datasource pull-request-available scala (was: datasource scala) > There should be a Scala DataSource > -- > > Key: FLINK-10126 > URL: https://issues.apache.org/jira/browse/FLINK-10126 > Project: Flink > Issue Type: Improvement >Reporter: Alexis Sarda-Espinosa >Assignee: vinoyang >Priority: Minor > Labels: datasource, pull-request-available, scala > > In Java, an ExecutionEnvironment's createInput method returns a DataSource, > whereas the Scala version returns a DataSet. There is no Scala DataSource > wrapper, and the Scala DataSet does not provide the Java DataSource methods, > such as getSplitDataProperties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10126) There should be a Scala DataSource
[ https://issues.apache.org/jira/browse/FLINK-10126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625073#comment-16625073 ] ASF GitHub Bot commented on FLINK-10126: yanghua opened a new pull request #6738: [FLINK-10126] There should be a Scala DataSource URL: https://github.com/apache/flink/pull/6738 ## What is the purpose of the change *This pull request add some APIs which exist flink-java's `DataSource` into flink-scala `DataSet`* ## Brief change log - *Add some APIs which exist flink-java's `DataSource` into flink-scala `DataSet`* - *Added test case `DataSetITCase`* ## Verifying this change This change is already covered by existing tests, such as *DataSetITCase*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > There should be a Scala DataSource > -- > > Key: FLINK-10126 > URL: https://issues.apache.org/jira/browse/FLINK-10126 > Project: Flink > Issue Type: Improvement >Reporter: Alexis Sarda-Espinosa >Assignee: vinoyang >Priority: Minor > Labels: datasource, pull-request-available, scala > > In Java, an ExecutionEnvironment's createInput method returns a DataSource, > whereas the Scala version returns a DataSet. There is no Scala DataSource > wrapper, and the Scala DataSet does not provide the Java DataSource methods, > such as getSplitDataProperties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yanghua opened a new pull request #6738: [FLINK-10126] There should be a Scala DataSource
yanghua opened a new pull request #6738: [FLINK-10126] There should be a Scala DataSource URL: https://github.com/apache/flink/pull/6738 ## What is the purpose of the change *This pull request add some APIs which exist flink-java's `DataSource` into flink-scala `DataSet`* ## Brief change log - *Add some APIs which exist flink-java's `DataSource` into flink-scala `DataSet`* - *Added test case `DataSetITCase`* ## Verifying this change This change is already covered by existing tests, such as *DataSetITCase*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Issue Comment Deleted] (FLINK-8037) Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort
[
https://issues.apache.org/jira/browse/FLINK-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-8037:
--
Comment: was deleted
(was: Please rebase PR.)
> Missing cast in integer arithmetic in
> TransactionalIdsGenerator#generateIdsToAbort
> --
>
> Key: FLINK-8037
> URL: https://issues.apache.org/jira/browse/FLINK-8037
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Greg Hogan
>Priority: Minor
> Labels: kafka, kafka-connect
>
> {code}
> public Set generateIdsToAbort() {
> Set idsToAbort = new HashSet<>();
> for (int i = 0; i < safeScaleDownFactor; i++) {
> idsToAbort.addAll(generateIdsToUse(i * poolSize *
> totalNumberOfSubtasks));
> {code}
> The operands are integers where generateIdsToUse() expects long parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tedyu commented on issue #6365: [FLINK-9849] [hbase] Hbase upgrade to 2.0.1
tedyu commented on issue #6365: [FLINK-9849] [hbase] Hbase upgrade to 2.0.1 URL: https://github.com/apache/flink/pull/6365#issuecomment-423807269 ``` 16:08:53.680 [ERROR] src/main/java/org/apache/flink/addons/hbase/TableInputFormat.java:[31] (imports) ImportOrder: Import org.apache.hadoop.hbase.client.Result appears after other imports that it should precede 16:08:53.680 [ERROR] src/test/java/org/apache/flink/addons/hbase/HBaseTestingClusterAutostarter.java:[38] (imports) ImportOrder: Import org.apache.hadoop.hbase.client.HBaseAdmin appears after other imports that it should precede ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9849) Create hbase connector for hbase version to 2.0.2
[ https://issues.apache.org/jira/browse/FLINK-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625062#comment-16625062 ] ASF GitHub Bot commented on FLINK-9849: --- tedyu commented on issue #6365: [FLINK-9849] [hbase] Hbase upgrade to 2.0.1 URL: https://github.com/apache/flink/pull/6365#issuecomment-423807269 ``` 16:08:53.680 [ERROR] src/main/java/org/apache/flink/addons/hbase/TableInputFormat.java:[31] (imports) ImportOrder: Import org.apache.hadoop.hbase.client.Result appears after other imports that it should precede 16:08:53.680 [ERROR] src/test/java/org/apache/flink/addons/hbase/HBaseTestingClusterAutostarter.java:[38] (imports) ImportOrder: Import org.apache.hadoop.hbase.client.HBaseAdmin appears after other imports that it should precede ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create hbase connector for hbase version to 2.0.2 > - > > Key: FLINK-9849 > URL: https://issues.apache.org/jira/browse/FLINK-9849 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > Attachments: hbase-2.1.0.dep > > > Currently hbase 1.4.3 is used for hbase connector. > We should create connector for hbase 2.0.2 which would be released. > Since there are API changes for the 2.0.2 release, a new hbase connector is > desirable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10400) Return failed JobResult if job terminates in state FAILED or CANCELED
[
https://issues.apache.org/jira/browse/FLINK-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625030#comment-16625030
]
tison commented on FLINK-10400:
---
Agree. It is a code wart that should be fixed.
To be more clear, return a {{JobResult}} with {{Exception}} as described,
{{addSuppressed}} if there is a failure cause.
> Return failed JobResult if job terminates in state FAILED or CANCELED
> -
>
> Key: FLINK-10400
> URL: https://issues.apache.org/jira/browse/FLINK-10400
> Project: Flink
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.6.1, 1.7.0, 1.5.4
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> If the job reaches the globally terminal state {{FAILED}} or {{CANCELED}},
> the {{JobResult}} must return a non-successful result. At the moment, it can
> happen that in the {{CANCELED}} state where we don't find a failure cause
> that we return a successful {{JobResult}}.
> In order to change this I propose to always return a {{JobResult}} with a
> {{JobCancellationException}} in case of {{CANCELED}} and a
> {{JobExecutionException}} in case of {{FAILED}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10400) Return failed JobResult if job terminates in state FAILED or CANCELED
Till Rohrmann created FLINK-10400:
-
Summary: Return failed JobResult if job terminates in state FAILED
or CANCELED
Key: FLINK-10400
URL: https://issues.apache.org/jira/browse/FLINK-10400
Project: Flink
Issue Type: Bug
Components: Client
Affects Versions: 1.5.4, 1.6.1, 1.7.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.7.0, 1.6.2, 1.5.5
If the job reaches the globally terminal state {{FAILED}} or {{CANCELED}}, the
{{JobResult}} must return a non-successful result. At the moment, it can happen
that in the {{CANCELED}} state where we don't find a failure cause that we
return a successful {{JobResult}}.
In order to change this I propose to always return a {{JobResult}} with a
{{JobCancellationException}} in case of {{CANCELED}} and a
{{JobExecutionException}} in case of {{FAILED}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10399) Refractor ParameterTool#fromArgs
[
https://issues.apache.org/jira/browse/FLINK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625022#comment-16625022
]
ASF GitHub Bot commented on FLINK-10399:
TisonKun commented on issue #6737: [FLINK-10399] Refractor
ParameterTool#fromArgs
URL: https://github.com/apache/flink/pull/6737#issuecomment-423799819
Follow the review guide, I would ask for the consensus of this refractor.
I think it is worth because the original one is quite mess and weird. Clean
code can express purpose better and prevent us from potential bugs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refractor ParameterTool#fromArgs
>
>
> Key: FLINK-10399
> URL: https://issues.apache.org/jira/browse/FLINK-10399
> Project: Flink
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{ParameterTool#fromArgs}} uses a weird implement which flink developer would
> fail to parse it fast.
> The main problem is that, when parse args, we always try to get a key-value
> pair, but the implement iterate by a {{for}} loop, thus introduce weird
> flag/mutable variable and branches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun edited a comment on issue #6737: [FLINK-10399] Refractor ParameterTool#fromArgs
TisonKun edited a comment on issue #6737: [FLINK-10399] Refractor ParameterTool#fromArgs URL: https://github.com/apache/flink/pull/6737#issuecomment-423799819 Follow the review guide, I would firstly ask for the consensus of this refractor. I think it is worth because the original one is quite mess and weird. Clean code can express purpose better and prevent us from potential bugs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10399) Refractor ParameterTool#fromArgs
[
https://issues.apache.org/jira/browse/FLINK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625023#comment-16625023
]
ASF GitHub Bot commented on FLINK-10399:
TisonKun edited a comment on issue #6737: [FLINK-10399] Refractor
ParameterTool#fromArgs
URL: https://github.com/apache/flink/pull/6737#issuecomment-423799819
Follow the review guide, I would firstly ask for the consensus of this
refractor.
I think it is worth because the original one is quite mess and weird. Clean
code can express purpose better and prevent us from potential bugs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refractor ParameterTool#fromArgs
>
>
> Key: FLINK-10399
> URL: https://issues.apache.org/jira/browse/FLINK-10399
> Project: Flink
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{ParameterTool#fromArgs}} uses a weird implement which flink developer would
> fail to parse it fast.
> The main problem is that, when parse args, we always try to get a key-value
> pair, but the implement iterate by a {{for}} loop, thus introduce weird
> flag/mutable variable and branches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun commented on issue #6737: [FLINK-10399] Refractor ParameterTool#fromArgs
TisonKun commented on issue #6737: [FLINK-10399] Refractor ParameterTool#fromArgs URL: https://github.com/apache/flink/pull/6737#issuecomment-423799819 Follow the review guide, I would ask for the consensus of this refractor. I think it is worth because the original one is quite mess and weird. Clean code can express purpose better and prevent us from potential bugs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10399) Refractor ParameterTool#fromArgs
[
https://issues.apache.org/jira/browse/FLINK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625019#comment-16625019
]
ASF GitHub Bot commented on FLINK-10399:
TisonKun commented on issue #6737: [FLINK-10399] Refractor
ParameterTool#fromArgs
URL: https://github.com/apache/flink/pull/6737#issuecomment-423799090
cc @zentol @gyfora @fhueske
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refractor ParameterTool#fromArgs
>
>
> Key: FLINK-10399
> URL: https://issues.apache.org/jira/browse/FLINK-10399
> Project: Flink
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{ParameterTool#fromArgs}} uses a weird implement which flink developer would
> fail to parse it fast.
> The main problem is that, when parse args, we always try to get a key-value
> pair, but the implement iterate by a {{for}} loop, thus introduce weird
> flag/mutable variable and branches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun commented on issue #6737: [FLINK-10399] Refractor ParameterTool#fromArgs
TisonKun commented on issue #6737: [FLINK-10399] Refractor ParameterTool#fromArgs URL: https://github.com/apache/flink/pull/6737#issuecomment-423799090 cc @zentol @gyfora @fhueske This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10399) Refractor ParameterTool#fromArgs
[
https://issues.apache.org/jira/browse/FLINK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10399:
---
Labels: pull-request-available (was: )
> Refractor ParameterTool#fromArgs
>
>
> Key: FLINK-10399
> URL: https://issues.apache.org/jira/browse/FLINK-10399
> Project: Flink
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{ParameterTool#fromArgs}} uses a weird implement which flink developer would
> fail to parse it fast.
> The main problem is that, when parse args, we always try to get a key-value
> pair, but the implement iterate by a {{for}} loop, thus introduce weird
> flag/mutable variable and branches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10399) Refractor ParameterTool#fromArgs
[
https://issues.apache.org/jira/browse/FLINK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625016#comment-16625016
]
ASF GitHub Bot commented on FLINK-10399:
TisonKun opened a new pull request #6737: [FLINK-10399] Refractor
ParameterTool#fromArgs
URL: https://github.com/apache/flink/pull/6737
## What is the purpose of the change
This pull request is about a refractor of `ParameterTool#fromArgs`.
Current `ParameterTool#fromArgs` uses a weird implement by which flink
developer would fail to parse it fast.
The main problem is that, when parse args, we always try to get a key-value
pair, but the implement iterate by a for loop, thus introduce weird
flag/mutable variable and branches.
## Brief change log
Refractor `ParameterTool#fromArgs`. Use a index(int iterator) and while loop
to replace the original for loop. Make it more clear for the purpose and reduce
weird branches and flags.
Place some `final` modifier where I cannot get the idea that the variable is
`final` when look at it.
## Verifying this change
`ParameterToolTest` should guard the refractor.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (**no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (**yes**, but only the implementation, the interface and
behavior kept.)
- The serializers: (**no**)
- The runtime per-record code paths (performance sensitive): (**no**)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**)
- The S3 file system connector: (**no**)
## Documentation
- Does this pull request introduce a new feature? (**no**)
- If yes, how is the feature documented? (**no**)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refractor ParameterTool#fromArgs
>
>
> Key: FLINK-10399
> URL: https://issues.apache.org/jira/browse/FLINK-10399
> Project: Flink
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{ParameterTool#fromArgs}} uses a weird implement which flink developer would
> fail to parse it fast.
> The main problem is that, when parse args, we always try to get a key-value
> pair, but the implement iterate by a {{for}} loop, thus introduce weird
> flag/mutable variable and branches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun opened a new pull request #6737: [FLINK-10399] Refractor ParameterTool#fromArgs
TisonKun opened a new pull request #6737: [FLINK-10399] Refractor ParameterTool#fromArgs URL: https://github.com/apache/flink/pull/6737 ## What is the purpose of the change This pull request is about a refractor of `ParameterTool#fromArgs`. Current `ParameterTool#fromArgs` uses a weird implement by which flink developer would fail to parse it fast. The main problem is that, when parse args, we always try to get a key-value pair, but the implement iterate by a for loop, thus introduce weird flag/mutable variable and branches. ## Brief change log Refractor `ParameterTool#fromArgs`. Use a index(int iterator) and while loop to replace the original for loop. Make it more clear for the purpose and reduce weird branches and flags. Place some `final` modifier where I cannot get the idea that the variable is `final` when look at it. ## Verifying this change `ParameterToolTest` should guard the refractor. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes**, but only the implementation, the interface and behavior kept.) - The serializers: (**no**) - The runtime per-record code paths (performance sensitive): (**no**) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**) - The S3 file system connector: (**no**) ## Documentation - Does this pull request introduce a new feature? (**no**) - If yes, how is the feature documented? (**no**) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10315) Let JDBCAppendTableSink be built with java.sql.Connection
[ https://issues.apache.org/jira/browse/FLINK-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625014#comment-16625014 ] vinoyang commented on FLINK-10315: -- Hi [~flacombe] , I think there are a few questions about your suggestion: 1) If you add the setConnection() method to the builder, the life cycle of the Connection will be uncontrollable, and the open/close of the OutputFormat interface is more reasonable to manage the creation and destruction of the db connection; 2) The connection should not be initialized when the program is built, because it will be used when it is actually executed on the TM. It makes no sense to establish a connection in advance, and it cannot be serialized and passed to the real TM. What do you think? > Let JDBCAppendTableSink be built with java.sql.Connection > - > > Key: FLINK-10315 > URL: https://issues.apache.org/jira/browse/FLINK-10315 > Project: Flink > Issue Type: Improvement > Components: Java API > Environment: I'm currently using Flink 1.6.0 Java. >Reporter: François Lacombe >Assignee: vinoyang >Priority: Major > > Currently, JDBCAppendTableSink is built with methods like setDBUrl, > setUsername, setPassword... and so on. > We can't use an existing Java SQL connection to build it. > It may be great to add a setConnection() method to the builder class as to > prevent sensitive data like username or password to transit through large > stacks from config connectors (often in main()) to JDBC sinks. > To be able to provide only one object is far lighter than 4 or 5 strings > > Thanks in advance -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10399) Refractor ParameterTool#fromArgs
tison created FLINK-10399:
-
Summary: Refractor ParameterTool#fromArgs
Key: FLINK-10399
URL: https://issues.apache.org/jira/browse/FLINK-10399
Project: Flink
Issue Type: Improvement
Components: Client
Affects Versions: 1.7.0
Reporter: tison
Assignee: tison
Fix For: 1.7.0
{{ParameterTool#fromArgs}} uses a weird implement which flink developer would
fail to parse it fast.
The main problem is that, when parse args, we always try to get a key-value
pair, but the implement iterate by a {{for}} loop, thus introduce weird
flag/mutable variable and branches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-10251) Handle oversized response messages in AkkaRpcActor
[
https://issues.apache.org/jira/browse/FLINK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tison reassigned FLINK-10251:
-
Assignee: tison
> Handle oversized response messages in AkkaRpcActor
> --
>
> Key: FLINK-10251
> URL: https://issues.apache.org/jira/browse/FLINK-10251
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: tison
>Priority: Major
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> The {{AkkaRpcActor}} should check whether an RPC response which is sent to a
> remote sender does not exceed the maximum framesize of the underlying
> {{ActorSystem}}. If this is the case we should fail fast instead. We can
> achieve this by serializing the response and sending the serialized byte
> array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-10252) Handle oversized metric messges
[
https://issues.apache.org/jira/browse/FLINK-10252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vinoyang reassigned FLINK-10252:
Assignee: vinoyang
> Handle oversized metric messges
> ---
>
> Key: FLINK-10252
> URL: https://issues.apache.org/jira/browse/FLINK-10252
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> Since the {{MetricQueryService}} is implemented as an Akka actor, it can only
> send messages of a smaller size then the current {{akka.framesize}}. We
> should check similarly to FLINK-10251 whether the payload exceeds the maximum
> framesize and fail fast if it is true.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10291) Generate JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint
[
https://issues.apache.org/jira/browse/FLINK-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625009#comment-16625009
]
ASF GitHub Bot commented on FLINK-10291:
yanghua commented on a change in pull request #6733: [FLINK-10291] Generate
JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint
URL: https://github.com/apache/flink/pull/6733#discussion_r219690536
##
File path:
flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneJobClusterEntryPoint.java
##
@@ -67,6 +68,8 @@
@Nonnull
private final SavepointRestoreSettings savepointRestoreSettings;
+ private final JobID jobID = new JobID(Long.MIN_VALUE, Long.MIN_VALUE);
Review comment:
@uce Your idea sounds good, I accept!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generate JobGraph with fixed/configurable JobID in
> StandaloneJobClusterEntrypoint
> -
>
> Key: FLINK-10291
> URL: https://issues.apache.org/jira/browse/FLINK-10291
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> The {{StandaloneJobClusterEntrypoint}} currently generates the {{JobGraph}}
> from the user code when being started. Due to the nature of how the
> {{JobGraph}} is generated, it will get a random {{JobID}} assigned. This is
> problematic in case of a failover because then, the {{JobMaster}} won't be
> able to detect the checkpoints. In order to solve this problem, we need to
> either fix the {{JobID}} assignment or make it configurable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on a change in pull request #6733: [FLINK-10291] Generate JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint
yanghua commented on a change in pull request #6733: [FLINK-10291] Generate JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint URL: https://github.com/apache/flink/pull/6733#discussion_r219690536 ## File path: flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneJobClusterEntryPoint.java ## @@ -67,6 +68,8 @@ @Nonnull private final SavepointRestoreSettings savepointRestoreSettings; + private final JobID jobID = new JobID(Long.MIN_VALUE, Long.MIN_VALUE); Review comment: @uce Your idea sounds good, I accept! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] uce commented on a change in pull request #6733: [FLINK-10291] Generate JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint
uce commented on a change in pull request #6733: [FLINK-10291] Generate JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint URL: https://github.com/apache/flink/pull/6733#discussion_r219690410 ## File path: flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneJobClusterEntryPoint.java ## @@ -67,6 +68,8 @@ @Nonnull private final SavepointRestoreSettings savepointRestoreSettings; + private final JobID jobID = new JobID(Long.MIN_VALUE, Long.MIN_VALUE); Review comment: The resulting `JobID` for this is: `80008000`. What do you think about using `new JobID(0, 0)` to give an all zero `JobID` instead? Since this is a fixed ID anyways, I think going with the simplest ID is preferable and an all zero ID might be easier to use when manually working with the REST API of a job instead of remembering to insert an `8` (which also looks very similar to a `0`). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10291) Generate JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint
[
https://issues.apache.org/jira/browse/FLINK-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625008#comment-16625008
]
ASF GitHub Bot commented on FLINK-10291:
uce commented on a change in pull request #6733: [FLINK-10291] Generate
JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint
URL: https://github.com/apache/flink/pull/6733#discussion_r219690410
##
File path:
flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneJobClusterEntryPoint.java
##
@@ -67,6 +68,8 @@
@Nonnull
private final SavepointRestoreSettings savepointRestoreSettings;
+ private final JobID jobID = new JobID(Long.MIN_VALUE, Long.MIN_VALUE);
Review comment:
The resulting `JobID` for this is: `80008000`.
What do you think about using `new JobID(0, 0)` to give an all zero `JobID`
instead? Since this is a fixed ID anyways, I think going with the simplest ID
is preferable and an all zero ID might be easier to use when manually working
with the REST API of a job instead of remembering to insert an `8` (which also
looks very similar to a `0`).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generate JobGraph with fixed/configurable JobID in
> StandaloneJobClusterEntrypoint
> -
>
> Key: FLINK-10291
> URL: https://issues.apache.org/jira/browse/FLINK-10291
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> The {{StandaloneJobClusterEntrypoint}} currently generates the {{JobGraph}}
> from the user code when being started. Due to the nature of how the
> {{JobGraph}} is generated, it will get a random {{JobID}} assigned. This is
> problematic in case of a failover because then, the {{JobMaster}} won't be
> able to detect the checkpoints. In order to solve this problem, we need to
> either fix the {{JobID}} assignment or make it configurable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
