[GitHub] [flink] flinkbot edited a comment on pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
flinkbot edited a comment on pull request #15729: URL: https://github.com/apache/flink/pull/15729#issuecomment-825360063 ## CI report: * fe76de4bed646d6d3a0993ed52f4fccdfac752fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17090) * e4708995ba193195b7830d9591454b9c37a9cb2d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15722: [FLINK-22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
flinkbot edited a comment on pull request #15722: URL: https://github.com/apache/flink/pull/15722#issuecomment-824810068 ## CI report: * a739239b57579b6e408becbbe8976edfd91eaa03 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17089) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
SteNicholas commented on pull request #15729: URL: https://github.com/apache/flink/pull/15729#issuecomment-825402972 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas removed a comment on pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
SteNicholas removed a comment on pull request #15729: URL: https://github.com/apache/flink/pull/15729#issuecomment-825402972 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15561: [FLINK-20695][ha] Clean up ha data for job if globally terminated
flinkbot edited a comment on pull request #15561: URL: https://github.com/apache/flink/pull/15561#issuecomment-817508957 ## CI report: * 5172b7b75cb1d1a100d9aef8804d98633621e235 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17087) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-22408) Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
[
https://issues.apache.org/jira/browse/FLINK-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329983#comment-17329983
]
Ma Jun edited comment on FLINK-22408 at 4/23/21, 5:08 AM:
--
I think the final Unparse result：
{code:java}
ALTER TABLE tbl DROP PARTITION (p1='a',p2=1), PARTITION (p1='b',p2=2);
{code}
was (Author: aidenma):
I think the final Unparse result：
{code:java}
ALTER TABLE tbl DROP PARTITION (p1='a',p2=1), PARTITION (p1='b',p2=2);
{code}
> Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
> --
>
> Key: FLINK-22408
> URL: https://issues.apache.org/jira/browse/FLINK-22408
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Planner
>Affects Versions: 1.11.3
>Reporter: Ma Jun
>Priority: Major
> Labels: pull-request-available
>
> Flink Table Parser is error：
> *Synopsis:*
>
> *SQL：*
> {code:java}
> alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);{code}
> *hive muit partition unparse toSqlString is :*
> {code:java}
> ALTER TABLE `TBL`\n" +
> "DROP\n" +
> "PARTITION (`P1` = 'a', `P2` = 1)\n" +
> "PARTITION (`P1` = 'b', `P2` = 2)
> {code}
> Missing comma in Partition SqlNodeList
> Hive syntax：
> {code:java}
> ALTER TABLE table_name DROP [IF EXISTS] PARTITION (partition_spec) [,
> PARTITION (partition_spec)];
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22408) Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
[
https://issues.apache.org/jira/browse/FLINK-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329983#comment-17329983
]
Ma Jun commented on FLINK-22408:
I think the final Unparse result：
{code:java}
alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);
{code}
> Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
> --
>
> Key: FLINK-22408
> URL: https://issues.apache.org/jira/browse/FLINK-22408
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Planner
>Affects Versions: 1.11.3
>Reporter: Ma Jun
>Priority: Major
> Labels: pull-request-available
>
> Flink Table Parser is error：
> *Synopsis:*
>
> *SQL：*
> {code:java}
> alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);{code}
> *hive muit partition unparse toSqlString is :*
> {code:java}
> ALTER TABLE `TBL`\n" +
> "DROP\n" +
> "PARTITION (`P1` = 'a', `P2` = 1)\n" +
> "PARTITION (`P1` = 'b', `P2` = 2)
> {code}
> Missing comma in Partition SqlNodeList
> Hive syntax：
> {code:java}
> ALTER TABLE table_name DROP [IF EXISTS] PARTITION (partition_spec) [,
> PARTITION (partition_spec)];
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22408) Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
[
https://issues.apache.org/jira/browse/FLINK-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329983#comment-17329983
]
Ma Jun edited comment on FLINK-22408 at 4/23/21, 5:07 AM:
--
I think the final Unparse result：
{code:java}
ALTER TABLE tbl DROP PARTITION (p1='a',p2=1), PARTITION (p1='b',p2=2);
{code}
was (Author: aidenma):
I think the final Unparse result：
{code:java}
alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);
{code}
> Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
> --
>
> Key: FLINK-22408
> URL: https://issues.apache.org/jira/browse/FLINK-22408
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Planner
>Affects Versions: 1.11.3
>Reporter: Ma Jun
>Priority: Major
> Labels: pull-request-available
>
> Flink Table Parser is error：
> *Synopsis:*
>
> *SQL：*
> {code:java}
> alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);{code}
> *hive muit partition unparse toSqlString is :*
> {code:java}
> ALTER TABLE `TBL`\n" +
> "DROP\n" +
> "PARTITION (`P1` = 'a', `P2` = 1)\n" +
> "PARTITION (`P1` = 'b', `P2` = 2)
> {code}
> Missing comma in Partition SqlNodeList
> Hive syntax：
> {code:java}
> ALTER TABLE table_name DROP [IF EXISTS] PARTITION (partition_spec) [,
> PARTITION (partition_spec)];
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22421) Does flink support multi cluster deployment with HA on multi kubernetes clusters
Bhagi created FLINK-22421: - Summary: Does flink support multi cluster deployment with HA on multi kubernetes clusters Key: FLINK-22421 URL: https://issues.apache.org/jira/browse/FLINK-22421 Project: Flink Issue Type: New Feature Components: Deployment / Kubernetes Affects Versions: 1.12.2 Environment: kubernetes Reporter: Bhagi Hi Team, i want to deploy flink in multi cluster mode on kubernetes with HA.. Please let us know, whether is this possible ?? or any solution is provided to have flink in multi cluster mode with HA .. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22408) Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
[
https://issues.apache.org/jira/browse/FLINK-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329981#comment-17329981
]
Ma Jun commented on FLINK-22408:
We mainly use hive in the table environment. We find this syntax problem in the
production environment. At present, there is a difference between hive and hive
syntax, which leads to Hive SQL failure.
> Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
> --
>
> Key: FLINK-22408
> URL: https://issues.apache.org/jira/browse/FLINK-22408
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Planner
>Affects Versions: 1.11.3
>Reporter: Ma Jun
>Priority: Major
> Labels: pull-request-available
>
> Flink Table Parser is error：
> *Synopsis:*
>
> *SQL：*
> {code:java}
> alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);{code}
> *hive muit partition unparse toSqlString is :*
> {code:java}
> ALTER TABLE `TBL`\n" +
> "DROP\n" +
> "PARTITION (`P1` = 'a', `P2` = 1)\n" +
> "PARTITION (`P1` = 'b', `P2` = 2)
> {code}
> Missing comma in Partition SqlNodeList
> Hive syntax：
> {code:java}
> ALTER TABLE table_name DROP [IF EXISTS] PARTITION (partition_spec) [,
> PARTITION (partition_spec)];
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21786) sample Job execution failed
[ https://issues.apache.org/jira/browse/FLINK-21786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329979#comment-17329979 ] Bhagi commented on FLINK-21786: --- yes i fixed it myself.Thanks > sample Job execution failed > --- > > Key: FLINK-21786 > URL: https://issues.apache.org/jira/browse/FLINK-21786 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.12.0 > Environment: kubernetes debian container >Reporter: Bhagi >Priority: Minor > Fix For: 1.12.0 > > Attachments: image-2021-03-15-10-40-49-263.png, > image-2021-03-15-10-42-00-269.png, job logs.txt, job logs.txt > > > Hi Team, > i am trying to test sample word count job , but job execution is failing. > unable to understand what exactly the issue is. Attached job execution info.. > !image-2021-03-15-10-40-49-263.png! !image-2021-03-15-10-42-00-269.png! > ./bin/flink run ./examples/streaming/WordCount.jar --input > file:///persistent/flinkData/test.txt --output > file:///persistent/flinkData/wordcount_out -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22139) Flink Jobmanager & Task Manger logs are not writing to the logs files
[
https://issues.apache.org/jira/browse/FLINK-22139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329977#comment-17329977
]
Bhagi commented on FLINK-22139:
---
Hi Till Rohrmann,
yes i deployed Flink's standalone session on kubernetes..
followed below link to setup.
https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/standalone/kubernetes.html
> Flink Jobmanager & Task Manger logs are not writing to the logs files
> -
>
> Key: FLINK-22139
> URL: https://issues.apache.org/jira/browse/FLINK-22139
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.2
> Environment: on kubernetes flink standalone deployment with
> jobmanager HA is enabled.
>Reporter: Bhagi
>Priority: Major
>
> Hi Team,
> I am submitting the jobs and restarting the job manager and task manager
> pods.. Log files are generating with the name task manager and job manager.
> but job manager & task manager log file size is '0', i am not sure any
> configuration missed..why logs are not writing to their log files..
> # Task Manager pod###
> flink@flink-taskmanager-85b6585b7-hhgl7:~$ ls -lart log/
> total 0
> -rw-r--r-- 1 flink flink 0 Apr 7 09:35
> flink--taskexecutor-0-flink-taskmanager-85b6585b7-hhgl7.log
> flink@flink-taskmanager-85b6585b7-hhgl7:~$
> ### Jobmanager pod Logs #
> flink@flink-jobmanager-f6db89b7f-lq4ps:~$
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:36
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-gtkx5.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:36
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-wnrfm.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:37
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-2b2fs.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 06:37
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-7kdhh.log
> -rw-r--r-- 1 7148739 flink 0 Apr 7 09:35
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-twhkt.log
> drwxrwxrwx 2 7148739 flink35 Apr 7 09:35 .
> -rw-r--r-- 1 7148739 flink 0 Apr 7 09:35
> flink--standalonesession-0-flink-jobmanager-f6db89b7f-lq4ps.log
> flink@flink-jobmanager-f6db89b7f-lq4ps:~$
> I configured log4j.properties for flink
> log4j.properties: |+
> monitorInterval=30
> rootLogger.level = INFO
> rootLogger.appenderRef.file.ref = MainAppender
> logger.flink.name = org.apache.flink
> logger.flink.level = INFO
> logger.akka.name = akka
> logger.akka.level = INFO
> appender.main.name = MainAppender
> appender.main.type = RollingFile
> appender.main.append = true
> appender.main.fileName = ${sys:log.file}
> appender.main.filePattern = ${sys:log.file}.%i
> appender.main.layout.type = PatternLayout
> appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x
> - %m%n
> appender.main.policies.type = Policies
> appender.main.policies.size.type = SizeBasedTriggeringPolicy
> appender.main.policies.size.size = 100MB
> appender.main.policies.startup.type = OnStartupTriggeringPolicy
> appender.main.strategy.type = DefaultRolloverStrategy
> appender.main.strategy.max = ${env:MAX_LOG_FILE_NUMBER:-10}
> logger.netty.name =
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
> logger.netty.level = OFF
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
flinkbot edited a comment on pull request #15729: URL: https://github.com/apache/flink/pull/15729#issuecomment-825360063 ## CI report: * fe76de4bed646d6d3a0993ed52f4fccdfac752fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17090) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19449) LEAD/LAG cannot work correctly in streaming mode
[ https://issues.apache.org/jira/browse/FLINK-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329970#comment-17329970 ] Jingsong Lee commented on FLINK-19449: -- Hi [~laiyijie], do you want to fix this? > LEAD/LAG cannot work correctly in streaming mode > > > Key: FLINK-19449 > URL: https://issues.apache.org/jira/browse/FLINK-19449 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.10.2, 1.11.2 >Reporter: Benchao Li >Priority: Major > Labels: stale-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] lindong28 commented on pull request #15713: [FLINK-22085][tests] Update TestUtils::tryExecute() to cancel the job after execution failure
lindong28 commented on pull request #15713: URL: https://github.com/apache/flink/pull/15713#issuecomment-825377168 Thank you @dawidwys for improving the PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 commented on a change in pull request #15713: [FLINK-22085][tests] Update TestUtils::tryExecute() to cancel the job after execution failure
lindong28 commented on a change in pull request #15713:
URL: https://github.com/apache/flink/pull/15713#discussion_r618929637
##
File path: flink-tests/src/test/java/org/apache/flink/test/util/TestUtils.java
##
@@ -18,41 +18,50 @@
package org.apache.flink.test.util;
-import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.common.JobID;
import org.apache.flink.client.ClientUtils;
import org.apache.flink.client.program.ClusterClient;
-import org.apache.flink.client.program.ProgramInvocationException;
-import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.core.execution.JobClient;
import org.apache.flink.runtime.client.JobInitializationException;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.graph.StreamGraph;
import static org.junit.Assert.fail;
/** Test utilities. */
public class TestUtils {
-public static JobExecutionResult tryExecute(StreamExecutionEnvironment
see, String name)
-throws Exception {
+// Execute the job and wait for the job result synchronously. The method
throws exception
+// iff one of the following conditions happens:
+// 1) The job finishes successfully without exception
+// 2) The job finishes with an exception that contains SuccessException.
+public static void tryExecute(StreamExecutionEnvironment see, String name)
throws Exception {
+JobClient jobClient = null;
try {
-return see.execute(name);
-} catch (ProgramInvocationException | JobExecutionException root) {
-Throwable cause = root.getCause();
-
-// search for nested SuccessExceptions
-int depth = 0;
-while (!(cause instanceof SuccessException)) {
-if (cause == null || depth++ == 20) {
-root.printStackTrace();
-fail("Test failed: " + root.getMessage());
-} else {
-cause = cause.getCause();
+StreamGraph graph = see.getStreamGraph(name);
+jobClient = see.executeAsync(graph);
+jobClient.getJobExecutionResult().get();
+} catch (Throwable root) {
+if (jobClient != null) {
+try {
+jobClient.cancel().get();
+} catch (Exception e) {
+// Exception could be thrown if the job has already
finished.
+// Ignore the exception.
}
}
-}
-return null;
+Throwable t = root;
Review comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] lindong28 commented on a change in pull request #15713: [FLINK-22085][tests] Update TestUtils::tryExecute() to cancel the job after execution failure
lindong28 commented on a change in pull request #15713:
URL: https://github.com/apache/flink/pull/15713#discussion_r618929520
##
File path: flink-tests/src/test/java/org/apache/flink/test/util/TestUtils.java
##
@@ -18,41 +18,50 @@
package org.apache.flink.test.util;
-import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.common.JobID;
import org.apache.flink.client.ClientUtils;
import org.apache.flink.client.program.ClusterClient;
-import org.apache.flink.client.program.ProgramInvocationException;
-import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.core.execution.JobClient;
import org.apache.flink.runtime.client.JobInitializationException;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.graph.StreamGraph;
import static org.junit.Assert.fail;
/** Test utilities. */
public class TestUtils {
-public static JobExecutionResult tryExecute(StreamExecutionEnvironment
see, String name)
-throws Exception {
+// Execute the job and wait for the job result synchronously. The method
throws exception
Review comment:
Thank you @dawidwys for catching this. The updated comment is much
better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-22408) Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
[
https://issues.apache.org/jira/browse/FLINK-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329965#comment-17329965
]
Jark Wu commented on FLINK-22408:
-
This is an unsupported feature and we should discuss the synax API first before
moving to pull request.
> Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
> --
>
> Key: FLINK-22408
> URL: https://issues.apache.org/jira/browse/FLINK-22408
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Planner
>Affects Versions: 1.11.3
>Reporter: Ma Jun
>Priority: Major
> Labels: pull-request-available
>
> Flink Table Parser is error：
> *Synopsis:*
>
> *SQL：*
> {code:java}
> alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);{code}
> *hive muit partition unparse toSqlString is :*
> {code:java}
> ALTER TABLE `TBL`\n" +
> "DROP\n" +
> "PARTITION (`P1` = 'a', `P2` = 1)\n" +
> "PARTITION (`P1` = 'b', `P2` = 2)
> {code}
> Missing comma in Partition SqlNodeList
> Hive syntax：
> {code:java}
> ALTER TABLE table_name DROP [IF EXISTS] PARTITION (partition_spec) [,
> PARTITION (partition_spec)];
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22408) Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
[
https://issues.apache.org/jira/browse/FLINK-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-22408:
Issue Type: New Feature (was: Bug)
> Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
> --
>
> Key: FLINK-22408
> URL: https://issues.apache.org/jira/browse/FLINK-22408
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Planner
>Affects Versions: 1.11.3
>Reporter: Ma Jun
>Priority: Major
> Labels: pull-request-available
>
> Flink Table Parser is error：
> *Synopsis:*
>
> *SQL：*
> {code:java}
> alter table tbl drop partition (p1='a',p2=1), partition(p1='b',p2=2);{code}
> *hive muit partition unparse toSqlString is :*
> {code:java}
> ALTER TABLE `TBL`\n" +
> "DROP\n" +
> "PARTITION (`P1` = 'a', `P2` = 1)\n" +
> "PARTITION (`P1` = 'b', `P2` = 2)
> {code}
> Missing comma in Partition SqlNodeList
> Hive syntax：
> {code:java}
> ALTER TABLE table_name DROP [IF EXISTS] PARTITION (partition_spec) [,
> PARTITION (partition_spec)];
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21909) Unify API and implementation for Hive and Filesystem source connector
[ https://issues.apache.org/jira/browse/FLINK-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-21909: Labels: (was: stale-major) > Unify API and implementation for Hive and Filesystem source connector > - > > Key: FLINK-21909 > URL: https://issues.apache.org/jira/browse/FLINK-21909 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem, Connectors / Hive >Reporter: Jark Wu >Priority: Major > > This should make Filesystem source connector have all the ability of Hive > source connector (including the watermark ability). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21905) Hive streaming source should use FIFO FileSplitAssigner instead of LIFO
[
https://issues.apache.org/jira/browse/FLINK-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-21905:
Labels: (was: stale-major)
> Hive streaming source should use FIFO FileSplitAssigner instead of LIFO
> ---
>
> Key: FLINK-21905
> URL: https://issues.apache.org/jira/browse/FLINK-21905
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive
>Reporter: Jark Wu
>Priority: Major
>
> Currently, Hive streaming source uses {{SimpleAssigner}} which hands out
> splits in LIFO order. However, it will result in out-of-order partition
> reading even if we add partition splits in order. Therefore, we should use a
> FIFO order {{FileSplitAssigner}} instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21908) Support HiveSource to emit watermarks which is extract from partition values
[ https://issues.apache.org/jira/browse/FLINK-21908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-21908: Labels: (was: stale-major) > Support HiveSource to emit watermarks which is extract from partition values > > > Key: FLINK-21908 > URL: https://issues.apache.org/jira/browse/FLINK-21908 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22401) Python tests fail with "Slot ... is not allocated by job ..."
[
https://issues.apache.org/jira/browse/FLINK-22401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329961#comment-17329961
]
Yingjie Cao commented on FLINK-22401:
-
[~trohrmann] [~xintongsong] Really thanks for the reply and now I understand
what is going on and what I should do.
> Python tests fail with "Slot ... is not allocated by job ..."
> -
>
> Key: FLINK-22401
> URL: https://issues.apache.org/jira/browse/FLINK-22401
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16965=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=22519
> {code}
> Apr 21 15:22:58 Caused by: org.apache.flink.util.FlinkException: Slot 0 on
> TaskExecutor 10.1.0.4:41973-2f5e95 is not allocated by job
> 7eb1e50ddf9e3cfecbcf6b0eb15df0cb.
> Apr 21 15:22:58 at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.failNoLongerAllocatedSlots(TaskExecutor.java:1993)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.syncSlotsWithSnapshotFromJobMaster(TaskExecutor.java:1977)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.access$3500(TaskExecutor.java:181)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.taskexecutor.TaskExecutor$JobManagerHeartbeatListener.lambda$reportPayload$1(TaskExecutor.java:2343)
> Apr 21 15:22:58 at java.util.Optional.ifPresent(Optional.java:159)
> Apr 21 15:22:58 at
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.taskexecutor.TaskExecutor$JobManagerHeartbeatListener.reportPayload(TaskExecutor.java:2341)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.taskexecutor.TaskExecutor$JobManagerHeartbeatListener.reportPayload(TaskExecutor.java:2317)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl.requestHeartbeat(HeartbeatManagerImpl.java:214)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.heartbeatFromJobManager(TaskExecutor.java:903)
> Apr 21 15:22:58 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Apr 21 15:22:58 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 21 15:22:58 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 21 15:22:58 at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:301)
> Apr 21 15:22:58 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> Apr 21 15:22:58 ... 19 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
flinkbot edited a comment on pull request #15729: URL: https://github.com/apache/flink/pull/15729#issuecomment-825360063 ## CI report: * fe76de4bed646d6d3a0993ed52f4fccdfac752fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17090) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22276) ExceptionHistoryEntryExtractor throws fatal error when task failure
[ https://issues.apache.org/jira/browse/FLINK-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329951#comment-17329951 ] Jin Xing commented on FLINK-22276: -- Thanks [~mapohl] ~ I tried the patch and it works very well :) > ExceptionHistoryEntryExtractor throws fatal error when task failure > --- > > Key: FLINK-22276 > URL: https://issues.apache.org/jira/browse/FLINK-22276 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.13.0 >Reporter: Jin Xing >Assignee: Matthias >Priority: Blocker > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-04-14-17-50-45-199.png, > image-2021-04-15-14-46-42-660.png, log > > > When running my batch job on Flink cluster, I got a fatal error as below and > JM exits: > !image-2021-04-14-17-50-45-199.png! > Digging into the code, when DefaultScheduler start archiving failure cause > ([https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultScheduler.java#L259),] > seems Execution#failureCause is not safely/correctly attached/updated. > I attached JM log, [~mapohl] Would you mind help verify on this ? > > The job topology is like below: > !image-2021-04-15-14-46-42-660.png|width=519,height=308! > # All edges are blocking – – parallelism is 2 and there are 6 pipelined > regions for all; > # Scheduled encountered fatal error when running ShuffleRead > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
flinkbot commented on pull request #15729: URL: https://github.com/apache/flink/pull/15729#issuecomment-825360063 ## CI report: * fe76de4bed646d6d3a0993ed52f4fccdfac752fb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15722: [FLINK-22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
flinkbot edited a comment on pull request #15722: URL: https://github.com/apache/flink/pull/15722#issuecomment-824810068 ## CI report: * 3e219e20b1dc429b5128159c24521586c0dcf9a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17051) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17053) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17088) * a739239b57579b6e408becbbe8976edfd91eaa03 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17089) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15691: [hotfix][docs] Fix some typos and sample codes syntax issues in sever…
flinkbot edited a comment on pull request #15691: URL: https://github.com/apache/flink/pull/15691#issuecomment-823715258 ## CI report: * b1233944fd7d2487e5da1284d8b39cc6712789e4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17085) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22420) UnalignedCheckpointITCase failed
[
https://issues.apache.org/jira/browse/FLINK-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-22420:
--
Description:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17052=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9442
{code:java}
Apr 22 14:28:21 at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
Apr 22 14:28:21 at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
Apr 22 14:28:21 at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
Apr 22 14:28:21 at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
Apr 22 14:28:21 at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
Apr 22 14:28:21 at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
Apr 22 14:28:21 at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
Apr 22 14:28:21 at akka.actor.ActorCell.invoke(ActorCell.scala:561)
Apr 22 14:28:21 at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
Apr 22 14:28:21 at akka.dispatch.Mailbox.run(Mailbox.scala:225)
Apr 22 14:28:21 at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
Apr 22 14:28:21 at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
Apr 22 14:28:21 at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
Apr 22 14:28:21 at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
Apr 22 14:28:21 at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Apr 22 14:28:21 Caused by: org.apache.flink.util.FlinkException: An
OperatorEvent from an OperatorCoordinator to a task was lost. Triggering task
failover to ensure consistency. Event: '[NoMoreSplitEvent]', targetTask:
Source: source (1/1) - execution #5
Apr 22 14:28:21 ... 26 more
Apr 22 14:28:21
{code}
As described in the comment
https://issues.apache.org/jira/browse/FLINK-21996?focusedCommentId=17326449=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17326449
we might need to adjust the tests to allow failover.
was:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17052=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9442
As described in the comment
https://issues.apache.org/jira/browse/FLINK-21996?focusedCommentId=17326449=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17326449
we might need to adjust the tests to allow failover.
> UnalignedCheckpointITCase failed
>
>
> Key: FLINK-22420
> URL: https://issues.apache.org/jira/browse/FLINK-22420
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17052=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9442
> {code:java}
> Apr 22 14:28:21 at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> Apr 22 14:28:21 at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> Apr 22 14:28:21 at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> Apr 22 14:28:21 at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> Apr 22 14:28:21 at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> Apr 22 14:28:21 at
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> Apr 22 14:28:21 at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> Apr 22 14:28:21 at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> Apr 22 14:28:21 at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> Apr 22 14:28:21 at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> Apr 22 14:28:21 at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> Apr 22 14:28:21 at
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> Apr 22 14:28:21 at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> Apr 22 14:28:21 at
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> Apr 22 14:28:21 at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Apr 22 14:28:21 Caused by: org.apache.flink.util.FlinkException: An
> OperatorEvent from an OperatorCoordinator to a task was lost. Triggering task
> failover to ensure consistency. Event: '[NoMoreSplitEvent]', targetTask:
> Source: source (1/1) - execution #5
> Apr 22 14:28:21 ... 26 more
> Apr 22
[jira] [Created] (FLINK-22420) UnalignedCheckpointITCase failed
Guowei Ma created FLINK-22420: - Summary: UnalignedCheckpointITCase failed Key: FLINK-22420 URL: https://issues.apache.org/jira/browse/FLINK-22420 Project: Flink Issue Type: Bug Components: Runtime / Checkpointing Affects Versions: 1.14.0 Reporter: Guowei Ma https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17052=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9442 As described in the comment https://issues.apache.org/jira/browse/FLINK-21996?focusedCommentId=17326449=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17326449 we might need to adjust the tests to allow failover. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21513) Rethink up-/down-/restartingTime metrics
[
https://issues.apache.org/jira/browse/FLINK-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329947#comment-17329947
]
Steven Zhen Wu commented on FLINK-21513:
[~trohrmann] thanks for tagging me here.
Yeah, availability can be a little tricky concept for Flink application. Users
typically ask what is the availability / uptime for my Flink application (4
nines). E.g., for micro services, availability can be measured as
success/failure rate in a naive way (there are more sophisticated and probably
accurate ways). How do we define availability for Flink? uptime doesn't capture
availability.
Also availability probably can be captured in different time scales (last hour,
last 4 hours, last 12 hours, last 24 hours, last week etc.).
> Rethink up-/down-/restartingTime metrics
>
>
> Key: FLINK-21513
> URL: https://issues.apache.org/jira/browse/FLINK-21513
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / Metrics
>Reporter: Chesnay Schepler
>Priority: Major
> Labels: stale-major
> Fix For: 1.13.0
>
>
> While thinking about FLINK-21510 I stumbled upon some issues in the the
> semantics of these metrics, both from a user perspective and from our own,
> and I think we need to clarify some things.
> h4. upTime
> This metric describes the time since the job transitioned RUNNING state.
> It is meant as a measure for how stably a deployment is.
> In the default scheduler this transitions happens before we do any actual
> scheduling work, and as a result this also includes the time it takes for the
> JM to request slots and deploy tasks. In practive this means we start the
> timer once the job has been submitted and the JobMaster/Scheduler/EG have
> been initialized.
> For the adaptive scheduler this now puts us a bit into an odd situation
> because it first acquires slots before actually transitioning the EG into a
> RUNNING state, so as is we'd end up measuring 2 slightly different things.
> The question now is whether this is a problem.
> While we could certainly stick with the definition of "time since EG switched
> to RUNNING", it raises the question what the semantics of this metric are
> should a scheduler use a different data-structure than the EG.
> In other words, what I'm looking for is a definition that is independent from
> existing data-structures; a crude example could be "The time since the job is
> in a state where the deployment of a task is possible.".
> An alternative for the adaptive scheduler would be to measure the time since
> we transitioned to WaitingForResources, with which we would also include the
> slot acquisition, but it would be inconsistent with the logs and UI (because
> they only display an INITIALIZING job).
> h4. restartingTime
> This metric describes the time since the job transitioned into a RESTARTING
> state.
> It is meant as a measure for how long the recovery in case of a job failure
> takes.
> In the default scheduler this in practice is the time between a failure
> arriving at the JM and the cancellation of tasks being completed / restart
> backoff (whichever is higher).
> This is consistent with the semantics of the upTime metric, because upTime
> also includes the time required for acquiring slots and deploying tasks.
> For the adaptive scheduler we can follow similar semantics, by measuring the
> time we spend in the {{Restarting}} state.
> However, if we stick to the definition of upTime as time spent in RUNNING,
> then we will end up with a gap for the time spent in WaitingForResources.
> h4. downTime
> This metric describes the time between the job transitioning from FAILING to
> RUNNING.
> It is meant as a measure for how long the recovery in case of a job failure
> takes.
> You may be wondering what the difference between {{downTime}} and
> {{restartingTime}} is meant to be. Unfortunately I do not have the answer to
> that.
> Presumably, at the time they were added, they were covering different parts
> of the recovery process, but since we never documented these steps explicitly
> the exact semantics are no longer clear and there are no specs that a
> scheduler can follow.
> Furthermore, this metric is currently broken because a FAILING job _never_
> transitions into RUNNING anymore.
> The default scheduler transitions from RUNNING -> RESTARTING -> RUNNING,
> whereas the adaptive scheduler cancels the job and creates a new EG.
> As it is we could probably merge downTime and restartingTime because they
> seem to cover the exact same thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22312) YARNSessionFIFOSecuredITCase>YARNSessionFIFOITCase.checkForProhibitedLogContents due to the heartbeat exception with Yarn RM
[
https://issues.apache.org/jira/browse/FLINK-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329946#comment-17329946
]
Guowei Ma commented on FLINK-22312:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17055=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf=29201
> YARNSessionFIFOSecuredITCase>YARNSessionFIFOITCase.checkForProhibitedLogContents
> due to the heartbeat exception with Yarn RM
>
>
> Key: FLINK-22312
> URL: https://issues.apache.org/jira/browse/FLINK-22312
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.3, 1.14.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16633=logs=fc5181b0-e452-5c8f-68de-1097947f6483=6b04ca5f-0b52-511d-19c9-52bf0d9fbdfa=26614
> {code:java}
> 2021-04-15T22:11:39.5648550Z java.io.InterruptedIOException: Call interrupted
> 2021-04-15T22:11:39.5649145Z at
> org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5649823Z at
> org.apache.hadoop.ipc.Client.call(Client.java:1435)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5650488Z at
> org.apache.hadoop.ipc.Client.call(Client.java:1345)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5651387Z at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5652193Z at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5652675Z at com.sun.proxy.$Proxy32.allocate(Unknown
> Source) ~[?:?]
> 2021-04-15T22:11:39.5653478Z at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> ~[hadoop-yarn-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5654223Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
> 2021-04-15T22:11:39.5654742Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_275]
> 2021-04-15T22:11:39.5655269Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_275]
> 2021-04-15T22:11:39.5655625Z ]
> 2021-04-15T22:11:39.5655853Z at org.junit.Assert.fail(Assert.java:88)
> 2021-04-15T22:11:39.5656281Z at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:576)
> 2021-04-15T22:11:39.5656831Z at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:86)
> 2021-04-15T22:11:39.5657360Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-04-15T22:11:39.565Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-04-15T22:11:39.5658252Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-04-15T22:11:39.5658723Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-04-15T22:11:39.5659311Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-04-15T22:11:39.5659780Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-04-15T22:11:39.5660248Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-04-15T22:11:39.5660829Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> 2021-04-15T22:11:39.5661247Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-15T22:11:39.5661652Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-04-15T22:11:39.5662006Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-04-15T22:11:39.5662379Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-04-15T22:11:39.5662812Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-04-15T22:11:39.5663260Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-04-15T22:11:39.5663935Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-04-15T22:11:39.5664384Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-04-15T22:11:39.5664849Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-04-15T22:11:39.5665283Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-04-15T22:11:39.5665681Z at
>
[jira] [Commented] (FLINK-21235) leaderChange_withBlockingJobManagerTermination_doesNotAffectNewLeader hang
[
https://issues.apache.org/jira/browse/FLINK-21235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329944#comment-17329944
]
Guowei Ma commented on FLINK-21235:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17078=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=66b5c59a-0094-561d-0e44-b149dfdd586d=7070
> leaderChange_withBlockingJobManagerTermination_doesNotAffectNewLeader hang
> --
>
> Key: FLINK-21235
> URL: https://issues.apache.org/jira/browse/FLINK-21235
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.3
>Reporter: Guowei Ma
>Priority: Critical
> Labels: stale-critical, test-stability
> Fix For: 1.13.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12759=logs=3b6ec2fd-a816-5e75-c775-06fb87cb6670=b33fdd4f-3de5-542e-2624-5d53167bb672]
> {code:java}
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:36)
> at
> org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunnerITCase.leaderChange_withBlockingJobManagerTermination_doesNotAffectNewLeader(DefaultDispatcherRunnerITCase.java:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithReru{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
flinkbot commented on pull request #15729: URL: https://github.com/apache/flink/pull/15729#issuecomment-825352699 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit fe76de4bed646d6d3a0993ed52f4fccdfac752fb (Fri Apr 23 03:03:28 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-22234).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329943#comment-17329943
]
Guowei Ma edited comment on FLINK-22266 at 4/23/21, 3:02 AM:
-
I found that the case still failed on the master branch
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17077=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=a0a633b8-47ef-5c5a-2806-3c13b9e48228=3885
{code:java}
Apr 22 21:44:59 [ERROR]
testRestartCheckpointCoordinatorIfStopWithSavepointFails(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
Time elapsed: 60.366 s <<< FAILURE!
Apr 22 21:44:59 java.lang.AssertionError
Apr 22 21:44:59 at org.junit.Assert.fail(Assert.java:86)
Apr 22 21:44:59 at org.junit.Assert.assertTrue(Assert.java:41)
Apr 22 21:44:59 at org.junit.Assert.assertTrue(Assert.java:52)
Apr 22 21:44:59 at
org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.testRestartCheckpointCoordinatorIfStopWithSavepointFails(JobMasterStopWithSavepointITCase.java:244)
Apr 22 21:44:59 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Apr 22 21:44:59 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 22 21:44:59 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 22 21:44:59 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 22 21:44:59 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 22 21:44:59 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Apr 22 21:44:59 at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
Apr 22 21:44:59 at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Apr 22 21:44:59 at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Apr 22 21:44:59 at
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
Apr 22 21:44:59 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
Apr 22 21:44:59 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
Apr 22 21:44:59 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
{code}
{code:java}
Apr 22 21:44:59 [ERROR]
testRestartCheckpointCoordinatorIfStopWithSavepointFails(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
Time elapsed: 60.366 s <<< ERROR!
Apr 22 21:44:59 java.lang.IllegalStateException: MiniCluster is not yet running
or has already been shut down.
Apr 22 21:44:59 at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
Apr 22 21:44:59 at
org.apache.flink.runtime.minicluster.MiniCluster.getDispatcherGatewayFuture(MiniCluster.java:850)
Apr 22 21:44:59 at
org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:750)
Apr 22 21:44:59 at
org.apache.flink.runtime.minicluster.MiniCluster.listJobs(MiniCluster.java:683)
Apr 22 21:44:59 at
org.apache.flink.client.program.MiniClusterClient.listJobs(MiniClusterClient.java:111)
Apr 22 21:44:59 at
org.apache.flink.test.util.AbstractTestBase.cleanupRunningJobs(AbstractTestBase.java:73)
Apr 22 21:44:59 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Apr 22 21:44:59 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 22 21:44:59 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 22 21:44:59 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 22 21:44:59 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 22 21:44:59 at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
Apr 22 21:44:59 at
[jira] [Reopened] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma reopened FLINK-22266:
---
I found that the case still failed.
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17077=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=a0a633b8-47ef-5c5a-2806-3c13b9e48228=3885
{code:java}
Apr 22 21:44:59 [ERROR]
testRestartCheckpointCoordinatorIfStopWithSavepointFails(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
Time elapsed: 60.366 s <<< FAILURE!
Apr 22 21:44:59 java.lang.AssertionError
Apr 22 21:44:59 at org.junit.Assert.fail(Assert.java:86)
Apr 22 21:44:59 at org.junit.Assert.assertTrue(Assert.java:41)
Apr 22 21:44:59 at org.junit.Assert.assertTrue(Assert.java:52)
Apr 22 21:44:59 at
org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.testRestartCheckpointCoordinatorIfStopWithSavepointFails(JobMasterStopWithSavepointITCase.java:244)
Apr 22 21:44:59 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Apr 22 21:44:59 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 22 21:44:59 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 22 21:44:59 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 22 21:44:59 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 22 21:44:59 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Apr 22 21:44:59 at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
Apr 22 21:44:59 at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Apr 22 21:44:59 at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Apr 22 21:44:59 at
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
Apr 22 21:44:59 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
Apr 22 21:44:59 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
Apr 22 21:44:59 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Apr 22 21:44:59 at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
{code}
{code:java}
Apr 22 21:44:59 [ERROR]
testRestartCheckpointCoordinatorIfStopWithSavepointFails(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
Time elapsed: 60.366 s <<< ERROR!
Apr 22 21:44:59 java.lang.IllegalStateException: MiniCluster is not yet running
or has already been shut down.
Apr 22 21:44:59 at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
Apr 22 21:44:59 at
org.apache.flink.runtime.minicluster.MiniCluster.getDispatcherGatewayFuture(MiniCluster.java:850)
Apr 22 21:44:59 at
org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:750)
Apr 22 21:44:59 at
org.apache.flink.runtime.minicluster.MiniCluster.listJobs(MiniCluster.java:683)
Apr 22 21:44:59 at
org.apache.flink.client.program.MiniClusterClient.listJobs(MiniClusterClient.java:111)
Apr 22 21:44:59 at
org.apache.flink.test.util.AbstractTestBase.cleanupRunningJobs(AbstractTestBase.java:73)
Apr 22 21:44:59 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Apr 22 21:44:59 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 22 21:44:59 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 22 21:44:59 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 22 21:44:59 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 22 21:44:59 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 22 21:44:59 at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
Apr 22 21:44:59 at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Apr 22 21:44:59 at
[jira] [Updated] (FLINK-22234) Read savepoint before creating ExecutionGraph
[ https://issues.apache.org/jira/browse/FLINK-22234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-22234: --- Labels: pull-request-available (was: ) > Read savepoint before creating ExecutionGraph > - > > Key: FLINK-22234 > URL: https://issues.apache.org/jira/browse/FLINK-22234 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > Savepoints are currently read when the ExecutionGraph is created. > This is causing problems when determining the (max) parallelism the job is > run with, because the savepoint may impose additional restrictions. > This caused quite a few headaches in FLINK-21844, and further requires the > execution vertices to have a mutable max parallelism. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] SteNicholas opened a new pull request #15729: [FLINK-22234][runtime] Read savepoint before creating ExecutionGraph
SteNicholas opened a new pull request #15729: URL: https://github.com/apache/flink/pull/15729 ## What is the purpose of the change *Savepoints are currently read when the `ExecutionGraph` is created. This is causing problems when determining the (max) parallelism the job is run with, because the savepoint may impose additional restrictions. This caused quite a few headaches in FLINK-21844, and further requires the execution vertices to have a mutable max parallelism. `DefaultExecutionGraphFactory` should load the savepoint and then extract from it potential constraints for the `ExecutionGraph` creation.* ## Brief change log - *Separate `Checkpoints.loadAndValidateCheckpoint` into `Checkpoints.loadCheckpoint` and `Checkpoints.validateCheckpoint` for loading the savepoint before the `ExecutionGraph` creation.* ## Verifying this change - *`CheckpointMetadataLoadingTest` updates `Checkpoints.loadAndValidateCheckpoint` to `Checkpoints.loadCheckpoint` and `Checkpoints.validateCheckpoint`.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15722: [FLINK-22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
flinkbot edited a comment on pull request #15722: URL: https://github.com/apache/flink/pull/15722#issuecomment-824810068 ## CI report: * 3e219e20b1dc429b5128159c24521586c0dcf9a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17051) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17053) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17088) * a739239b57579b6e408becbbe8976edfd91eaa03 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15727: [BP-1.13][FLINK-22368] Deque channel after releasing on EndOfPartition
flinkbot edited a comment on pull request #15727: URL: https://github.com/apache/flink/pull/15727#issuecomment-824941920 ## CI report: * 77eaa9e82f48fcb3e3613d2d77e6ca17f8625ee5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17082) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-22266:
--
Affects Version/s: 1.14.0
> Harden JobMasterStopWithSavepointITCase
> ---
>
> Key: FLINK-22266
> URL: https://issues.apache.org/jira/browse/FLINK-22266
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16451=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=3884
> {code:java}
> [ERROR]
> throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
> Time elapsed: 0.154 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithoutRestartsHelper(JobMasterStopWithSavepointITCase.java:154)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(JobMasterStopWithSavepointITCase.java:138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22345) CoordinatorEventsExactlyOnceITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-22345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-22345:
--
Affects Version/s: 1.14.0
> CoordinatorEventsExactlyOnceITCase hangs on azure
> -
>
> Key: FLINK-22345
> URL: https://issues.apache.org/jira/browse/FLINK-22345
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Dawid Wysakowicz
>Assignee: Stephan Ewen
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0, 1.12.3
>
> Attachments: screenshot-1.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16731=logs=02c4e775-43bf-5625-d1cc-542b5209e072=e5961b24-88d9-5c77-efd3-955422674c25=9896
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fa8c800b800 nid=0x58b3 waiting on
> condition [0x7fa8cfd1c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x8147a7e8> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:802)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.test(CoordinatorEventsExactlyOnceITCase.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22345) CoordinatorEventsExactlyOnceITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-22345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329941#comment-17329941
]
Guowei Ma edited comment on FLINK-22345 at 4/23/21, 2:53 AM:
-
The case failed in the master branch after the fixes merged
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17077=logs=0e7be18f-84f2-53f0-a32d-4a5e4a174679=7030a106-e977-5851-a05e-535de648c9c9=3
was (Author: maguowei):
I still find the case in the master barnch
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17077=logs=0e7be18f-84f2-53f0-a32d-4a5e4a174679=7030a106-e977-5851-a05e-535de648c9c9=3
> CoordinatorEventsExactlyOnceITCase hangs on azure
> -
>
> Key: FLINK-22345
> URL: https://issues.apache.org/jira/browse/FLINK-22345
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Assignee: Stephan Ewen
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0, 1.12.3
>
> Attachments: screenshot-1.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16731=logs=02c4e775-43bf-5625-d1cc-542b5209e072=e5961b24-88d9-5c77-efd3-955422674c25=9896
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fa8c800b800 nid=0x58b3 waiting on
> condition [0x7fa8cfd1c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x8147a7e8> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:802)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.test(CoordinatorEventsExactlyOnceITCase.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
>
[jira] [Reopened] (FLINK-22345) CoordinatorEventsExactlyOnceITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-22345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma reopened FLINK-22345:
---
I still find the case in the master barnch
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17077=logs=0e7be18f-84f2-53f0-a32d-4a5e4a174679=7030a106-e977-5851-a05e-535de648c9c9=3
> CoordinatorEventsExactlyOnceITCase hangs on azure
> -
>
> Key: FLINK-22345
> URL: https://issues.apache.org/jira/browse/FLINK-22345
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Assignee: Stephan Ewen
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0, 1.12.3
>
> Attachments: screenshot-1.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16731=logs=02c4e775-43bf-5625-d1cc-542b5209e072=e5961b24-88d9-5c77-efd3-955422674c25=9896
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fa8c800b800 nid=0x58b3 waiting on
> condition [0x7fa8cfd1c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x8147a7e8> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:802)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.test(CoordinatorEventsExactlyOnceITCase.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22418) The currently activated tab in the same page isn't consistent
[ https://issues.apache.org/jira/browse/FLINK-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-22418: Fix Version/s: 1.13.0 > The currently activated tab in the same page isn't consistent > - > > Key: FLINK-22418 > URL: https://issues.apache.org/jira/browse/FLINK-22418 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Dian Fu >Priority: Major > Fix For: 1.13.0 > > Attachments: image-2021-04-23-10-41-06-577.png > > > Currently, the activated tab isn't always the same in the same page, for > example: > !image-2021-04-23-10-41-06-577.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22419) testScheduleRunAsync fail
Guowei Ma created FLINK-22419:
-
Summary: testScheduleRunAsync fail
Key: FLINK-22419
URL: https://issues.apache.org/jira/browse/FLINK-22419
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.14.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17077=logs=a549b384-c55a-52c0-c451-00e0477ab6db=81f2da51-a161-54c7-5b84-6001fed26530=6833
{code:java}
Apr 22 22:56:40 [ERROR]
testScheduleRunAsync(org.apache.flink.runtime.rpc.RpcEndpointTest) Time
elapsed: 0.404 s <<< FAILURE!
Apr 22 22:56:40 java.lang.AssertionError
Apr 22 22:56:40 at org.junit.Assert.fail(Assert.java:86)
Apr 22 22:56:40 at org.junit.Assert.assertTrue(Assert.java:41)
Apr 22 22:56:40 at org.junit.Assert.assertTrue(Assert.java:52)
Apr 22 22:56:40 at
org.apache.flink.runtime.rpc.RpcEndpointTest.testScheduleRunAsync(RpcEndpointTest.java:318)
Apr 22 22:56:40 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Apr 22 22:56:40 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 22 22:56:40 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 22 22:56:40 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 22 22:56:40 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 22 22:56:40 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 22 22:56:40 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 22 22:56:40 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Apr 22 22:56:40 at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Apr 22 22:56:40 at
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
Apr 22 22:56:40 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Apr 22 22:56:40 at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
Apr 22 22:56:40 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
Apr 22 22:56:40 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
Apr 22 22:56:40 at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Apr 22 22:56:40 at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Apr 22 22:56:40 at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Apr 22 22:56:40 at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
Apr 22 22:56:40 at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
Apr 22 22:56:40 at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Apr 22 22:56:40 at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
Apr 22 22:56:40 at
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22418) The currently activated tab in the same page isn't consistent
Dian Fu created FLINK-22418: --- Summary: The currently activated tab in the same page isn't consistent Key: FLINK-22418 URL: https://issues.apache.org/jira/browse/FLINK-22418 Project: Flink Issue Type: Improvement Components: Documentation Affects Versions: 1.13.0 Reporter: Dian Fu Attachments: image-2021-04-23-10-41-06-577.png Currently, the activated tab isn't always the same in the same page, for example: !image-2021-04-23-10-41-06-577.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15722: [FLINK-22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
flinkbot edited a comment on pull request #15722: URL: https://github.com/apache/flink/pull/15722#issuecomment-824810068 ## CI report: * 3e219e20b1dc429b5128159c24521586c0dcf9a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17051) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17053) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17088) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] githubMJ commented on pull request #15722: [FLINK-22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error
githubMJ commented on pull request #15722: URL: https://github.com/apache/flink/pull/15722#issuecomment-825344412 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15716: [FLINK-22368] Deque channel after releasing on EndOfPartition
flinkbot edited a comment on pull request #15716: URL: https://github.com/apache/flink/pull/15716#issuecomment-82440 ## CI report: * b5559a35836d27b05b7d28d8ce85e0bbe1478372 UNKNOWN * 24807f6df37404def669c6d2bf6f82fb3cce71e0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17080) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15561: [FLINK-20695][ha] Clean up ha data for job if globally terminated
flinkbot edited a comment on pull request #15561: URL: https://github.com/apache/flink/pull/15561#issuecomment-817508957 ## CI report: * 85afc0e14a28cdf84f44a2013306e3257421 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16824) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16345) * 5172b7b75cb1d1a100d9aef8804d98633621e235 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17087) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13459: [FLINK-18199] [chinese-translation]: translate FileSystem SQL Connector page into Chinese
flinkbot edited a comment on pull request #13459: URL: https://github.com/apache/flink/pull/13459#issuecomment-697158140 ## CI report: * 95208be8c69642606dce41ba727db17831a0a580 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17084) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #15341: [FLINK-21923][table-planner-blink] Fix SplitAggregateRule not support both filter, count/sum, avg
godfreyhe commented on a change in pull request #15341: URL: https://github.com/apache/flink/pull/15341#discussion_r618880846 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/SplitAggregateRule.scala ## @@ -280,11 +282,15 @@ class SplitAggregateRule extends RelOptRule( } // STEP 2.3: construct partial aggregates -relBuilder.aggregate( - relBuilder.groupKey(fullGroupSet, ImmutableList.of[ImmutableBitSet](fullGroupSet)), +// Create aggregate node directly to avoid ClassCastException, +// Please see https://issues.apache.org/jira/browse/FLINK-21923 for more details. +val partialAggregate = FlinkLogicalAggregate.create( Review comment: can you add a TODO here (such as `// TODO reuse aggregate function, see FLINK-22412`) and add some tests about window aggregate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kanata163 commented on pull request #15705: [FLINK-22398][runtime]fix incorrect comments in InputOutputFormatVertex
kanata163 commented on pull request #15705: URL: https://github.com/apache/flink/pull/15705#issuecomment-825324955 @wuchong Could you please take a few minutes to review this? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15691: [hotfix][docs] Fix some typos and sample codes syntax issues in sever…
flinkbot edited a comment on pull request #15691: URL: https://github.com/apache/flink/pull/15691#issuecomment-823715258 ## CI report: * 8d4fb4f0ecc62b2cf95a9110517dd160b1de820d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17006) * b1233944fd7d2487e5da1284d8b39cc6712789e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17085) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15561: [FLINK-20695][ha] Clean up ha data for job if globally terminated
flinkbot edited a comment on pull request #15561: URL: https://github.com/apache/flink/pull/15561#issuecomment-817508957 ## CI report: * 85afc0e14a28cdf84f44a2013306e3257421 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16824) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16345) * 5172b7b75cb1d1a100d9aef8804d98633621e235 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-22409) Rename ResourceProfile class throwUnsupportedOperationExecptionIfUnknown method
[
https://issues.apache.org/jira/browse/FLINK-22409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-22409.
Fix Version/s: (was: 1.13.0)
Resolution: Fixed
Fixed via
* master (1.14): 59cb0b43f867acfe00a4cac97ab1fe6551857c64
> Rename ResourceProfile class throwUnsupportedOperationExecptionIfUnknown
> method
> ---
>
> Key: FLINK-22409
> URL: https://issues.apache.org/jira/browse/FLINK-22409
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.12.2
>Reporter: simenliuxing
>Assignee: simenliuxing
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> throwUnsupportedOperationExecptionIfUnknown method has a mistake name. It
> should be 'throwUnsupportedOperationExceptionIfUnknown' , and All other
> references to this method are changed to
> 'throwUnsupportedOperationExceptionIfUnknown' .
> {code:java}
> // mistake name.
> private void throwUnsupportedOperationExecptionIfUnknown() {
> if (this.equals(UNKNOWN)) {
> throw new UnsupportedOperationException();
> }
> }
> // new
> private void throwUnsupportedOperationExceptionIfUnknown() {
> if (this.equals(UNKNOWN)) {
> throw new UnsupportedOperationException();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] xintongsong closed pull request #15723: [FLINK-22409][runtime]Rename ResourceProfile class throwUnsupportedOperationExecptionIfUnknown method
xintongsong closed pull request #15723: URL: https://github.com/apache/flink/pull/15723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15691: [hotfix][docs] Fix some typos and sample codes syntax issues in sever…
flinkbot edited a comment on pull request #15691: URL: https://github.com/apache/flink/pull/15691#issuecomment-823715258 ## CI report: * 8d4fb4f0ecc62b2cf95a9110517dd160b1de820d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17006) * b1233944fd7d2487e5da1284d8b39cc6712789e4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15674: [FLINK-22311] [connector/jdbc] Validate maxRetries for XA Sink
flinkbot edited a comment on pull request #15674: URL: https://github.com/apache/flink/pull/15674#issuecomment-822428583 ## CI report: * fdb03ab25669629a5d26808e5e82360beb63a8d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17083) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-12491) Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
[
https://issues.apache.org/jira/browse/FLINK-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kezhu Wang updated FLINK-12491:
---
Fix Version/s: 1.14.0
> Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
> -
>
> Key: FLINK-12491
> URL: https://issues.apache.org/jira/browse/FLINK-12491
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Affects Versions: 1.6.4, 1.7.2, 1.8.0, 1.9.0
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Minor
> Labels: pull-request-available, stale-assigned
> Fix For: 1.14.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{CoreOptions.TMP_DIRS}} and {{ConfigConstants.TASK_MANAGER_TMP_DIR_KEY}}
> both say that:
> {quote}
> The config parameter defining the directories for temporary files, separated
> by
>* ",", "|", or the system's \{@link java.io.File#pathSeparator}.
> {quote}
> But the parsing phase uses {{String.split}} with argument {{",|" +
> File.pathSeparator}} eventually. However, in fact the sole parameter of
> {{String.split}} is a regular expression, so the directory path separators
> are "," or {{java.io.File#pathSeparator}}. After digging into history, I
> found that the documentation was introduced in commit
> {{a7c407ace4f6cbfbde3e247071cee5a755ae66db}} and inherited by
> {{76abcaa55d0d6ab704b7ab8164718e8e2dcae2c4}}. So, I think it is safe to drop
> "|" from documentation.
> {code:title=ConfigurationUtils.java}
> public class ConfigurationUtils {
> private static String[] splitPaths(@Nonnull String separatedPaths) {
> return separatedPaths.length() > 0 ? separatedPaths.split(",|"
> + File.pathSeparator) : EMPTY;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21564) CommonTestUtils.waitUntilCondition could fail with condition meets before
[
https://issues.apache.org/jira/browse/FLINK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kezhu Wang updated FLINK-21564:
---
Fix Version/s: 1.14.0
> CommonTestUtils.waitUntilCondition could fail with condition meets before
> -
>
> Key: FLINK-21564
> URL: https://issues.apache.org/jira/browse/FLINK-21564
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> {code}
> public static void waitUntilCondition(
> SupplierWithException condition,
> Deadline timeout,
> long retryIntervalMillis,
> String errorMsg)
> throws Exception {
> while (timeout.hasTimeLeft() && !condition.get()) {
> final long timeLeft = Math.max(0, timeout.timeLeft().toMillis());
> Thread.sleep(Math.min(retryIntervalMillis, timeLeft));
> }
> if (!timeout.hasTimeLeft()) {
> throw new TimeoutException(errorMsg);
> }
> }
> {code}
> The timeout could run off between truth condition and last checking.
> Besides this, I also see time-out blocking condition in some tests, the
> combination could be worse.
> Not a big issue, but worth to be aware of and solved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12491) Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
[
https://issues.apache.org/jira/browse/FLINK-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kezhu Wang updated FLINK-12491:
---
Labels: pull-request-available (was: pull-request-available stale-assigned)
> Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
> -
>
> Key: FLINK-12491
> URL: https://issues.apache.org/jira/browse/FLINK-12491
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Affects Versions: 1.6.4, 1.7.2, 1.8.0, 1.9.0
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.14.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{CoreOptions.TMP_DIRS}} and {{ConfigConstants.TASK_MANAGER_TMP_DIR_KEY}}
> both say that:
> {quote}
> The config parameter defining the directories for temporary files, separated
> by
>* ",", "|", or the system's \{@link java.io.File#pathSeparator}.
> {quote}
> But the parsing phase uses {{String.split}} with argument {{",|" +
> File.pathSeparator}} eventually. However, in fact the sole parameter of
> {{String.split}} is a regular expression, so the directory path separators
> are "," or {{java.io.File#pathSeparator}}. After digging into history, I
> found that the documentation was introduced in commit
> {{a7c407ace4f6cbfbde3e247071cee5a755ae66db}} and inherited by
> {{76abcaa55d0d6ab704b7ab8164718e8e2dcae2c4}}. So, I think it is safe to drop
> "|" from documentation.
> {code:title=ConfigurationUtils.java}
> public class ConfigurationUtils {
> private static String[] splitPaths(@Nonnull String separatedPaths) {
> return separatedPaths.length() > 0 ? separatedPaths.split(",|"
> + File.pathSeparator) : EMPTY;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-11409) Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
[ https://issues.apache.org/jira/browse/FLINK-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kezhu Wang updated FLINK-11409: --- Fix Version/s: 2.0.0 > Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces > > > Key: FLINK-11409 > URL: https://issues.apache.org/jira/browse/FLINK-11409 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Kezhu Wang >Assignee: Kezhu Wang >Priority: Major > Labels: Breaking-Change, stale-assigned > Fix For: 2.0.0 > > > I found these functions express no opinionated demands from implementing > classes. It would be nice to implement as interfaces not abstract classes as > abstract class is intrusive and hampers caller user cases. For example, > client can't write an `AbstractFlinkRichFunction` to unify lifecycle > management for all data processing functions in easy way. > I dive history of some of these functions, and find that some functions were > converted as abstract class from interface due to default method > implementation, such as `ProcessFunction` and `CoProcessFunction` were > converted to abstract classes in FLINK-4460 which predate -FLINK-7242-. After > -FLINK-7242-, [Java 8 default > method|https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html] > would be a better solution. > I notice also that some functions which are introduced after -FLINK-7242-, > such as `ProcessJoinFunction`, are implemented as abstract classes. I think > it would be better to establish a well-known principle to guide both api > authors and callers of data processing functions. > Personally, I prefer interface for all exported function callbacks for the > reason I express in first paragraph. > Besides this, with `AbstractRichFunction` and interfaces for data processing > functions I think lots of rich data processing functions can be eliminated as > they are plain classes extending `AbstractRichFunction` and implementing data > processing interfaces, clients can write this in one line code with clear > intention of both data processing and lifecycle management. > Following is a possible incomplete list of data processing functions > implemented as abstract classes currently: > * `ProcessFunction`, `KeyedProcessFunction`, `CoProcessFunction` and > `ProcessJoinFunction` > * `ProcessWindowFunction` and `ProcessAllWindowFunction` > * `BaseBroadcastProcessFunction`, `BroadcastProcessFunction` and > `KeyedBroadcastProcessFunction` > All above functions are annotated with `@PublicEvolving`, making they > interfaces won't break Flink's compatibility guarantee but compatibility is > still a big consideration to evaluate this proposal. > Any thoughts on this proposal ? Please must comment out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12491) Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
[
https://issues.apache.org/jira/browse/FLINK-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329916#comment-17329916
]
Kezhu Wang commented on FLINK-12491:
I guess it is still valid.
> Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
> -
>
> Key: FLINK-12491
> URL: https://issues.apache.org/jira/browse/FLINK-12491
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Affects Versions: 1.6.4, 1.7.2, 1.8.0, 1.9.0
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Minor
> Labels: pull-request-available, stale-assigned
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{CoreOptions.TMP_DIRS}} and {{ConfigConstants.TASK_MANAGER_TMP_DIR_KEY}}
> both say that:
> {quote}
> The config parameter defining the directories for temporary files, separated
> by
>* ",", "|", or the system's \{@link java.io.File#pathSeparator}.
> {quote}
> But the parsing phase uses {{String.split}} with argument {{",|" +
> File.pathSeparator}} eventually. However, in fact the sole parameter of
> {{String.split}} is a regular expression, so the directory path separators
> are "," or {{java.io.File#pathSeparator}}. After digging into history, I
> found that the documentation was introduced in commit
> {{a7c407ace4f6cbfbde3e247071cee5a755ae66db}} and inherited by
> {{76abcaa55d0d6ab704b7ab8164718e8e2dcae2c4}}. So, I think it is safe to drop
> "|" from documentation.
> {code:title=ConfigurationUtils.java}
> public class ConfigurationUtils {
> private static String[] splitPaths(@Nonnull String separatedPaths) {
> return separatedPaths.length() > 0 ? separatedPaths.split(",|"
> + File.pathSeparator) : EMPTY;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-13734) Support DDL in SQL CLI
[ https://issues.apache.org/jira/browse/FLINK-13734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-13734. -- Resolution: Implemented I believe this is already supported. > Support DDL in SQL CLI > -- > > Key: FLINK-13734 > URL: https://issues.apache.org/jira/browse/FLINK-13734 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Jark Wu >Priority: Major > Labels: stale-major > > We have supported DDL in TableEnvironment. We should also support to execute > DDL on SQL client to make the feature to be used more easily. However, this > might need to modify the current architecture of SQL Client. More detailed > design should be attached and discussed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21317) Downstream keyed state not work after FlinkKafkaShuffle
[
https://issues.apache.org/jira/browse/FLINK-21317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kezhu Wang updated FLINK-21317:
---
Fix Version/s: 1.13.1
1.14.0
> Downstream keyed state not work after FlinkKafkaShuffle
> ---
>
> Key: FLINK-21317
> URL: https://issues.apache.org/jira/browse/FLINK-21317
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
> Labels: stale-major
> Fix For: 1.14.0, 1.13.1
>
>
> {{FlinkKafkaShuffle}} uses
> {{KeyGroupRangeAssignment.assignKeyToParallelOperator}} to assign partition
> records to kafka topic partition. The assignment works as follow:
> # {{KeyGroupRangeAssignment.assignToKeyGroup(Object key, int
> maxParallelism)}} assigns key to key group.
> # {{KeyGroupRangeAssignment.computeOperatorIndexForKeyGroup(int
> maxParallelism, int parallelism, int keyGroupId)}} assigns that key group to
> operator/subtask index.
> When kafka topic partitions are consumed, they are redistributed by
> {{KafkaTopicPartitionAssigner.assign(KafkaTopicPartition partition, int
> numParallelSubtasks)}}. I copied code of this redistribution here.
> {code:java}
> public class KafkaTopicPartitionAssigner {
> public static int assign(KafkaTopicPartition partition, int
> numParallelSubtasks) {
> int startIndex =
> ((partition.getTopic().hashCode() * 31) & 0x7FFF) %
> numParallelSubtasks;
> // here, the assumption is that the id of Kafka partitions are always
> ascending
> // starting from 0, and therefore can be used directly as the offset
> clockwise from the
> // start index
> return (startIndex + partition.getPartition()) % numParallelSubtasks;
> }
> }
> {code}
> This partition redistribution breaks prerequisites for
> {{DataStreamUtils.reinterpretAsKeyedStream}}, that is key groups are messed
> up. The consequence is unusable keyed state. I list deepest stack trace
> captured here:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.state.heap.StateTable.transform(StateTable.java:205)
> at
> org.apache.flink.runtime.state.heap.HeapReducingState.add(HeapReducingState.java:100)
> {noformat}
> cc [~ym] [~sewen] [~AHeise] [~pnowojski]
> Below is my proposed changes:
> * Make assignment between partition and subtask customizable.
> * Provide a 0-based round-robin assignment. (This is making {{startIndex}} 0
> in existing assignment algorithms.)
> I saw FLINK-8570, above changes could be helpful if we finally decide to
> deliver FLINK-8570.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21317) Downstream keyed state not work after FlinkKafkaShuffle
[
https://issues.apache.org/jira/browse/FLINK-21317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329911#comment-17329911
]
Kezhu Wang commented on FLINK-21317:
[~flink-jira-bot] Still valid. I will update fixVersion to gain more attentions.
> Downstream keyed state not work after FlinkKafkaShuffle
> ---
>
> Key: FLINK-21317
> URL: https://issues.apache.org/jira/browse/FLINK-21317
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
> Labels: stale-major
>
> {{FlinkKafkaShuffle}} uses
> {{KeyGroupRangeAssignment.assignKeyToParallelOperator}} to assign partition
> records to kafka topic partition. The assignment works as follow:
> # {{KeyGroupRangeAssignment.assignToKeyGroup(Object key, int
> maxParallelism)}} assigns key to key group.
> # {{KeyGroupRangeAssignment.computeOperatorIndexForKeyGroup(int
> maxParallelism, int parallelism, int keyGroupId)}} assigns that key group to
> operator/subtask index.
> When kafka topic partitions are consumed, they are redistributed by
> {{KafkaTopicPartitionAssigner.assign(KafkaTopicPartition partition, int
> numParallelSubtasks)}}. I copied code of this redistribution here.
> {code:java}
> public class KafkaTopicPartitionAssigner {
> public static int assign(KafkaTopicPartition partition, int
> numParallelSubtasks) {
> int startIndex =
> ((partition.getTopic().hashCode() * 31) & 0x7FFF) %
> numParallelSubtasks;
> // here, the assumption is that the id of Kafka partitions are always
> ascending
> // starting from 0, and therefore can be used directly as the offset
> clockwise from the
> // start index
> return (startIndex + partition.getPartition()) % numParallelSubtasks;
> }
> }
> {code}
> This partition redistribution breaks prerequisites for
> {{DataStreamUtils.reinterpretAsKeyedStream}}, that is key groups are messed
> up. The consequence is unusable keyed state. I list deepest stack trace
> captured here:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.state.heap.StateTable.transform(StateTable.java:205)
> at
> org.apache.flink.runtime.state.heap.HeapReducingState.add(HeapReducingState.java:100)
> {noformat}
> cc [~ym] [~sewen] [~AHeise] [~pnowojski]
> Below is my proposed changes:
> * Make assignment between partition and subtask customizable.
> * Provide a 0-based round-robin assignment. (This is making {{startIndex}} 0
> in existing assignment algorithms.)
> I saw FLINK-8570, above changes could be helpful if we finally decide to
> deliver FLINK-8570.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21578) Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition
[
https://issues.apache.org/jira/browse/FLINK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329910#comment-17329910
]
Kezhu Wang commented on FLINK-21578:
Still valid. I will update fix version to 1.14 to gain attention in that
release cycle.
> Closeable Sink Committer/GlobalCommitter were created to function in onestep
> during job graph composition
> -
>
> Key: FLINK-21578
> URL: https://issues.apache.org/jira/browse/FLINK-21578
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
> Labels: stale-major
>
> Normally, functions/operators are created in job graph composition phase for
> serialization and transmission. Them are "opened" in flink cluster to
> function. This two steps procedure succeed in that there will be no
> resource-cleanup requirement in job graph composition phase.
> While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but
> they were created in job graph composition phase.
> Following are fixes I could image if we converge to "this is problematic".
> # Add {{open}} or similar method for these two classes.
> # Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make
> {{createCommitter}} and others not optional(enforce this in runtime).
> Personally, I am a bit preferring second approach for possible less code path
> touching in job graph composition phase. But first approach has advantage
> that it could be an no breaking change.
> There might be other approaches though.
> cc [~guoweima] [~gaoyunhaii] [~aljoscha] [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21578) Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition
[
https://issues.apache.org/jira/browse/FLINK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kezhu Wang updated FLINK-21578:
---
Fix Version/s: 1.14.0
> Closeable Sink Committer/GlobalCommitter were created to function in onestep
> during job graph composition
> -
>
> Key: FLINK-21578
> URL: https://issues.apache.org/jira/browse/FLINK-21578
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
> Fix For: 1.14.0
>
>
> Normally, functions/operators are created in job graph composition phase for
> serialization and transmission. Them are "opened" in flink cluster to
> function. This two steps procedure succeed in that there will be no
> resource-cleanup requirement in job graph composition phase.
> While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but
> they were created in job graph composition phase.
> Following are fixes I could image if we converge to "this is problematic".
> # Add {{open}} or similar method for these two classes.
> # Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make
> {{createCommitter}} and others not optional(enforce this in runtime).
> Personally, I am a bit preferring second approach for possible less code path
> touching in job graph composition phase. But first approach has advantage
> that it could be an no breaking change.
> There might be other approaches though.
> cc [~guoweima] [~gaoyunhaii] [~aljoscha] [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21578) Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition
[
https://issues.apache.org/jira/browse/FLINK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kezhu Wang updated FLINK-21578:
---
Labels: (was: stale-major)
> Closeable Sink Committer/GlobalCommitter were created to function in onestep
> during job graph composition
> -
>
> Key: FLINK-21578
> URL: https://issues.apache.org/jira/browse/FLINK-21578
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
>
> Normally, functions/operators are created in job graph composition phase for
> serialization and transmission. Them are "opened" in flink cluster to
> function. This two steps procedure succeed in that there will be no
> resource-cleanup requirement in job graph composition phase.
> While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but
> they were created in job graph composition phase.
> Following are fixes I could image if we converge to "this is problematic".
> # Add {{open}} or similar method for these two classes.
> # Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make
> {{createCommitter}} and others not optional(enforce this in runtime).
> Personally, I am a bit preferring second approach for possible less code path
> touching in job graph composition phase. But first approach has advantage
> that it could be an no breaking change.
> There might be other approaches though.
> cc [~guoweima] [~gaoyunhaii] [~aljoscha] [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] paul8263 commented on pull request #15691: [hotfix][docs] Fix some typos and sample codes syntax issues in sever…
paul8263 commented on pull request #15691: URL: https://github.com/apache/flink/pull/15691#issuecomment-825308568 > Could you also fix the code-related issues in the chinese version of the docs? (under docs/content.zh) Those corresponding typos and syntax issues were fixed in docs/content.zh folder. Is there anything else I can help? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13459: [FLINK-18199] [chinese-translation]: translate FileSystem SQL Connector page into Chinese
flinkbot edited a comment on pull request #13459: URL: https://github.com/apache/flink/pull/13459#issuecomment-697158140 ## CI report: * c611b4e8d50ab8fd9607a2bf44bd8a357de2f491 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17047) * 95208be8c69642606dce41ba727db17831a0a580 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17084) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13459: [FLINK-18199] [chinese-translation]: translate FileSystem SQL Connector page into Chinese
flinkbot edited a comment on pull request #13459: URL: https://github.com/apache/flink/pull/13459#issuecomment-697158140 ## CI report: * c611b4e8d50ab8fd9607a2bf44bd8a357de2f491 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17047) * 95208be8c69642606dce41ba727db17831a0a580 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-1294) Add Performance and Tuning Guide to documentation
[ https://issues.apache.org/jira/browse/FLINK-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1294: -- Labels: auto-closed (was: stale-minor) > Add Performance and Tuning Guide to documentation > - > > Key: FLINK-1294 > URL: https://issues.apache.org/jira/browse/FLINK-1294 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Robert Metzger >Priority: Minor > Labels: auto-closed > > Our documentation currently lacks some advice on how to write Flink > applications efficiently and how to tune Flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-6029) Strange linebreaks in web dashboard graph view make it hard to read text
[
https://issues.apache.org/jira/browse/FLINK-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-6029:
--
Labels: auto-closed (was: stale-minor)
> Strange linebreaks in web dashboard graph view make it hard to read text
>
>
> Key: FLINK-6029
> URL: https://issues.apache.org/jira/browse/FLINK-6029
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Web Frontend
>Affects Versions: 1.2.0
>Reporter: Luke Hutchison
>Priority: Minor
> Labels: auto-closed
> Attachments: Screenshot_2017-03-13_01-40-52.png
>
>
> Text lines inside the boxes in the Flink web dashboard have linebreaks at
> very odd places, and it can make the content of the boxes hard to read. (See
> attached screenshot.)
> For clarity the content could be tabulated and indented, e.g with each "->"
> arrow at the start of a new text line along with the operation type ("->
> Filter"), and the description indented on the line below, with long
> descriptions wrapped to the same start indentation, so that it's easy to
> visually separate out operations from descriptions based on indentation:
> {noformat}
> Filter
> (filter for problems with date <= 2017-01)
> -> FlatMap
> (calculate problem severity scores for
> date 2017-01)
> -> Combine
> (sum all severity scores for (bin, apt) for
> date 2017-01)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-6889) Improve Path parsing for Windows paths with scheme
[
https://issues.apache.org/jira/browse/FLINK-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329778#comment-17329778
]
Flink Jira Bot commented on FLINK-6889:
---
This issue has been labeled "stale-minor" for 7 days. It is closed now. If you
are still affected by this or would like to raise the priority of this ticket
please re-open, removing the label "auto-closed" and raise the ticket priority
accordingly.
> Improve Path parsing for Windows paths with scheme
> --
>
> Key: FLINK-6889
> URL: https://issues.apache.org/jira/browse/FLINK-6889
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: stale-minor
>
> The {{Path}} parsing for Windows paths with a scheme (for example
> {{file:\\\C:\text.txt}} could be improved a bit.
> For example, if a scheme is present the path is never identified as a Windows
> path because that is determined by whether it starts with the drive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-6322) Mesos task labels
[ https://issues.apache.org/jira/browse/FLINK-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-6322: -- Labels: auto-closed (was: stale-minor) > Mesos task labels > - > > Key: FLINK-6322 > URL: https://issues.apache.org/jira/browse/FLINK-6322 > Project: Flink > Issue Type: Improvement > Components: Deployment / Mesos >Reporter: Eron Wright >Priority: Minor > Labels: auto-closed > > Task labels serve many purposes in Mesos, such a tagging tasks for > log-aggregation purposes. > I propose a new configuration setting for a list of 'key=value' labels to be > applied to TM instances. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14193) Update Web UI for fine grained TM/Slot resources
[ https://issues.apache.org/jira/browse/FLINK-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329884#comment-17329884 ] Flink Jira Bot commented on FLINK-14193: This major issue is unassigned and itself and all of its Sub-Tasks have not been updated for 30 days. So, it has been labeled "stale-major". If this ticket is indeed "major", please either assign yourself or give an update. Afterwards, please remove the label. In 7 days the issue will be deprioritized. > Update Web UI for fine grained TM/Slot resources > > > Key: FLINK-14193 > URL: https://issues.apache.org/jira/browse/FLINK-14193 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination, Runtime / Web Frontend >Reporter: Xintong Song >Priority: Major > Labels: stale-major > > * Update RestAPI / WebUI to properly display information of available > resources and allocated slots of task executors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-9639) Order "User Configuration" Alphabetically on Flink Dashboard
[ https://issues.apache.org/jira/browse/FLINK-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329732#comment-17329732 ] Flink Jira Bot commented on FLINK-9639: --- This issue has been labeled "stale-minor" for 7 days. It is closed now. If you are still affected by this or would like to raise the priority of this ticket please re-open, removing the label "auto-closed" and raise the ticket priority accordingly. > Order "User Configuration" Alphabetically on Flink Dashboard > > > Key: FLINK-9639 > URL: https://issues.apache.org/jira/browse/FLINK-9639 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.4.0 >Reporter: Guilherme Nobre >Priority: Minor > Labels: stale-minor > > The *User Configuration* list is not ordered alphabetically. Would be nice to > do so, even more for people that like me have configurations with prefixes. > This way the configurations would be nicely ordered :) > You can access the configuration screen here: > [flinkdashboard]/#/jobs/[jobId]/config - replace [flinkdashboard] and [jobId] > with your own > Or just click on a job and then on the *Configuration* tab -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-11255) RemoteStreamEnvironment should be able to execute in detached mode
[
https://issues.apache.org/jira/browse/FLINK-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot closed FLINK-11255.
--
Resolution: Auto Closed
> RemoteStreamEnvironment should be able to execute in detached mode
> --
>
> Key: FLINK-11255
> URL: https://issues.apache.org/jira/browse/FLINK-11255
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.7.0
>Reporter: Benjamin Lee
>Priority: Minor
> Labels: auto-closed
>
> h1. Overview
> Currently RemoteStreamEnvironment is not capable of submitting a job via
> detached mode.
> h1. Proposed Changes
> * Modify the signature of
> {code:java}
> StreamExecutionEnvironment#createRemoteEnvironment(...) :
> StreamExecutionEnvironment{code}
> to
> {code:java}
> StreamExecutionEnvironment#createRemoteEnvironment(...) :
> RemoteStreamEnvironment{code}
> * Add an public overloaded _execute_ method in _RemoteStreamEnvironment_
> that allows detached executions.
> There was a related Jira FLINK-6224 opened a while ago, but I don't think it
> makes sense to change the abstract class _StreamExecutionEnvironment_ since
> detached submission only applies to _RemoteStreamEnvironment_?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21438) Broken links in content.zh/docs/concepts/flink-architecture.md
[ https://issues.apache.org/jira/browse/FLINK-21438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-21438: --- Labels: pull-request-available stale-major (was: pull-request-available) > Broken links in content.zh/docs/concepts/flink-architecture.md > -- > > Key: FLINK-21438 > URL: https://issues.apache.org/jira/browse/FLINK-21438 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Ting Sun >Priority: Major > Labels: pull-request-available, stale-major > Fix For: 1.13.0 > > > When reading the Chinese doc I find some links are broken and the responses > for these links are 404. And I find that this is because of the format error: > these links are different from their corresponding original links in the > English doc, while the links which are identical to the corresponding links > in the English doc are OK for me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21926) Document Fine Grained Resource Management
[ https://issues.apache.org/jira/browse/FLINK-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-21926: --- Labels: stale-major (was: ) > Document Fine Grained Resource Management > - > > Key: FLINK-21926 > URL: https://issues.apache.org/jira/browse/FLINK-21926 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Xintong Song >Priority: Major > Labels: stale-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-1261) Handling of full temporary directories at startup and runtime
[ https://issues.apache.org/jira/browse/FLINK-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1261: -- Labels: auto-closed (was: stale-minor) > Handling of full temporary directories at startup and runtime > - > > Key: FLINK-1261 > URL: https://issues.apache.org/jira/browse/FLINK-1261 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 0.8.0 >Reporter: Ufuk Celebi >Priority: Minor > Labels: auto-closed > > On a related issue about logging the temporary directory paths, we had a > discussion about what to do about full temporary directories at startup and > runtime. > This issue is the new home for that discussion. ;-) See FLINK-1142 and the > [PR on GitHub for the > discussion|https://github.com/apache/incubator-flink/pull/188] for the > previous discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21317) Downstream keyed state not work after FlinkKafkaShuffle
[
https://issues.apache.org/jira/browse/FLINK-21317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329881#comment-17329881
]
Flink Jira Bot commented on FLINK-21317:
This major issue is unassigned and itself and all of its Sub-Tasks have not
been updated for 30 days. So, it has been labeled "stale-major". If this ticket
is indeed "major", please either assign yourself or give an update. Afterwards,
please remove the label. In 7 days the issue will be deprioritized.
> Downstream keyed state not work after FlinkKafkaShuffle
> ---
>
> Key: FLINK-21317
> URL: https://issues.apache.org/jira/browse/FLINK-21317
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
> Labels: stale-major
>
> {{FlinkKafkaShuffle}} uses
> {{KeyGroupRangeAssignment.assignKeyToParallelOperator}} to assign partition
> records to kafka topic partition. The assignment works as follow:
> # {{KeyGroupRangeAssignment.assignToKeyGroup(Object key, int
> maxParallelism)}} assigns key to key group.
> # {{KeyGroupRangeAssignment.computeOperatorIndexForKeyGroup(int
> maxParallelism, int parallelism, int keyGroupId)}} assigns that key group to
> operator/subtask index.
> When kafka topic partitions are consumed, they are redistributed by
> {{KafkaTopicPartitionAssigner.assign(KafkaTopicPartition partition, int
> numParallelSubtasks)}}. I copied code of this redistribution here.
> {code:java}
> public class KafkaTopicPartitionAssigner {
> public static int assign(KafkaTopicPartition partition, int
> numParallelSubtasks) {
> int startIndex =
> ((partition.getTopic().hashCode() * 31) & 0x7FFF) %
> numParallelSubtasks;
> // here, the assumption is that the id of Kafka partitions are always
> ascending
> // starting from 0, and therefore can be used directly as the offset
> clockwise from the
> // start index
> return (startIndex + partition.getPartition()) % numParallelSubtasks;
> }
> }
> {code}
> This partition redistribution breaks prerequisites for
> {{DataStreamUtils.reinterpretAsKeyedStream}}, that is key groups are messed
> up. The consequence is unusable keyed state. I list deepest stack trace
> captured here:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.state.heap.StateTable.transform(StateTable.java:205)
> at
> org.apache.flink.runtime.state.heap.HeapReducingState.add(HeapReducingState.java:100)
> {noformat}
> cc [~ym] [~sewen] [~AHeise] [~pnowojski]
> Below is my proposed changes:
> * Make assignment between partition and subtask customizable.
> * Provide a 0-based round-robin assignment. (This is making {{startIndex}} 0
> in existing assignment algorithms.)
> I saw FLINK-8570, above changes could be helpful if we finally decide to
> deliver FLINK-8570.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-1044) Website: Offer a zip archive with a pre-setup user project
[ https://issues.apache.org/jira/browse/FLINK-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot closed FLINK-1044. - Resolution: Auto Closed > Website: Offer a zip archive with a pre-setup user project > -- > > Key: FLINK-1044 > URL: https://issues.apache.org/jira/browse/FLINK-1044 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Stephan Ewen >Priority: Minor > Labels: auto-closed, starter > Attachments: flink-project.zip > > > This is basically a shortcut for those that are not familiar with maven > archetypes or do not have maven installed (other then as part of the Eclipse > IDE or so). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-1298) Consolidate Handling of User Code ClassLoader
[ https://issues.apache.org/jira/browse/FLINK-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1298: -- Labels: auto-closed (was: stale-minor) > Consolidate Handling of User Code ClassLoader > - > > Key: FLINK-1298 > URL: https://issues.apache.org/jira/browse/FLINK-1298 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: Aljoscha Krettek >Priority: Minor > Labels: auto-closed > > Right now, we use setContextClassLoader() and getContextClassLoader() in some > places and in other places we implicitly pass the ClassLoader around. This is > not consistent and in some places we might not be sure where to get the class > loader from. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-1335) Reintroduce documentation for the runtime/execution engine
[ https://issues.apache.org/jira/browse/FLINK-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329864#comment-17329864 ] Flink Jira Bot commented on FLINK-1335: --- This issue has been labeled "stale-minor" for 7 days. It is closed now. If you are still affected by this or would like to raise the priority of this ticket please re-open, removing the label "auto-closed" and raise the ticket priority accordingly. > Reintroduce documentation for the runtime/execution engine > -- > > Key: FLINK-1335 > URL: https://issues.apache.org/jira/browse/FLINK-1335 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Jonas Traub >Priority: Minor > Labels: stale-minor > > It has been some month now, since I started implementing a runtime-operator > for Flink the fist time. Thereby, I discovered that there is no good > introduction to the concepts/programming model of the flink runtime. Anyhow, > it is necessary to have knowledge about this in order to implement a new > operators. > There is a short documentation at [1], but this is not really sufficient. > I googled a bit and found [2] and [3]. Although, this documentation is not > complete and possibly outdated at some points, it gave me good introduction. > It would be nice if the content could be updated and re-added to the > documentation. > [1]http://flink.incubator.apache.org/docs/0.7-incubating/internal_add_operator.html#implementing-a-new-runtime-operator > [2]http://robertmetzger.de/stratosphere/docs/pre-0.4/internals/nephele.html > [3]http://robertmetzger.de/stratosphere/docs/pre-0.4/internals/pact.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-7751) Documentation for Homebrew installation is unclear
[
https://issues.apache.org/jira/browse/FLINK-7751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-7751:
--
Labels: auto-closed (was: stale-minor)
> Documentation for Homebrew installation is unclear
> --
>
> Key: FLINK-7751
> URL: https://issues.apache.org/jira/browse/FLINK-7751
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.3.2
> Environment: MacOS
>Reporter: Chris Ward
>Priority: Minor
> Labels: auto-closed
>
> Relating to the addition of Homebrew as an installation options for Mac in
> FLINK-3903, the steps for installing on a Mac are not helpful, as the next
> step mentioned
> {code:none}
> $ ./bin/start-local.sh # Start Flink
> {code}
> Has no information on where to find this script. Solutions could be to
> mention in the docs where to find this, or instead to use
> https://github.com/Homebrew/homebrew-services to connect to the script(s) for
> a better experience in the long term. I'll probably undertake the first
> option and see how plausible the second is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21288) Support redundant task managers for fine grained resource management
[ https://issues.apache.org/jira/browse/FLINK-21288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-21288: --- Labels: stale-major (was: ) > Support redundant task managers for fine grained resource management > > > Key: FLINK-21288 > URL: https://issues.apache.org/jira/browse/FLINK-21288 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Xintong Song >Priority: Major > Labels: stale-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-1513) Remove GlobalConfiguration Singleton
[ https://issues.apache.org/jira/browse/FLINK-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1513: -- Labels: auto-closed (was: stale-minor) > Remove GlobalConfiguration Singleton > > > Key: FLINK-1513 > URL: https://issues.apache.org/jira/browse/FLINK-1513 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 0.9 >Reporter: Stephan Ewen >Priority: Minor > Labels: auto-closed > > The global configuration is currently held as a singleton. This makes some > accessing config values convenient, but discourages proper passing of > parameters between components. It also makes it very clumsy to set up local > execution and totally prevents testing heterogeneous config settings. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21925) Support configuring fine grained resource requirements via DataStream API
[ https://issues.apache.org/jira/browse/FLINK-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329878#comment-17329878 ] Flink Jira Bot commented on FLINK-21925: This major issue is unassigned and itself and all of its Sub-Tasks have not been updated for 30 days. So, it has been labeled "stale-major". If this ticket is indeed "major", please either assign yourself or give an update. Afterwards, please remove the label. In 7 days the issue will be deprioritized. > Support configuring fine grained resource requirements via DataStream API > - > > Key: FLINK-21925 > URL: https://issues.apache.org/jira/browse/FLINK-21925 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Reporter: Xintong Song >Priority: Major > Labels: stale-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-1092) Allow operations, other than join and cogroup, on the solution set
[ https://issues.apache.org/jira/browse/FLINK-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329874#comment-17329874 ] Flink Jira Bot commented on FLINK-1092: --- This issue has been labeled "stale-minor" for 7 days. It is closed now. If you are still affected by this or would like to raise the priority of this ticket please re-open, removing the label "auto-closed" and raise the ticket priority accordingly. > Allow operations, other than join and cogroup, on the solution set > -- > > Key: FLINK-1092 > URL: https://issues.apache.org/jira/browse/FLINK-1092 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: Vasia Kalavri >Priority: Minor > Labels: feature, stale-minor > > Currently, join and cogroup are the only operations allowed on the solution > set. We should make the solution set usable for any type of operation, > possibly using special functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-1256) Use Kryo for user code object serialization
[ https://issues.apache.org/jira/browse/FLINK-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot closed FLINK-1256. - Resolution: Auto Closed > Use Kryo for user code object serialization > --- > > Key: FLINK-1256 > URL: https://issues.apache.org/jira/browse/FLINK-1256 > Project: Flink > Issue Type: Improvement > Components: API / DataSet, API / DataStream >Reporter: Till Rohrmann >Priority: Minor > Labels: auto-closed > > Currently, Flink uses java serialization for user code objects (UDFs). This > limits the set of supported types which can be caught in the closure or are > contained as members in the user code object. Maybe we can add Kryo as a > second serialization mechanism which is used if the Java serialization fails. > That way we would support more complex user code objects which finally > improves the user experience. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20838) Implement SlotRequestAdapter for the FineGrainedSlotManager
[ https://issues.apache.org/jira/browse/FLINK-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20838: --- Labels: stale-major (was: ) > Implement SlotRequestAdapter for the FineGrainedSlotManager > --- > > Key: FLINK-20838 > URL: https://issues.apache.org/jira/browse/FLINK-20838 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Yangze Guo >Priority: Major > Labels: stale-major > > Implement an adapter for the deprecated slot request protocol. The adapter > will be removed along with the slot request protocol in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-1258) Merge iterative and non-iterative tasks
[ https://issues.apache.org/jira/browse/FLINK-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot closed FLINK-1258. - Resolution: Auto Closed > Merge iterative and non-iterative tasks > --- > > Key: FLINK-1258 > URL: https://issues.apache.org/jira/browse/FLINK-1258 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 0.7.0-incubating >Reporter: Ufuk Celebi >Priority: Minor > Labels: auto-closed > > Currently, the local runtime distinguishes between iterative > (AbstractIterativeTask) and non-iterative (BatchTask) tasks. > We could simplify the task hierarchy and logic by merging these. > Non-iterative tasks would simply be tasks, which have only a single iteration. > I think that this could also translate to simplified logic for the readers on > the lower distributed runtime layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-1092) Allow operations, other than join and cogroup, on the solution set
[ https://issues.apache.org/jira/browse/FLINK-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1092: -- Labels: auto-closed feature (was: feature stale-minor) > Allow operations, other than join and cogroup, on the solution set > -- > > Key: FLINK-1092 > URL: https://issues.apache.org/jira/browse/FLINK-1092 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: Vasia Kalavri >Priority: Minor > Labels: auto-closed, feature > > Currently, join and cogroup are the only operations allowed on the solution > set. We should make the solution set usable for any type of operation, > possibly using special functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-2217) Web client does not remove uploaded JARs
[ https://issues.apache.org/jira/browse/FLINK-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot closed FLINK-2217. - Resolution: Auto Closed > Web client does not remove uploaded JARs > > > Key: FLINK-2217 > URL: https://issues.apache.org/jira/browse/FLINK-2217 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 0.10.0 >Reporter: Ufuk Celebi >Priority: Minor > Labels: auto-closed > > I don't know if this is a bug or feature: after stopping the webclient and > restarting it, previously uploaded jar files are still available under the > program listings. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-6912) Consider changing the RichFunction#open method signature to take no arguments.
[ https://issues.apache.org/jira/browse/FLINK-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-6912: -- Labels: auto-closed (was: stale-minor) > Consider changing the RichFunction#open method signature to take no arguments. > -- > > Key: FLINK-6912 > URL: https://issues.apache.org/jira/browse/FLINK-6912 > Project: Flink > Issue Type: Sub-task > Components: API / DataSet, API / DataStream >Affects Versions: 1.3.0 >Reporter: Mikhail Pryakhin >Priority: Minor > Labels: auto-closed > Fix For: 2.0.0 > > > RichFunction#open(org.apache.flink.configuration.Configuration) method takes > a Configuration instance as an argument which is always [passed as a new > instance|https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractUdfStreamOperator.java#L111] > bearing no configuration parameters. As I figured out it is a remnant of the > past since that method signature originates from the Record API. Consider > changing the RichFunction#open method signature to take no arguments as well > as actualizing java docs. > You can find the complete discussion > [here|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/RichMapFunction-setup-method-td13696.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-10561) Flink doesn't start locally if remove log package
[ https://issues.apache.org/jira/browse/FLINK-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-10561: --- Labels: auto-closed (was: stale-minor) > Flink doesn't start locally if remove log package > - > > Key: FLINK-10561 > URL: https://issues.apache.org/jira/browse/FLINK-10561 > Project: Flink > Issue Type: Bug > Components: Deployment / Scripts >Affects Versions: 1.6.1 > Environment: windows 7 >Reporter: Aleksei Starikov >Priority: Minor > Labels: auto-closed > Attachments: start-cluster_without_log_package.png > > > Download flink from > [https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-scala_2.11.tgz]. > Unzip archive. > Remove log package. > Try to start locally using command .\bin\start-cluster.bat > > Flink doesn't start and you get message: > > D:\work\prog\flink-1.6.1>.\bin\start-cluster.bat > Системе не удается найти указанный путь. > Starting a local cluster with one JobManager process and one TaskManager > process. > You can terminate the processes via CTRL-C in the spawned shell windows. > Web interface by default on [http://localhost:8081/]. > Системе не удается найти указанный путь. > Системе не удается найти указанный путь. > > In Russian "Системе не удается найти указанный путь" means "The system cannot > find the path specified". > > Flink can create this package himself or make more sensible message. -- This message was sent by Atlassian Jira (v8.3.4#803005)
