[GitHub] [flink] flinkbot edited a comment on pull request #12962: [FLINK-18694] Add unaligned checkpoint config to web ui
flinkbot edited a comment on pull request #12962: URL: https://github.com/apache/flink/pull/12962#issuecomment-662526701 ## CI report: * Unknown: [CANCELED](TBD) * d2275584151f5a0a342af15e1d06f33da1237d62 UNKNOWN * d49f2f7ffe75a2e39dc3b8925831c60b56683952 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 217c88f0f382bf45143dcce82a089568528dad34 UNKNOWN * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #9188: [FLINK-12940][docs-zh] Translate Apache Cassandra Connector page into…
flinkbot edited a comment on pull request #9188: URL: https://github.com/apache/flink/pull/9188#issuecomment-667675114 ## CI report: * fc4dfc21708ce2a5f96883943aabc3f763a936c3 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/178156194) * 02633f3d642efda655c420da187d232616c55c0a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5151) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #9008: [FLINK-12942][docs-zh] Translate Elasticsearch Connector page into Ch…
flinkbot edited a comment on pull request #9008: URL: https://github.com/apache/flink/pull/9008#issuecomment-667675075 ## CI report: * 5c26184832b9ed1b3aa75e4af2d3b645aba3cf9a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5150) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12962: [FLINK-18694] Add unaligned checkpoint config to web ui
flinkbot edited a comment on pull request #12962: URL: https://github.com/apache/flink/pull/12962#issuecomment-662526701 ## CI report: * Unknown: [CANCELED](TBD) * d2275584151f5a0a342af15e1d06f33da1237d62 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kbohinski commented on pull request #12962: [FLINK-18694] Add unaligned checkpoint config to web ui
kbohinski commented on pull request #12962: URL: https://github.com/apache/flink/pull/12962#issuecomment-668388353 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #9188: [FLINK-12940][docs-zh] Translate Apache Cassandra Connector page into…
flinkbot edited a comment on pull request #9188: URL: https://github.com/apache/flink/pull/9188#issuecomment-667675114 ## CI report: * fc4dfc21708ce2a5f96883943aabc3f763a936c3 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/178156194) * 02633f3d642efda655c420da187d232616c55c0a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #9008: [FLINK-12942][docs-zh] Translate Elasticsearch Connector page into Ch…
flinkbot edited a comment on pull request #9008: URL: https://github.com/apache/flink/pull/9008#issuecomment-667675075 ## CI report: * 4aeb231dd64d18cfb39dad3558516ec9228d980c Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/178156187) * 5c26184832b9ed1b3aa75e4af2d3b645aba3cf9a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 217c88f0f382bf45143dcce82a089568528dad34 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] klion26 commented on pull request #13045: [FLINK-18748][checkpointing] trigger unperiodic checkpoint immediately
klion26 commented on pull request #13045: URL: https://github.com/apache/flink/pull/13045#issuecomment-668383757 @DG-Wangtao thanks for the work. LGTM from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XuQianJin-Stars commented on pull request #9008: [FLINK-12942][docs-zh] Translate Elasticsearch Connector page into Ch…
XuQianJin-Stars commented on pull request #9008: URL: https://github.com/apache/flink/pull/9008#issuecomment-668378044 > @XuQianJin-Stars thanks for the contribution. Seems there are some conflicts now, could you please rebase the master to resolve the conflict? well, let me rebase it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18806) Taskmanager doesn't start up with error in config
[
https://issues.apache.org/jira/browse/FLINK-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170542#comment-17170542
]
Yangze Guo commented on FLINK-18806:
FYI: That has been fixed in FLINK-18639.
> Taskmanager doesn't start up with error in config
> -
>
> Key: FLINK-18806
> URL: https://issues.apache.org/jira/browse/FLINK-18806
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Scripts
>Affects Versions: 1.11.1
>Reporter: Nico Kruber
>Priority: Major
>
> With the following (wrong) configuration setting in {{flink-conf.yaml}}, a
> taskmanager will not start up, basically print nothing on the command line,
> and have no log file to look at:
> {code}
> taskmanager.memory.managed.fraction: '0.4'
> {code}
> Console output:
> {code}
> > ./bin/start-cluster.sh
> Starting cluster.
> Starting standalonesession daemon on host Nico-PC.lan.
> [ERROR] The execution result is empty.
> [ERROR] Could not get JVM parameters and dynamic configurations properly.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16361) FLIP-84: Improve & Refactor API of TableEnvironment & Table
[
https://issues.apache.org/jira/browse/FLINK-16361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170540#comment-17170540
]
godfrey he commented on FLINK-16361:
Hi [~dian.fu], all the subtasks are finished in 1.11, except the last one
[FLINK-17774|https://issues.apache.org/jira/browse/FLINK-17774]. which is
finished in 1.12.
> FLIP-84: Improve & Refactor API of TableEnvironment & Table
> ---
>
> Key: FLINK-16361
> URL: https://issues.apache.org/jira/browse/FLINK-16361
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.12.0
>
>
> as the
> [FLIP-84|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878]
> document described,
> We propose to deprecate the following methods:
> {code:java}
> TableEnvironment.sqlUpdate(String)
> TableEnvironment.insertInto(String, Table)
> TableEnvironment.execute(String)
> TableEnvironment.explain(boolean)
> TableEnvironment.fromTableSource(TableSource)
> Table.insertInto(String)
> {code}
> meanwhile, we propose to introduce the following new methods:
> {code:java}
> interface TableEnvironment {
> // execute the given single statement, and return the execution result.
> TableResult executeSql(String statement);
>
> // get the AST and the execution plan for the given single statement
> (DQL, DML)
> String explainSql(String statement, ExplainDetail... extraDetails);
>
> // create a StatementSet instance which can add DML statements or Tables
> // to the set and explain or execute them as a batch.
> StatementSet createStatementSet();
> }
> interface Table {
> // write the Table to a TableSink that was registered
> // under the specified path.
> TableResult executeInsert(String tablePath);
>
> // write the Table to a TableSink that was registered
> // under the specified path.
> TableResult executeInsert(String tablePath, boolean overwrite);
>
> // create a StatementSet instance which can add DML statements or Tables
> // to the set and explain or execute them as a batch.
> String explain(ExplainDetail... extraDetails);
>
> // get the contents of the current table.
> TableResult execute();
> }
> interface TableResult {
> // return JobClient if a Flink job is submitted
> // (for DML/DQL statement), else return empty (e.g. for DDL).
> Optional getJobClient();
>
> // return the schema of the result
> TableSchema getTableSchema();
>
> // return the ResultKind which can avoid custom parsing of
> // an "OK" row in programming
> ResultKind getResultKind();
>
> // get the row contents as an iterable rows
> Iterator collect();
>
> // print the result contents
> void print();
> }
> public enum ResultKind {
> // for DDL, DCL and statements with a simple "OK"
> SUCCESS,
>
> // rows with important content are available (DML, DQL)
> SUCCESS_WITH_CONTENT
> }
> interface StatementSet {
> // add single INSERT statement into the set
> StatementSet addInsertSql(String statement);
>
> // add Table with the given sink table name to the set
> StatementSet addInsert(String targetPath, Table table);
>
> // add Table with the given sink table name to the set
> StatementSet addInsert(String targetPath, Table table, boolean overwrite);
>
> // returns the AST and the execution plan to compute
> // the result of all statements and Tables
> String explain(ExplainDetail... extraDetails);
>
> // execute all statements and Tables as a batch
> TableResult execute();
> }
> public enum ExplainDetail {
>STATE_SIZE_ESTIMATE,
>UID,
>HINTS,
>...
> }
> {code}
> We unify the Flink table program trigger behavior, and propose that: for
> {{TableEnvironment}} and {{StreamTableEnvironment}}, you must use
> {{TableEnvironment.execute()}} to trigger table program execution, once you
> convert the table program to a {{DataStream}} program (through
> {{toAppendStream}} or {{toRetractStream}} method), you must use
> {{StreamExecutionEnvironment.execute}} to trigger the {{DataStream}} program.
> Similar rule for BatchTableEnvironment, you must use
> `TableEnvironment.execute()` to trigger batch table program execution, once
> you convert the table program (through `toDataSet` method) to a DataSet
> program, you must use `ExecutionEnvironment.execute` to trigger the DataSet
> program.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18756) Support IF NOT EXISTS for CREATE TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-18756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-18756. --- Resolution: Fixed Implemented in master (1.12.0): 8df10fa589b0897071321aeb3a4db2936e69b778 > Support IF NOT EXISTS for CREATE TABLE statement > > > Key: FLINK-18756 > URL: https://issues.apache.org/jira/browse/FLINK-18756 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Jark Wu >Assignee: Leonard Xu >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > > Currently, the CREATE TABLE DDL statement doesn't support IF NOT EXISTS. I > think this is a useful feature we missed to support, because all the other > CREATE DDLs support it. > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #13037: [FLINK-18756][table-api] Support IF NOT EXISTS for CREATE TABLE statement
wuchong merged pull request #13037: URL: https://github.com/apache/flink/pull/13037 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
flinkbot edited a comment on pull request #13009: URL: https://github.com/apache/flink/pull/13009#issuecomment-665109698 ## CI report: * d7133401edf323e5be894758398098e70cf70506 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5138) * 557e982407c488191ade7cc66807662796b3a306 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5148) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18803) JobGraph cannot be GC when submit via RemoteStreamEnvironment in attach mode
[
https://issues.apache.org/jira/browse/FLINK-18803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Libin Qin updated FLINK-18803:
--
Description:
When submit job using RemoteStreamEnvironment in attach mode. The client
submission thread is blocked on "jobResultFuture.get()" in the "submitJob"
method of RestClusterClient.java, it holds the local variable jobGraph, if the
job is complex with lots of vertexs and edges or client submits quite a lot of
jobs. The size of jobGraph become large and the client may OOM. I think there
is no need for client to hold it.
The biggest objects of client heap is as below ,The number of tasks of this
job is 408
!image-2020-08-03-18-03-29-748.png!
!image-2020-08-03-18-08-50-811.png!
perhaps we can null out it after success of submission
{code:java}
//代码占位符
public JobSubmissionResult run(FlinkPlan compiledPlan,
List libraries, List classpaths, ClassLoader classLoader,
SavepointRestoreSettings savepointSettings)
throws ProgramInvocationException {
return submitJob(() -> getJobGraph(flinkConfig, compiledPlan, libraries,
classpaths, savepointSettings), classLoader);
}
public JobSubmissionResult submitJob(Supplier jobGraphSupplier,
ClassLoader classLoader) throws ProgramInvocationException {
JobGraph jobGraph = jobGraphSupplier.get();
JobID jobID = jobGraph.getJobID();
log.info("Submitting job {} (detached: {}).", jobID, isDetached());
final CompletableFuture jobSubmissionFuture =
submitJob(jobGraph);
JobSubmissionResult result;
try {
result = jobSubmissionFuture.get();
//help GC
jobGraph = null;
} catch (Exception e) {
throw new ProgramInvocationException("Could not submit job",
jobID, ExceptionUtils.stripExecutionException(e));
}
if (isDetached()) {
return result;
} else {
final CompletableFuture jobResultFuture =
requestJobResult(jobID);
final JobResult jobResult;
try {
jobResult = jobResultFuture.get();
} catch (Exception e) {
throw new ProgramInvocationException("Could not retrieve the execution
result.",
jobID, ExceptionUtils.stripExecutionException(e));
}
try {
this.lastJobExecutionResult =
jobResult.toJobExecutionResult(classLoader);
return lastJobExecutionResult;
} catch (JobExecutionException e) {
throw new ProgramInvocationException("Job failed.", jobID, e);
} catch (IOException | ClassNotFoundException e) {
throw new ProgramInvocationException("Job failed.", jobID, e);
}
}
}
{code}
we can see the job graph has been GC
!image-2020-08-03-18-10-08-353.png!
was:
When submit job using RemoteStreamEnvironment in attach mode. The client
submission thread is blocked on "jobResultFuture.get()" in the "submitJob"
method of RestClusterClient.java, it holds the local variable jobGraph, if the
job is complex with lots of vertexs and edges or client submits quite a lot of
jobs. The size of jobGraph become large and the client may OOM. I think there
is no need for client to hold it.
The biggest objects of client heap is as below ,The number of tasks of this
job is more than 408
!image-2020-08-03-18-03-29-748.png!
!image-2020-08-03-18-08-50-811.png!
perhaps we can null out it after success of submission
{code:java}
//代码占位符
public JobSubmissionResult run(FlinkPlan compiledPlan,
List libraries, List classpaths, ClassLoader classLoader,
SavepointRestoreSettings savepointSettings)
throws ProgramInvocationException {
return submitJob(() -> getJobGraph(flinkConfig, compiledPlan, libraries,
classpaths, savepointSettings), classLoader);
}
public JobSubmissionResult submitJob(Supplier jobGraphSupplier,
ClassLoader classLoader) throws ProgramInvocationException {
JobGraph jobGraph = jobGraphSupplier.get();
JobID jobID = jobGraph.getJobID();
log.info("Submitting job {} (detached: {}).", jobID, isDetached());
final CompletableFuture jobSubmissionFuture =
submitJob(jobGraph);
JobSubmissionResult result;
try {
result = jobSubmissionFuture.get();
//help GC
jobGraph = null;
} catch (Exception e) {
throw new ProgramInvocationException("Could not submit job",
jobID, ExceptionUtils.stripExecutionException(e));
}
if (isDetached()) {
return result;
} else {
final CompletableFuture jobResultFuture =
requestJobResult(jobID);
final JobResult jobResult;
try {
jobResult = jobResultFuture.get();
} catch (Exception e) {
throw new ProgramInvocationException("Could not retrieve the execution
result.",
jobID, ExceptionUtils.stripExecutionException(e));
}
try {
this.lastJobExecutionResult =
jobResult.toJobExecutionResult(classLoader);
return lastJobExecutionResult;
} catch
[jira] [Created] (FLINK-18808) Task-level numRecordsOut metric may be underestimated
ming li created FLINK-18808:
---
Summary: Task-level numRecordsOut metric may be underestimated
Key: FLINK-18808
URL: https://issues.apache.org/jira/browse/FLINK-18808
Project: Flink
Issue Type: Improvement
Components: Runtime / Metrics
Affects Versions: 1.11.1
Environment: !image-2020-08-04-11-26-21-490.png!
Reporter: ming li
Attachments: image-2020-08-04-11-28-13-800.png,
image-2020-08-04-11-32-20-678.png
At present, we only register task-level numRecordsOut metric by reusing
operator output record counter at the end of OperatorChain.
{code:java}
//代码占位符
if (config.isChainEnd()) {
operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
}
{code}
If we only send data out through the last operator of OperatorChain, there is
no problem with this statistics. But consider the following scenario:
!image-2020-08-04-11-28-13-800.png|width=507,height=174!
In this JobGraph, we not only send data in the last operator, but also send
data in the middle operator of OperatorChain (the map operator just returns the
original value directly). Below is one of our test topology, we can see that
the statistics actually only have half of the total data received by the
downstream.
!image-2020-08-04-11-32-20-678.png|width=648,height=251!
I think the data sent out by the intermediate operator should also be counted
into the numRecordsOut of the Task. But currently we are not reusing operators
output record counters in the intermediate operators, which leads to our
task-level numRecordsOut metric is underestimated (although this has no effect
on the actual operation of the job, it may affect our monitoring).
A simple idea of mine is to modify the condition of reusing operators output
record counter:
{code:java}
//代码占位符
if (!config.getNonChainedOutputs(getUserCodeClassloader()).isEmpty()) {
operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
}{code}
In addition, I have another question: If a record is broadcast to all
downstream, should the numRecordsOut counter increase by one or the downstream
number? It seems that currently we are adding one to calculate the
numRecordsOut metric.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18808) Task-level numRecordsOut metric may be underestimated
[
https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ming li updated FLINK-18808:
Environment: (was: !image-2020-08-04-11-26-21-490.png!)
> Task-level numRecordsOut metric may be underestimated
> -
>
> Key: FLINK-18808
> URL: https://issues.apache.org/jira/browse/FLINK-18808
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Affects Versions: 1.11.1
>Reporter: ming li
>Priority: Major
> Attachments: image-2020-08-04-11-28-13-800.png,
> image-2020-08-04-11-32-20-678.png
>
>
> At present, we only register task-level numRecordsOut metric by reusing
> operator output record counter at the end of OperatorChain.
> {code:java}
> //代码占位符
> if (config.isChainEnd()) {
>operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
> }
> {code}
> If we only send data out through the last operator of OperatorChain, there is
> no problem with this statistics. But consider the following scenario:
> !image-2020-08-04-11-28-13-800.png|width=507,height=174!
> In this JobGraph, we not only send data in the last operator, but also send
> data in the middle operator of OperatorChain (the map operator just returns
> the original value directly). Below is one of our test topology, we can see
> that the statistics actually only have half of the total data received by the
> downstream.
> !image-2020-08-04-11-32-20-678.png|width=648,height=251!
> I think the data sent out by the intermediate operator should also be counted
> into the numRecordsOut of the Task. But currently we are not reusing
> operators output record counters in the intermediate operators, which leads
> to our task-level numRecordsOut metric is underestimated (although this has
> no effect on the actual operation of the job, it may affect our monitoring).
> A simple idea of mine is to modify the condition of reusing operators
> output record counter:
> {code:java}
> //代码占位符
> if (!config.getNonChainedOutputs(getUserCodeClassloader()).isEmpty()) {
>operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
> }{code}
> In addition, I have another question: If a record is broadcast to all
> downstream, should the numRecordsOut counter increase by one or the
> downstream number? It seems that currently we are adding one to calculate the
> numRecordsOut metric.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
flinkbot edited a comment on pull request #13009: URL: https://github.com/apache/flink/pull/13009#issuecomment-665109698 ## CI report: * d7133401edf323e5be894758398098e70cf70506 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5138) * 557e982407c488191ade7cc66807662796b3a306 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17726) Scheduler should take care of tasks directly canceled by TaskManager
[ https://issues.apache.org/jira/browse/FLINK-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170531#comment-17170531 ] Zhu Zhu commented on FLINK-17726: - Hi [~nicholasjiang], is there any updates for the design? > Scheduler should take care of tasks directly canceled by TaskManager > > > Key: FLINK-17726 > URL: https://issues.apache.org/jira/browse/FLINK-17726 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination, Runtime / Task >Affects Versions: 1.11.0, 1.12.0 >Reporter: Zhu Zhu >Assignee: Nicholas Jiang >Priority: Critical > Fix For: 1.12.0, 1.11.2 > > > JobManager will not trigger failure handling when receiving CANCELED task > update. > This is because CANCELED tasks are usually caused by another FAILED task. > These CANCELED tasks will be restarted by the failover process triggered > FAILED task. > However, if a task is directly CANCELED by TaskManager due to its own runtime > issue, the task will not be recovered by JM and thus the job would hang. > This is a potential issue and we should avoid it. > A possible solution is to let JobManager treat tasks transitioning to > CANCELED from all states except from CANCELING as failed tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18641) "Failure to finalize checkpoint" error in MasterTriggerRestoreHook
[
https://issues.apache.org/jira/browse/FLINK-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170530#comment-17170530
]
Jiangjie Qin commented on FLINK-18641:
--
[~pnowojski] [~SleePy] Somehow the PR info was not updated in the ticket. Would
you help take a look?
[https://github.com/apache/flink/pull/13044]
> "Failure to finalize checkpoint" error in MasterTriggerRestoreHook
> --
>
> Key: FLINK-18641
> URL: https://issues.apache.org/jira/browse/FLINK-18641
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.0
>Reporter: Brian Zhou
>Assignee: Jiangjie Qin
>Priority: Major
> Labels: pull-request-available
>
> https://github.com/pravega/flink-connectors is a Pravega connector for Flink.
> The ReaderCheckpointHook[1] class uses the Flink `MasterTriggerRestoreHook`
> interface to trigger the Pravega checkpoint during Flink checkpoints to make
> sure the data recovery. The checkpoint recovery tests are running fine in
> Flink 1.10, but it has below issues in Flink 1.11 causing the tests time out.
> Suspect it is related to the checkpoint coordinator thread model changes in
> Flink 1.11
> Error stacktrace:
> {code}
> 2020-07-09 15:39:39,999 30945 [jobmanager-future-thread-5] WARN
> o.a.f.runtime.jobmaster.JobMaster - Error while processing checkpoint
> acknowledgement message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize
> the pending checkpoint 3. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1033)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:948)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.lambda$acknowledgeCheckpoint$4(SchedulerBase.java:802)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.SerializedThrowable: Pending checkpoint has
> not been fully acknowledged yet
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
> at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:298)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1021)
> ... 9 common frames omitted
> {code}
> More detail in this mailing thread:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Pravega-connector-cannot-recover-from-the-checkpoint-due-to-quot-Failure-to-finalize-checkpoint-quot-td36652.html
> Also in https://github.com/pravega/flink-connectors/issues/387
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16579) Upgrade Calcite version to 1.23 for Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170525#comment-17170525 ] Leonard Xu commented on FLINK-16579: +1 for this. > Upgrade Calcite version to 1.23 for Flink SQL > - > > Key: FLINK-16579 > URL: https://issues.apache.org/jira/browse/FLINK-16579 > Project: Flink > Issue Type: Task > Components: Table SQL / Planner >Reporter: Jark Wu >Priority: Major > > A taks to upgrade Calcite version to 1.23. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-web] KarmaGYZ edited a comment on pull request #365: Add blog post for External Resources
KarmaGYZ edited a comment on pull request #365: URL: https://github.com/apache/flink-web/pull/365#issuecomment-668356135 cc @tillrohrmann @MarkSfik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-16824) Creating Temporal Table Function via DDL
[ https://issues.apache.org/jira/browse/FLINK-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-16824: --- Assignee: Leonard Xu (was: Leonardo Aoun) > Creating Temporal Table Function via DDL > > > Key: FLINK-16824 > URL: https://issues.apache.org/jira/browse/FLINK-16824 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Konstantin Knauf >Assignee: Leonard Xu >Priority: Major > Fix For: 1.12.0 > > > Currently, a Temporal Table Function can only be created via the Table API or > indirectly via the configuration file of the SQL Client. > It would be great, if this was also possible in pure SQL via a DDL statement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-16824) Creating Temporal Table Function via DDL
[ https://issues.apache.org/jira/browse/FLINK-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-16824: --- Assignee: Leonardo Aoun > Creating Temporal Table Function via DDL > > > Key: FLINK-16824 > URL: https://issues.apache.org/jira/browse/FLINK-16824 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Konstantin Knauf >Assignee: Leonardo Aoun >Priority: Major > Fix For: 1.12.0 > > > Currently, a Temporal Table Function can only be created via the Table API or > indirectly via the configuration file of the SQL Client. > It would be great, if this was also possible in pure SQL via a DDL statement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-web] KarmaGYZ commented on pull request #365: Add blog post for External Resources
KarmaGYZ commented on pull request #365: URL: https://github.com/apache/flink-web/pull/365#issuecomment-668356135 cc @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] KarmaGYZ opened a new pull request #365: Add blog post for External Resources
KarmaGYZ opened a new pull request #365: URL: https://github.com/apache/flink-web/pull/365 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12966: [FLINK-17427][table sql/planner]Support SupportsPartitionPushDown in …
flinkbot edited a comment on pull request #12966: URL: https://github.com/apache/flink/pull/12966#issuecomment-662861328 ## CI report: * 6697110fb14fef778707f74b66227df2953c487c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5107) * 1790d93b79cfdfc3f65a7805e444699736f80d93 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5122) * 9a05a030c661def4a45b36370dbcfa5e786ed8dc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5147) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16824) Creating Temporal Table Function via DDL
[ https://issues.apache.org/jira/browse/FLINK-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170524#comment-17170524 ] Leonard Xu commented on FLINK-16824: I'd like to support this feature, [~jark] could you assign this ticket to me ? > Creating Temporal Table Function via DDL > > > Key: FLINK-16824 > URL: https://issues.apache.org/jira/browse/FLINK-16824 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Konstantin Knauf >Priority: Major > Fix For: 1.12.0 > > > Currently, a Temporal Table Function can only be created via the Table API or > indirectly via the configuration file of the SQL Client. > It would be great, if this was also possible in pure SQL via a DDL statement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12966: [FLINK-17427][table sql/planner]Support SupportsPartitionPushDown in …
flinkbot edited a comment on pull request #12966: URL: https://github.com/apache/flink/pull/12966#issuecomment-662861328 ## CI report: * 6697110fb14fef778707f74b66227df2953c487c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5107) * 1790d93b79cfdfc3f65a7805e444699736f80d93 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5122) * 9a05a030c661def4a45b36370dbcfa5e786ed8dc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15792) Make Flink logs accessible via kubectl logs per default
[
https://issues.apache.org/jira/browse/FLINK-15792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170521#comment-17170521
]
Yang Wang commented on FLINK-15792:
---
[~rmetzger] Thanks for your input. I will update the old attached PR today.
> Make Flink logs accessible via kubectl logs per default
> ---
>
> Key: FLINK-15792
> URL: https://issues.apache.org/jira/browse/FLINK-15792
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> I think we should make Flink's logs accessible via {{kubectl logs}} per
> default. Firstly, this is the idiomatic way to obtain the logs from a
> container on Kubernetes. Secondly, especially if something does not work and
> the container cannot start/stops abruptly, there is no way to log into the
> container and look for the log.file. This makes debugging the setup quite
> hard.
> I think the best way would be to create the Flink Docker image in such a way
> that it logs to stdout. In order to allow access to the log file from the web
> ui, it should also create a log file. One way to achieve this is to add a
> ConsoleAppender to the respective logging configuration. Another way could be
> to start the process in the console mode and then to teeing the stdout output
> into the log file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] RocMarshal commented on pull request #13056: [FLINK-18798][docs-zh] Translate "Debugging Windows & Event Time" page of "Debugging & Monitoring" into Chinese
RocMarshal commented on pull request #13056: URL: https://github.com/apache/flink/pull/13056#issuecomment-668313570 Hi, @klion26 . Could you help me to review this PR if you have free time? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16361) FLIP-84: Improve & Refactor API of TableEnvironment & Table
[
https://issues.apache.org/jira/browse/FLINK-16361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170509#comment-17170509
]
Dian Fu commented on FLINK-16361:
-
Hi [~godfreyhe], it seems that all the subtasks have already been merged to
1.11. Is there still work need to do? Otherwise I think we can close this
ticket and update the target version as "1.11.0"?
> FLIP-84: Improve & Refactor API of TableEnvironment & Table
> ---
>
> Key: FLINK-16361
> URL: https://issues.apache.org/jira/browse/FLINK-16361
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.12.0
>
>
> as the
> [FLIP-84|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878]
> document described,
> We propose to deprecate the following methods:
> {code:java}
> TableEnvironment.sqlUpdate(String)
> TableEnvironment.insertInto(String, Table)
> TableEnvironment.execute(String)
> TableEnvironment.explain(boolean)
> TableEnvironment.fromTableSource(TableSource)
> Table.insertInto(String)
> {code}
> meanwhile, we propose to introduce the following new methods:
> {code:java}
> interface TableEnvironment {
> // execute the given single statement, and return the execution result.
> TableResult executeSql(String statement);
>
> // get the AST and the execution plan for the given single statement
> (DQL, DML)
> String explainSql(String statement, ExplainDetail... extraDetails);
>
> // create a StatementSet instance which can add DML statements or Tables
> // to the set and explain or execute them as a batch.
> StatementSet createStatementSet();
> }
> interface Table {
> // write the Table to a TableSink that was registered
> // under the specified path.
> TableResult executeInsert(String tablePath);
>
> // write the Table to a TableSink that was registered
> // under the specified path.
> TableResult executeInsert(String tablePath, boolean overwrite);
>
> // create a StatementSet instance which can add DML statements or Tables
> // to the set and explain or execute them as a batch.
> String explain(ExplainDetail... extraDetails);
>
> // get the contents of the current table.
> TableResult execute();
> }
> interface TableResult {
> // return JobClient if a Flink job is submitted
> // (for DML/DQL statement), else return empty (e.g. for DDL).
> Optional getJobClient();
>
> // return the schema of the result
> TableSchema getTableSchema();
>
> // return the ResultKind which can avoid custom parsing of
> // an "OK" row in programming
> ResultKind getResultKind();
>
> // get the row contents as an iterable rows
> Iterator collect();
>
> // print the result contents
> void print();
> }
> public enum ResultKind {
> // for DDL, DCL and statements with a simple "OK"
> SUCCESS,
>
> // rows with important content are available (DML, DQL)
> SUCCESS_WITH_CONTENT
> }
> interface StatementSet {
> // add single INSERT statement into the set
> StatementSet addInsertSql(String statement);
>
> // add Table with the given sink table name to the set
> StatementSet addInsert(String targetPath, Table table);
>
> // add Table with the given sink table name to the set
> StatementSet addInsert(String targetPath, Table table, boolean overwrite);
>
> // returns the AST and the execution plan to compute
> // the result of all statements and Tables
> String explain(ExplainDetail... extraDetails);
>
> // execute all statements and Tables as a batch
> TableResult execute();
> }
> public enum ExplainDetail {
>STATE_SIZE_ESTIMATE,
>UID,
>HINTS,
>...
> }
> {code}
> We unify the Flink table program trigger behavior, and propose that: for
> {{TableEnvironment}} and {{StreamTableEnvironment}}, you must use
> {{TableEnvironment.execute()}} to trigger table program execution, once you
> convert the table program to a {{DataStream}} program (through
> {{toAppendStream}} or {{toRetractStream}} method), you must use
> {{StreamExecutionEnvironment.execute}} to trigger the {{DataStream}} program.
> Similar rule for BatchTableEnvironment, you must use
> `TableEnvironment.execute()` to trigger batch table program execution, once
> you convert the table program (through `toDataSet` method) to a DataSet
> program, you must use `ExecutionEnvironment.execute` to trigger the DataSet
> program.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hackergin commented on a change in pull request #11896: [FLINK-14356] [single-value] Introduce "single-value" format to (de)serialize message to a single field
hackergin commented on a change in pull request #11896:
URL: https://github.com/apache/flink/pull/11896#discussion_r464431110
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/format/single/SingleValueFormatFactory.java
##
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.format.single;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory.Context;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import java.util.Collections;
+import java.util.Set;
+
+/**
+ * SingleValueFormatFactory for single value.
+ */
+public class SingleValueFormatFactory implements DeserializationFormatFactory,
SerializationFormatFactory {
Review comment:
I tried move the code to flink-table-api-java-bridge , but
ScanRuntimeProviderContext , SinkRuntimeProviderContext, and RowDataTypeInfo
are needed by the unit test. But they are in flink-table-runtime module. I
don't know whether if there is a good way to replace them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
flinkbot edited a comment on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-667904442 ## CI report: * 1a2f39a71a931295f82a9bdbd36f2707f42298dd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5115) * e274e166aed3ffacc4ac58409aaf3384c937cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5146) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18675) Checkpoint not maintaining minimum pause duration between checkpoints
[
https://issues.apache.org/jira/browse/FLINK-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170497#comment-17170497
]
Congxian Qiu(klion26) commented on FLINK-18675:
---
[~raviratnakar] I think the problem here is that {{CheckpointRequestDecider}}
has a wrong value of {{lastCheckpointCompletionRelativeTime}} when checking
whether the checkpoint request is too early.
1. We retrieve the value of {{lastCheckpointCompletionRelativeTime}} when
calling {{CheckpointRequestDecider#chooseRequestToExecute}} in
{{CheckpointCoordinator#triggerCheckpoint}}
2. A pending checkpoint complete, and update the valuable
{{pendingCheckpoints}} and {{lastCheckpointCompletionRelativeTime}}
3. In {{CheckpointRequestDecider#chooseRequestToExecute}} we use the previous
{{lastCheckpointCompletionRelativeTime}} to check whether current checkpoint
request is too early
I think we can get the value of {{lastCheckpointCompletionRelativeTime}} in
{{CheckpointRequestDecider#chooseRequestToExecute}} here to solve the problem
here.
> Checkpoint not maintaining minimum pause duration between checkpoints
> -
>
> Key: FLINK-18675
> URL: https://issues.apache.org/jira/browse/FLINK-18675
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.0
> Environment: !image.png!
>Reporter: Ravi Bhushan Ratnakar
>Priority: Critical
> Attachments: image.png
>
>
> I am running a streaming job with Flink 1.11.0 using kubernetes
> infrastructure. I have configured checkpoint configuration like below
> Interval - 3 minutes
> Minimum pause between checkpoints - 3 minutes
> Checkpoint timeout - 10 minutes
> Checkpointing Mode - Exactly Once
> Number of Concurrent Checkpoint - 1
>
> Other configs
> Time Characteristics - Processing Time
>
> I am observing an usual behaviour. *When a checkpoint completes successfully*
> *and if it's end to end duration is almost equal or greater than Minimum
> pause duration then the next checkpoint gets triggered immediately without
> maintaining the Minimum pause duration*. Kindly notice this behaviour from
> checkpoint id 194 onward in the attached screenshot
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16768) HadoopS3RecoverableWriterITCase.testRecoverWithStateWithMultiPart hangs
[
https://issues.apache.org/jira/browse/FLINK-16768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170492#comment-17170492
]
Dian Fu commented on FLINK-16768:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5140=logs=e9af9cde-9a65-5281-a58e-2c8511d36983=603cb7fd-6f38-5c99-efca-877e1439232f
> HadoopS3RecoverableWriterITCase.testRecoverWithStateWithMultiPart hangs
> ---
>
> Key: FLINK-16768
> URL: https://issues.apache.org/jira/browse/FLINK-16768
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.10.0, 1.11.0, 1.12.0
>Reporter: Zhijiang
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> Logs:
> [https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6584=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=d26b3528-38b0-53d2-05f7-37557c2405e4]
> {code:java}
> 2020-03-24T15:52:18.9196862Z "main" #1 prio=5 os_prio=0
> tid=0x7fd36c00b800 nid=0xc21 runnable [0x7fd3743ce000]
> 2020-03-24T15:52:18.9197235Zjava.lang.Thread.State: RUNNABLE
> 2020-03-24T15:52:18.9197536Z at
> java.net.SocketInputStream.socketRead0(Native Method)
> 2020-03-24T15:52:18.9197931Z at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> 2020-03-24T15:52:18.9198340Z at
> java.net.SocketInputStream.read(SocketInputStream.java:171)
> 2020-03-24T15:52:18.9198749Z at
> java.net.SocketInputStream.read(SocketInputStream.java:141)
> 2020-03-24T15:52:18.9199171Z at
> sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> 2020-03-24T15:52:18.9199840Z at
> sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
> 2020-03-24T15:52:18.9200265Z at
> sun.security.ssl.InputRecord.read(InputRecord.java:532)
> 2020-03-24T15:52:18.9200663Z at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
> 2020-03-24T15:52:18.9201213Z - locked <0x927583d8> (a
> java.lang.Object)
> 2020-03-24T15:52:18.9201589Z at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
> 2020-03-24T15:52:18.9202026Z at
> sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
> 2020-03-24T15:52:18.9202583Z - locked <0x92758c00> (a
> sun.security.ssl.AppInputStream)
> 2020-03-24T15:52:18.9203029Z at
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
> 2020-03-24T15:52:18.9203558Z at
> org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198)
> 2020-03-24T15:52:18.9204121Z at
> org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
> 2020-03-24T15:52:18.9204626Z at
> org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
> 2020-03-24T15:52:18.9205121Z at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> 2020-03-24T15:52:18.9205679Z at
> com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> 2020-03-24T15:52:18.9206164Z at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> 2020-03-24T15:52:18.9206786Z at
> com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125)
> 2020-03-24T15:52:18.9207361Z at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> 2020-03-24T15:52:18.9207839Z at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> 2020-03-24T15:52:18.9208327Z at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> 2020-03-24T15:52:18.9208809Z at
> com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> 2020-03-24T15:52:18.9209273Z at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> 2020-03-24T15:52:18.9210003Z at
> com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107)
> 2020-03-24T15:52:18.9210658Z at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> 2020-03-24T15:52:18.9211154Z at
> org.apache.hadoop.fs.s3a.S3AInputStream.lambda$read$3(S3AInputStream.java:445)
> 2020-03-24T15:52:18.9211631Z at
> org.apache.hadoop.fs.s3a.S3AInputStream$$Lambda$42/1936375962.execute(Unknown
> Source)
> 2020-03-24T15:52:18.9212044Z at
> org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> 2020-03-24T15:52:18.9212553Z at
> org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
> 2020-03-24T15:52:18.9212972Z at
> org.apache.hadoop.fs.s3a.Invoker$$Lambda$23/1457226878.execute(Unknown Source)
> 2020-03-24T15:52:18.9213408Z at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
> 2020-03-24T15:52:18.9213866Z at
>
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18681) The jar package version conflict causes the task to continue to increase and grab resources
[
https://issues.apache.org/jira/browse/FLINK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170491#comment-17170491
]
Tao Yang commented on FLINK-18681:
--
Hi, [~xintongsong], [~apach...@163.com], sorry for responding so late.
I'm not sure what happened in RM, I think there should be something wrong in RM
if the scenario is actually as you said. Can you offer some details by which at
least we can trace all the lifecycle of one case?
> The jar package version conflict causes the task to continue to increase and
> grab resources
> ---
>
> Key: FLINK-18681
> URL: https://issues.apache.org/jira/browse/FLINK-18681
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: wangtaiyang
>Priority: Major
> Attachments: appId.log, dependency.log,
> image-2020-07-28-15-32-51-851.png,
> yarn-hadoop-resourcemanager-x.x.x.15.log.2020-07-22-17.log
>
>
> When I submit a flink task to yarn, the default resource configuration is
> 1G&1core, but in fact this task will always increase resources 2core, 3core,
> and so on. . . 200core. . . Then I went to look at the JM log and found the
> following error:
> {code:java}
> //代码占位符
> java.lang.NoSuchMethodError:
> org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;java.lang.NoSuchMethodError:
>
> org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
> at
> org.apache.flink.runtime.entrypoint.parser.CommandLineOptions.(CommandLineOptions.java:28)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1] at
> org.apache.flink.runtime.clusterframework.BootstrapTools.lambda$getDynamicPropertiesAsString$0(BootstrapTools.java:648)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1] at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
> ~[?:1.8.0_191]
> ...
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.flink.runtime.entrypoint.parser.CommandLineOptionsjava.lang.NoClassDefFoundError:
> Could not initialize class
> org.apache.flink.runtime.entrypoint.parser.CommandLineOptions at
> org.apache.flink.runtime.clusterframework.BootstrapTools.lambda$getDynamicPropertiesAsString$0(BootstrapTools.java:648)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1] at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
> ~[?:1.8.0_191] at
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
> ~[?:1.8.0_191] at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> ~[?:1.8.0_191]{code}
> Finally, it is confirmed that it is caused by the commands-cli version
> conflict, but the task reporting error has not stopped and will continue to
> grab resources and increase. Is this a bug?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
flinkbot edited a comment on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-667904442 ## CI report: * 1a2f39a71a931295f82a9bdbd36f2707f42298dd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5115) * e274e166aed3ffacc4ac58409aaf3384c937cb66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18807) FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout expired after 60000milliseconds while awaiting EndTxn(COMMIT)"
[ https://issues.apache.org/jira/browse/FLINK-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18807: Labels: test-stability (was: ) > FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout > expired after 6milliseconds while awaiting EndTxn(COMMIT)" > > > Key: FLINK-18807 > URL: https://issues.apache.org/jira/browse/FLINK-18807 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Tests >Affects Versions: 1.12.0 >Reporter: Dian Fu >Priority: Major > Labels: test-stability > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12929: [FLINK-18631] [serialization] Serializer for scala sealed trait hierarchies
flinkbot edited a comment on pull request #12929: URL: https://github.com/apache/flink/pull/12929#issuecomment-660169673 ## CI report: * feb4cadb5b16140e0477944d4ecf99cc208a4d38 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5145) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4737) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18807) FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout expired after 60000milliseconds while awaiting EndTxn(COMMIT)"
[
https://issues.apache.org/jira/browse/FLINK-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18807:
Description:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5142=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
{code}
2020-08-03T22:06:45.9078498Z [ERROR]
testScaleUpAfterScalingDown(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
Time elapsed: 76.498 s <<< ERROR!
2020-08-03T22:06:45.9079233Z org.apache.kafka.common.errors.TimeoutException:
Timeout expired after 6milliseconds while awaiting EndTxn(COMMIT)
{code}
> FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout
> expired after 6milliseconds while awaiting EndTxn(COMMIT)"
>
>
> Key: FLINK-18807
> URL: https://issues.apache.org/jira/browse/FLINK-18807
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5142=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> 2020-08-03T22:06:45.9078498Z [ERROR]
> testScaleUpAfterScalingDown(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 76.498 s <<< ERROR!
> 2020-08-03T22:06:45.9079233Z org.apache.kafka.common.errors.TimeoutException:
> Timeout expired after 6milliseconds while awaiting EndTxn(COMMIT)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18807) FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout expired after 60000milliseconds while awaiting EndTxn(COMMIT)"
Dian Fu created FLINK-18807: --- Summary: FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout expired after 6milliseconds while awaiting EndTxn(COMMIT)" Key: FLINK-18807 URL: https://issues.apache.org/jira/browse/FLINK-18807 Project: Flink Issue Type: Bug Components: Connectors / Kafka Affects Versions: 1.12.0 Reporter: Dian Fu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18807) FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout expired after 60000milliseconds while awaiting EndTxn(COMMIT)"
[ https://issues.apache.org/jira/browse/FLINK-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18807: Component/s: Tests > FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed with "Timeout > expired after 6milliseconds while awaiting EndTxn(COMMIT)" > > > Key: FLINK-18807 > URL: https://issues.apache.org/jira/browse/FLINK-18807 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Tests >Affects Versions: 1.12.0 >Reporter: Dian Fu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] sunjincheng121 commented on pull request #364: Add blog post for Pandas support in PyFlink.
sunjincheng121 commented on pull request #364: URL: https://github.com/apache/flink-web/pull/364#issuecomment-668296976 Thanks for the detail review @morsapaes @dianfu . Hi Dian, before merging we can have a little bit discussion about virtual env part, because，if `python` command can't launch the python 3 interpreter, we have many ways to solve it. We can leave space for users or have more detail info in the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
zentol commented on a change in pull request #12917:
URL: https://github.com/apache/flink/pull/12917#discussion_r464710835
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolTestBase.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import
org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.resourcemanager.ResourceManagerGateway;
+import
org.apache.flink.runtime.resourcemanager.utils.TestingResourceManagerGateway;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Test base for {@link SlotPool} related test cases.
+ */
+public abstract class SlotPoolTestBase extends TestLogger {
Review comment:
We very much want to avoid introducing inheritance into test classes;
there is rarely a need for it.
This instance here could be trivially implemented as a junit resource.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] sjwiesman opened a new pull request #128: [hotfix] fix python sdk example
sjwiesman opened a new pull request #128: URL: https://github.com/apache/flink-statefun/pull/128 The python delayed message example code was incorrect. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12929: [FLINK-18631] [serialization] Serializer for scala sealed trait hierarchies
flinkbot edited a comment on pull request #12929: URL: https://github.com/apache/flink/pull/12929#issuecomment-660169673 ## CI report: * feb4cadb5b16140e0477944d4ecf99cc208a4d38 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5145) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4737) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18796) FlinkKinesisProducer.backpressureLatch should be volatile
[ https://issues.apache.org/jira/browse/FLINK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170437#comment-17170437 ] Roman Khachatryan commented on FLINK-18796: --- I'm not sure whether it's actually a problem, maybe there is some implicit memory fence after assignment. But if it *is* a problem I'm also not sure how does it manifest itself, as it does not necessarily fail the sink; and sink is not exactly-once. Which pattern do you mean? > FlinkKinesisProducer.backpressureLatch should be volatile > - > > Key: FLINK-18796 > URL: https://issues.apache.org/jira/browse/FLINK-18796 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.11.1 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Minor > Fix For: 1.12.0 > > > (confirm first) > > cc: [~rmetzger] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12929: [FLINK-18631] [serialization] Serializer for scala sealed trait hierarchies
flinkbot edited a comment on pull request #12929: URL: https://github.com/apache/flink/pull/12929#issuecomment-660169673 ## CI report: * c2a61b6c21c4b47c83bde923101c01b6c169e30d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4716) * feb4cadb5b16140e0477944d4ecf99cc208a4d38 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4737) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5145) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shuttie commented on pull request #12929: [FLINK-18631] [serialization] Serializer for scala sealed trait hierarchies
shuttie commented on pull request #12929: URL: https://github.com/apache/flink/pull/12929#issuecomment-668238686 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18796) FlinkKinesisProducer.backpressureLatch should be volatile
[ https://issues.apache.org/jira/browse/FLINK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170419#comment-17170419 ] Chesnay Schepler commented on FLINK-18796: -- Why would this be a problem _now_? We've used this pattern for years. > FlinkKinesisProducer.backpressureLatch should be volatile > - > > Key: FLINK-18796 > URL: https://issues.apache.org/jira/browse/FLINK-18796 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.11.1 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Minor > Fix For: 1.12.0 > > > (confirm first) > > cc: [~rmetzger] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13045: [FLINK-18748][checkpointing] trigger unperiodic checkpoint immediately
flinkbot edited a comment on pull request #13045: URL: https://github.com/apache/flink/pull/13045#issuecomment-667703208 ## CI report: * bc1a115c8825526c96674b35effceabc7542d0c2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5136) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-5552) Introduce JVM-wide JMX server
[ https://issues.apache.org/jira/browse/FLINK-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5552: Summary: Introduce JVM-wide JMX server (was: Make the JMX port available through RESTful API) > Introduce JVM-wide JMX server > - > > Key: FLINK-5552 > URL: https://issues.apache.org/jira/browse/FLINK-5552 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Web Frontend >Reporter: david.wang >Assignee: Rong Rong >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently, JMXReporter will create a server for JMX viewer retrieving JMX > stat. The port can be configured through configuration options, but for large > cluster with many machines running many Flink instances and other processes, > we cant set a fixed port to JMX server, making it difficult to get the JMX > port. > This JIRA is to suggest adding an api at web frontend so that it is very easy > to get the JMX port for JM and TM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-5552) Introduce JVM-wide JMX server
[ https://issues.apache.org/jira/browse/FLINK-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5552. --- Fix Version/s: 1.12.0 Resolution: Fixed master: 2b699d6cb20823cd3a437b49bc96f015f4c1b72e > Introduce JVM-wide JMX server > - > > Key: FLINK-5552 > URL: https://issues.apache.org/jira/browse/FLINK-5552 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Web Frontend >Reporter: david.wang >Assignee: Rong Rong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, JMXReporter will create a server for JMX viewer retrieving JMX > stat. The port can be configured through configuration options, but for large > cluster with many machines running many Flink instances and other processes, > we cant set a fixed port to JMX server, making it difficult to get the JMX > port. > This JIRA is to suggest adding an api at web frontend so that it is very easy > to get the JMX port for JM and TM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #11586: [FLINK-5552][runtime] make JMXServer static per JVM
zentol merged pull request #11586: URL: https://github.com/apache/flink/pull/11586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-16566) Change the log level of the launching command and dynamic properties from DEBUG to INFO in Mesos integration
[ https://issues.apache.org/jira/browse/FLINK-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-16566. Fix Version/s: 1.12.0 Resolution: Fixed master: 8bd4c403d9c3605db73c1320e498215fb99dea1d > Change the log level of the launching command and dynamic properties from > DEBUG to INFO in Mesos integration > > > Key: FLINK-16566 > URL: https://issues.apache.org/jira/browse/FLINK-16566 > Project: Flink > Issue Type: Improvement > Components: Deployment / Mesos >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > As discussed in > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-1-10-container-memory-configuration-with-Mesos-td33594.html > . It would helpful for debugging to log the launching command and dynamic > properties at INFO level. Since such logs occur only when workers started, it > would not be massive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #13046: [FLINK-16566][mesos] Change the log level of the launching command and dynamic properties from DEBUG to INFO in Mesos
zentol merged pull request #13046: URL: https://github.com/apache/flink/pull/13046 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18663) RestServerEndpoint may prevent server shutdown
[
https://issues.apache.org/jira/browse/FLINK-18663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-18663:
-
Summary: RestServerEndpoint may prevent server shutdown (was: Fix Flink On
YARN AM not exit)
> RestServerEndpoint may prevent server shutdown
> --
>
> Key: FLINK-18663
> URL: https://issues.apache.org/jira/browse/FLINK-18663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.10.0, 1.10.1, 1.11.0
>Reporter: tartarus
>Assignee: tartarus
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.2, 1.12.0, 1.11.2
>
> Attachments: 110.png, 111.png,
> C49A7310-F932-451B-A203-6D17F3140C0D.png,
> e18e00dd6664485c2ff55284fe969474.png, jobmanager.log.noyarn.tar.gz
>
>
> AbstractHandler throw NPE cause by FlinkHttpObjectAggregator is null
> when rest throw exception, it will do this code
> {code:java}
> private CompletableFuture handleException(Throwable throwable,
> ChannelHandlerContext ctx, HttpRequest httpRequest) {
> FlinkHttpObjectAggregator flinkHttpObjectAggregator =
> ctx.pipeline().get(FlinkHttpObjectAggregator.class);
> int maxLength = flinkHttpObjectAggregator.maxContentLength() -
> OTHER_RESP_PAYLOAD_OVERHEAD;
> if (throwable instanceof RestHandlerException) {
> RestHandlerException rhe = (RestHandlerException) throwable;
> String stackTrace = ExceptionUtils.stringifyException(rhe);
> String truncatedStackTrace = Ascii.truncate(stackTrace,
> maxLength, "...");
> if (log.isDebugEnabled()) {
> log.error("Exception occurred in REST handler.", rhe);
> } else {
> log.error("Exception occurred in REST handler: {}",
> rhe.getMessage());
> }
> return HandlerUtils.sendErrorResponse(
> ctx,
> httpRequest,
> new ErrorResponseBody(truncatedStackTrace),
> rhe.getHttpResponseStatus(),
> responseHeaders);
> } else {
> log.error("Unhandled exception.", throwable);
> String stackTrace = String.format(" side:%n%s%nEnd of exception on server side>",
> ExceptionUtils.stringifyException(throwable));
> String truncatedStackTrace = Ascii.truncate(stackTrace,
> maxLength, "...");
> return HandlerUtils.sendErrorResponse(
> ctx,
> httpRequest,
> new ErrorResponseBody(Arrays.asList("Internal server
> error.", truncatedStackTrace)),
> HttpResponseStatus.INTERNAL_SERVER_ERROR,
> responseHeaders);
> }
> }
> {code}
> but flinkHttpObjectAggregator some case is null,so this will throw NPE,but
> this method called by AbstractHandler#respondAsLeader
> {code:java}
> requestProcessingFuture
> .whenComplete((Void ignored, Throwable throwable) -> {
> if (throwable != null) {
>
> handleException(ExceptionUtils.stripCompletionException(throwable), ctx,
> httpRequest)
> .whenComplete((Void ignored2, Throwable
> throwable2) -> finalizeRequestProcessing(finalUploadedFiles));
> } else {
> finalizeRequestProcessing(finalUploadedFiles);
> }
> });
> {code}
> the result is InFlightRequestTracker Cannot be cleared.
> so the CompletableFuture does‘t complete that handler's closeAsync returned
> !C49A7310-F932-451B-A203-6D17F3140C0D.png!
> !e18e00dd6664485c2ff55284fe969474.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18663) Fix Flink On YARN AM not exit
[
https://issues.apache.org/jira/browse/FLINK-18663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-18663.
Fix Version/s: 1.11.2
1.12.0
1.10.2
Resolution: Fixed
master:
ee4b27f96e8156c6a1c4b825484d238055cecf5f
7619db019181347a16fd37f0e4e04c420f75eee9
1.11:
4c2f7c44fc7085edfd3f910135fbf7f54d5eadf1
862ac0f6921a7cf0a1f7fabfd3cff11292817fb0
1.10:
bda138361e55437c84ea000fe38a6bce69cf7520
52a77e695ea9dea79f67f6b6a7281ceb1079f65e
> Fix Flink On YARN AM not exit
> -
>
> Key: FLINK-18663
> URL: https://issues.apache.org/jira/browse/FLINK-18663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.10.0, 1.10.1, 1.11.0
>Reporter: tartarus
>Assignee: tartarus
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.2, 1.12.0, 1.11.2
>
> Attachments: 110.png, 111.png,
> C49A7310-F932-451B-A203-6D17F3140C0D.png,
> e18e00dd6664485c2ff55284fe969474.png, jobmanager.log.noyarn.tar.gz
>
>
> AbstractHandler throw NPE cause by FlinkHttpObjectAggregator is null
> when rest throw exception, it will do this code
> {code:java}
> private CompletableFuture handleException(Throwable throwable,
> ChannelHandlerContext ctx, HttpRequest httpRequest) {
> FlinkHttpObjectAggregator flinkHttpObjectAggregator =
> ctx.pipeline().get(FlinkHttpObjectAggregator.class);
> int maxLength = flinkHttpObjectAggregator.maxContentLength() -
> OTHER_RESP_PAYLOAD_OVERHEAD;
> if (throwable instanceof RestHandlerException) {
> RestHandlerException rhe = (RestHandlerException) throwable;
> String stackTrace = ExceptionUtils.stringifyException(rhe);
> String truncatedStackTrace = Ascii.truncate(stackTrace,
> maxLength, "...");
> if (log.isDebugEnabled()) {
> log.error("Exception occurred in REST handler.", rhe);
> } else {
> log.error("Exception occurred in REST handler: {}",
> rhe.getMessage());
> }
> return HandlerUtils.sendErrorResponse(
> ctx,
> httpRequest,
> new ErrorResponseBody(truncatedStackTrace),
> rhe.getHttpResponseStatus(),
> responseHeaders);
> } else {
> log.error("Unhandled exception.", throwable);
> String stackTrace = String.format(" side:%n%s%nEnd of exception on server side>",
> ExceptionUtils.stringifyException(throwable));
> String truncatedStackTrace = Ascii.truncate(stackTrace,
> maxLength, "...");
> return HandlerUtils.sendErrorResponse(
> ctx,
> httpRequest,
> new ErrorResponseBody(Arrays.asList("Internal server
> error.", truncatedStackTrace)),
> HttpResponseStatus.INTERNAL_SERVER_ERROR,
> responseHeaders);
> }
> }
> {code}
> but flinkHttpObjectAggregator some case is null,so this will throw NPE,but
> this method called by AbstractHandler#respondAsLeader
> {code:java}
> requestProcessingFuture
> .whenComplete((Void ignored, Throwable throwable) -> {
> if (throwable != null) {
>
> handleException(ExceptionUtils.stripCompletionException(throwable), ctx,
> httpRequest)
> .whenComplete((Void ignored2, Throwable
> throwable2) -> finalizeRequestProcessing(finalUploadedFiles));
> } else {
> finalizeRequestProcessing(finalUploadedFiles);
> }
> });
> {code}
> the result is InFlightRequestTracker Cannot be cleared.
> so the CompletableFuture does‘t complete that handler's closeAsync returned
> !C49A7310-F932-451B-A203-6D17F3140C0D.png!
> !e18e00dd6664485c2ff55284fe969474.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #13020: [FLINK-18663][rest] Fix the exception occurred on AbstractHandler#handleException but not handled
zentol merged pull request #13020: URL: https://github.com/apache/flink/pull/13020 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-16619) Misleading SlotManagerImpl logging for slot reports of unknown task manager
[
https://issues.apache.org/jira/browse/FLINK-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-16619.
Fix Version/s: 1.11.2
1.12.0
1.10.2
Resolution: Fixed
master: 7d39a2c7cd23c990800671787916751e04720252
1.11: 79e33e619612a7e91778ab18c181630e9d6a0ff0
1.10: 7627f5414a5e06f1ae465df9021c37c04b7ee553
> Misleading SlotManagerImpl logging for slot reports of unknown task manager
> ---
>
> Key: FLINK-16619
> URL: https://issues.apache.org/jira/browse/FLINK-16619
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.2, 1.12.0, 1.11.2
>
>
> If the SlotManager receives a slot report from an unknown task manager it
> logs 2 messages:
> {code}
> public boolean reportSlotStatus(InstanceID instanceId, SlotReport slotReport)
> {
> [...]
> LOG.debug("Received slot report from instance {}: {}.", instanceId,
> slotReport);
> TaskManagerRegistration taskManagerRegistration =
> taskManagerRegistrations.get(instanceId);
> if (null != taskManagerRegistration) {
> [...]
> } else {
> LOG.debug("Received slot report for unknown task manager with
> instance id {}. Ignoring this report.", instanceId);
> [...]
> }
> }
> {code}
> This leads to misleading output since it appears like the slot manager
> received 2 separate slot reports, with the first being for a known instance,
> the latter for an unknown one. This cost some time as I couldn't figure out
> why the "latter" report was suddenly being rejected.
> I propose moving the first debug message into the non-null branch.
> [~trohrmann] WDYT?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #13049: [FLINK-16619][coordination] Log reception of slot reports only once
zentol merged pull request #13049: URL: https://github.com/apache/flink/pull/13049 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13018: [FLINK-18709][Coordination] Implement PhysicalSlotProvider
flinkbot edited a comment on pull request #13018: URL: https://github.com/apache/flink/pull/13018#issuecomment-665565029 ## CI report: * 82cf917b212932113f018bd05fc02c3f6379 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5135) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13055: FLINK-18677: [fix] Added handling of suspended or lost connections
flinkbot edited a comment on pull request #13055: URL: https://github.com/apache/flink/pull/13055#issuecomment-668083003 ## CI report: * ed6fb2c77645701eb585d2700c852657b1f66fc9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5133) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kbohinski commented on pull request #12962: [FLINK-18694] Add unaligned checkpoint config to web ui
kbohinski commented on pull request #12962: URL: https://github.com/apache/flink/pull/12962#issuecomment-668187486 Ah sorry, thought it had to do with azure command. Sorry & thx! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
flinkbot edited a comment on pull request #13009: URL: https://github.com/apache/flink/pull/13009#issuecomment-665109698 ## CI report: * d7133401edf323e5be894758398098e70cf70506 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5138) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12962: [FLINK-18694] Add unaligned checkpoint config to web ui
rmetzger commented on pull request #12962: URL: https://github.com/apache/flink/pull/12962#issuecomment-668167696 This is the error the CI system is reporting ``` 2020-07-23T23:08:09.6743463Z java.lang.AssertionError: The API was modified in a compatible way, but the snapshot was not updated. To update the snapshot, re-run this test with -Dgenerate-rest-snapshot being set. 2020-07-23T23:08:09.6743967Z at org.junit.Assert.fail(Assert.java:88) 2020-07-23T23:08:09.6744412Z at org.apache.flink.runtime.rest.compatibility.RestAPIStabilityTest.assertCompatible(RestAPIStabilityTest.java:130) 2020-07-23T23:08:09.6745021Z at org.apache.flink.runtime.rest.compatibility.RestAPIStabilityTest.testDispatcherRestAPIStability(RestAPIStabilityTest.java:89) 2020-07-23T23:08:09.6745500Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2020-07-23T23:08:09.6745859Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2020-07-23T23:08:09.6746312Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2020-07-23T23:08:09.6746694Z at java.lang.reflect.Method.invoke(Method.java:498) 2020-07-23T23:08:09.6747087Z at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) 2020-07-23T23:08:09.6747652Z at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) 2020-07-23T23:08:09.6748116Z at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) 2020-07-23T23:08:09.6748585Z at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) 2020-07-23T23:08:09.6749500Z at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) 2020-07-23T23:08:09.6750017Z at org.junit.rules.RunRules.evaluate(RunRules.java:20) 2020-07-23T23:08:09.6750377Z at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) 2020-07-23T23:08:09.6750806Z at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) 2020-07-23T23:08:09.6751258Z at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) 2020-07-23T23:08:09.6751684Z at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) 2020-07-23T23:08:09.6752056Z at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) 2020-07-23T23:08:09.6752541Z at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) 2020-07-23T23:08:09.6752928Z at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) 2020-07-23T23:08:09.6753423Z at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) 2020-07-23T23:08:09.6753812Z at org.junit.runners.ParentRunner.run(ParentRunner.java:363) 2020-07-23T23:08:09.6754164Z at org.junit.runners.Suite.runChild(Suite.java:128) 2020-07-23T23:08:09.6754501Z at org.junit.runners.Suite.runChild(Suite.java:27) 2020-07-23T23:08:09.6754840Z at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) 2020-07-23T23:08:09.6755235Z at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) 2020-07-23T23:08:09.6755622Z at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) 2020-07-23T23:08:09.6756196Z at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) 2020-07-23T23:08:09.6756577Z at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) 2020-07-23T23:08:09.6756960Z at org.junit.runners.ParentRunner.run(ParentRunner.java:363) 2020-07-23T23:08:09.6757547Z at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) 2020-07-23T23:08:09.6758228Z at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) 2020-07-23T23:08:09.6758721Z at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) 2020-07-23T23:08:09.6759199Z at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) 2020-07-23T23:08:09.6759684Z at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) 2020-07-23T23:08:09.6760199Z at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) 2020-07-23T23:08:09.6760655Z at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) 2020-07-23T23:08:09.6761096Z at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) ``` I guess you need to follow the instructions. The command will create some changes locally that you have to commit as part of this PR.
[GitHub] [flink] dawidwys commented on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
dawidwys commented on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-668164712 Are we planning to bump Calcite version for 1.12? @wuchong @twalthr If we do, I'd prefer we do not add this workaround. If we don't the PR looks good to me This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
flinkbot edited a comment on pull request #13009: URL: https://github.com/apache/flink/pull/13009#issuecomment-665109698 ## CI report: * 044578bc01da8b57e29a483ee2560a0e8f61148a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4971) * d7133401edf323e5be894758398098e70cf70506 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5138) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #13018: [FLINK-18709][Coordination] Implement PhysicalSlotProvider
zhuzhurk commented on a change in pull request #13018:
URL: https://github.com/apache/flink/pull/13018#discussion_r464561183
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotProviderImplTest.java
##
@@ -78,12 +76,11 @@ public void teardown() {
}
@Test
- public void testBulkSlotAllocationFulfilledWithAvailableSlots()
- throws InterruptedException,
java.util.concurrent.ExecutionException, java.util.concurrent.TimeoutException {
+ public void testBulkSlotAllocationFulfilledWithAvailableSlots() throws
InterruptedException, ExecutionException {
PhysicalSlotRequest request = createPhysicalSlotRequest();
addSlotToSlotPool();
CompletableFuture slotFuture =
allocateSlot(request);
- PhysicalSlotRequest.Result result =
slotFuture.get(TIMEOUT.getSize(), TIMEOUT.getUnit());
+ PhysicalSlotRequest.Result result = slotFuture.get();
Review comment:
What I had thought is to use `slotFuture.getNow(null)` and then doing a
non-null check for the returned result.
Then we can say that allocation can be fulfilled immediately without delay
if there are available slots.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] kbohinski commented on pull request #12962: [FLINK-18694] Add unaligned checkpoint config to web ui
kbohinski commented on pull request #12962: URL: https://github.com/apache/flink/pull/12962#issuecomment-668147556 Hey @rmetzger not sure what to do with build output. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13020: [FLINK-18663][rest] Fix the exception occurred on AbstractHandler#handleException but not handled
flinkbot edited a comment on pull request #13020: URL: https://github.com/apache/flink/pull/13020#issuecomment-665606861 ## CI report: * 0d374f52f0c0c40585dc9bad5962aa52cb602b21 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5125) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
flinkbot edited a comment on pull request #13009: URL: https://github.com/apache/flink/pull/13009#issuecomment-665109698 ## CI report: * 044578bc01da8b57e29a483ee2560a0e8f61148a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4971) * d7133401edf323e5be894758398098e70cf70506 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
zhuzhurk commented on a change in pull request #13009:
URL: https://github.com/apache/flink/pull/13009#discussion_r464556912
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/LocalInputPreferredSlotSharingStrategy.java
##
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraintDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroupDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.SlotSharingGroup;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingExecutionVertex;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingTopology;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This strategy tries to reduce remote data exchanges. Execution vertices,
which are connected
+ * and belong to the same SlotSharingGroup, tend to be put in the same
ExecutionSlotSharingGroup.
+ * Co-location constraints will be respected.
+ */
+class LocalInputPreferredSlotSharingStrategy implements SlotSharingStrategy {
+
+ private final Map
executionSlotSharingGroupMap;
+
+ LocalInputPreferredSlotSharingStrategy(
+ final SchedulingTopology topology,
+ final Set logicalSlotSharingGroups,
+ final Set coLocationGroups) {
+
+ this.executionSlotSharingGroupMap = new
ExecutionSlotSharingGroupBuilder(
+ topology,
+ logicalSlotSharingGroups,
+ coLocationGroups).build();
+ }
+
+ @Override
+ public ExecutionSlotSharingGroup getExecutionSlotSharingGroup(final
ExecutionVertexID executionVertexId) {
+ return executionSlotSharingGroupMap.get(executionVertexId);
+ }
+
+ @Override
+ public Set getExecutionSlotSharingGroups() {
+ return new HashSet<>(executionSlotSharingGroupMap.values());
+ }
+
+ static class Factory implements SlotSharingStrategy.Factory {
+
+ public LocalInputPreferredSlotSharingStrategy create(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ return new
LocalInputPreferredSlotSharingStrategy(topology, logicalSlotSharingGroups,
coLocationGroups);
+ }
+ }
+
+ private static class ExecutionSlotSharingGroupBuilder {
+ private final SchedulingTopology topology;
+
+ private final Map
slotSharingGroupMap;
+
+ private final Map
coLocationGroupMap;
+
+ private final Map
executionSlotSharingGroupMap;
+
+ final Map
constraintToExecutionSlotSharingGroupMap;
+
+ final Map>
executionSlotSharingGroups;
+
+ private final Map>
assignedJobVerticesForGroups;
+
+ private ExecutionSlotSharingGroupBuilder(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ this.topology = checkNotNull(topology);
+
+ this.slotSharingGroupMap = new HashMap<>();
+ for (SlotSharingGroup slotSharingGroup :
logicalSlotSharingGroups) {
+ for (JobVertexID jobVertexId :
slotSharingGroup.getJobVertexIds()) {
+
[GitHub] [flink] zhuzhurk commented on a change in pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
zhuzhurk commented on a change in pull request #13009:
URL: https://github.com/apache/flink/pull/13009#discussion_r464556766
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/LocalInputPreferredSlotSharingStrategy.java
##
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraintDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroupDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.SlotSharingGroup;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingExecutionVertex;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingTopology;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This strategy tries to reduce remote data exchanges. Execution vertices,
which are connected
+ * and belong to the same SlotSharingGroup, tend to be put in the same
ExecutionSlotSharingGroup.
+ * Co-location constraints will be respected.
+ */
+class LocalInputPreferredSlotSharingStrategy implements SlotSharingStrategy {
+
+ private final Map
executionSlotSharingGroupMap;
+
+ LocalInputPreferredSlotSharingStrategy(
+ final SchedulingTopology topology,
+ final Set logicalSlotSharingGroups,
+ final Set coLocationGroups) {
+
+ this.executionSlotSharingGroupMap = new
ExecutionSlotSharingGroupBuilder(
+ topology,
+ logicalSlotSharingGroups,
+ coLocationGroups).build();
+ }
+
+ @Override
+ public ExecutionSlotSharingGroup getExecutionSlotSharingGroup(final
ExecutionVertexID executionVertexId) {
+ return executionSlotSharingGroupMap.get(executionVertexId);
+ }
+
+ @Override
+ public Set getExecutionSlotSharingGroups() {
+ return new HashSet<>(executionSlotSharingGroupMap.values());
+ }
+
+ static class Factory implements SlotSharingStrategy.Factory {
+
+ public LocalInputPreferredSlotSharingStrategy create(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ return new
LocalInputPreferredSlotSharingStrategy(topology, logicalSlotSharingGroups,
coLocationGroups);
+ }
+ }
+
+ private static class ExecutionSlotSharingGroupBuilder {
+ private final SchedulingTopology topology;
+
+ private final Map
slotSharingGroupMap;
+
+ private final Map
coLocationGroupMap;
+
+ private final Map
executionSlotSharingGroupMap;
+
+ final Map
constraintToExecutionSlotSharingGroupMap;
+
+ final Map>
executionSlotSharingGroups;
+
+ private final Map>
assignedJobVerticesForGroups;
+
+ private ExecutionSlotSharingGroupBuilder(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ this.topology = checkNotNull(topology);
+
+ this.slotSharingGroupMap = new HashMap<>();
+ for (SlotSharingGroup slotSharingGroup :
logicalSlotSharingGroups) {
+ for (JobVertexID jobVertexId :
slotSharingGroup.getJobVertexIds()) {
+
[GitHub] [flink] zhuzhurk commented on a change in pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
zhuzhurk commented on a change in pull request #13009:
URL: https://github.com/apache/flink/pull/13009#discussion_r464556118
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/LocalInputPreferredSlotSharingStrategy.java
##
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraintDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroupDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.SlotSharingGroup;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingExecutionVertex;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingTopology;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This strategy tries to reduce remote data exchanges. Execution vertices,
which are connected
+ * and belong to the same SlotSharingGroup, tend to be put in the same
ExecutionSlotSharingGroup.
+ * Co-location constraints will be respected.
+ */
+class LocalInputPreferredSlotSharingStrategy implements SlotSharingStrategy {
+
+ private final Map
executionSlotSharingGroupMap;
+
+ LocalInputPreferredSlotSharingStrategy(
+ final SchedulingTopology topology,
+ final Set logicalSlotSharingGroups,
+ final Set coLocationGroups) {
+
+ this.executionSlotSharingGroupMap = new
ExecutionSlotSharingGroupBuilder(
+ topology,
+ logicalSlotSharingGroups,
+ coLocationGroups).build();
+ }
+
+ @Override
+ public ExecutionSlotSharingGroup getExecutionSlotSharingGroup(final
ExecutionVertexID executionVertexId) {
+ return executionSlotSharingGroupMap.get(executionVertexId);
+ }
+
+ @Override
+ public Set getExecutionSlotSharingGroups() {
+ return new HashSet<>(executionSlotSharingGroupMap.values());
+ }
+
+ static class Factory implements SlotSharingStrategy.Factory {
+
+ public LocalInputPreferredSlotSharingStrategy create(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ return new
LocalInputPreferredSlotSharingStrategy(topology, logicalSlotSharingGroups,
coLocationGroups);
+ }
+ }
+
+ private static class ExecutionSlotSharingGroupBuilder {
+ private final SchedulingTopology topology;
+
+ private final Map
slotSharingGroupMap;
+
+ private final Map
coLocationGroupMap;
+
+ private final Map
executionSlotSharingGroupMap;
+
+ final Map
constraintToExecutionSlotSharingGroupMap;
+
+ final Map>
executionSlotSharingGroups;
+
+ private final Map>
assignedJobVerticesForGroups;
+
+ private ExecutionSlotSharingGroupBuilder(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ this.topology = checkNotNull(topology);
+
+ this.slotSharingGroupMap = new HashMap<>();
+ for (SlotSharingGroup slotSharingGroup :
logicalSlotSharingGroups) {
+ for (JobVertexID jobVertexId :
slotSharingGroup.getJobVertexIds()) {
+
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * ae4c6182cbbf1441b85df2d82dfd36c3daf7db33 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5117) * da90885d695cf70c147481bbaa8c8be005a13a4c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #13009: [FLINK-18690][runtime] Implement LocalInputPreferredSlotSharingStrategy
zhuzhurk commented on a change in pull request #13009:
URL: https://github.com/apache/flink/pull/13009#discussion_r464542910
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/LocalInputPreferredSlotSharingStrategy.java
##
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraintDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroupDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.SlotSharingGroup;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingExecutionVertex;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingTopology;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This strategy tries to reduce remote data exchanges. Execution vertices,
which are connected
+ * and belong to the same SlotSharingGroup, tend to be put in the same
ExecutionSlotSharingGroup.
+ * Co-location constraints will be respected.
+ */
+class LocalInputPreferredSlotSharingStrategy implements SlotSharingStrategy {
+
+ private final Map
executionSlotSharingGroupMap;
+
+ LocalInputPreferredSlotSharingStrategy(
+ final SchedulingTopology topology,
+ final Set logicalSlotSharingGroups,
+ final Set coLocationGroups) {
+
+ this.executionSlotSharingGroupMap = new
ExecutionSlotSharingGroupBuilder(
+ topology,
+ logicalSlotSharingGroups,
+ coLocationGroups).build();
+ }
+
+ @Override
+ public ExecutionSlotSharingGroup getExecutionSlotSharingGroup(final
ExecutionVertexID executionVertexId) {
+ return executionSlotSharingGroupMap.get(executionVertexId);
+ }
+
+ @Override
+ public Set getExecutionSlotSharingGroups() {
+ return new HashSet<>(executionSlotSharingGroupMap.values());
+ }
+
+ static class Factory implements SlotSharingStrategy.Factory {
+
+ public LocalInputPreferredSlotSharingStrategy create(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ return new
LocalInputPreferredSlotSharingStrategy(topology, logicalSlotSharingGroups,
coLocationGroups);
+ }
+ }
+
+ private static class ExecutionSlotSharingGroupBuilder {
+ private final SchedulingTopology topology;
+
+ private final Map
slotSharingGroupMap;
+
+ private final Map
coLocationGroupMap;
+
+ private final Map
executionSlotSharingGroupMap;
+
+ final Map
constraintToExecutionSlotSharingGroupMap;
+
+ final Map>
executionSlotSharingGroups;
+
+ private final Map>
assignedJobVerticesForGroups;
+
+ private ExecutionSlotSharingGroupBuilder(
+ final SchedulingTopology topology,
+ final Set
logicalSlotSharingGroups,
+ final Set
coLocationGroups) {
+
+ this.topology = checkNotNull(topology);
+
+ this.slotSharingGroupMap = new HashMap<>();
+ for (SlotSharingGroup slotSharingGroup :
logicalSlotSharingGroups) {
+ for (JobVertexID jobVertexId :
slotSharingGroup.getJobVertexIds()) {
+
