date:20210427



flinkbot edited a comment on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828062007


   
   ## CI report:
   
   * 57608d10dc311491f674c9e0c1f350d127b49c4c UNKNOWN
   * 1c2dd4a9e8e93256e53daec1825a8c30a199e0e7 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17326)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15788: [FLINK_22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error



flinkbot edited a comment on pull request #15788:
URL: https://github.com/apache/flink/pull/15788#issuecomment-827674618


   
   ## CI report:
   
   * a7918f2545747f468780b682c54e9c909c298f21 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17309)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17310)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17330)
 
   * 4528190b411a0c3b2eddba01603514258236c60a Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17331)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15753: [FLINK-22428][docs][table] Translate timezone page into Chinese



flinkbot edited a comment on pull request #15753:
URL: https://github.com/apache/flink/pull/15753#issuecomment-826339948


   
   ## CI report:
   
   * ef16adf7941acfdf56818b000c9e7b2763138564 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17325)
 
   * 6e0aa31580be24371fa804c673670daeb6c3301d UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] hackergin commented on pull request #15785: [hotfix][doc] Fix invalid sql comments and words in timezone.md



hackergin commented on pull request #15785:
URL: https://github.com/apache/flink/pull/15785#issuecomment-828149161


   move changes to  https://github.com/apache/flink/pull/15753  and close this 
pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] hackergin closed pull request #15785: [hotfix][doc] Fix invalid sql comments and words in timezone.md



hackergin closed pull request #15785:
URL: https://github.com/apache/flink/pull/15785


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15788: [FLINK_22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error



flinkbot edited a comment on pull request #15788:
URL: https://github.com/apache/flink/pull/15788#issuecomment-827674618


   
   ## CI report:
   
   * a7918f2545747f468780b682c54e9c909c298f21 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17309)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17310)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17330)
 
   * 4528190b411a0c3b2eddba01603514258236c60a UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15789: [WIP][FLINK-21181][runtime] Wait for Invokable cancellation before releasing network resources



flinkbot edited a comment on pull request #15789:
URL: https://github.com/apache/flink/pull/15789#issuecomment-827996694


   
   ## CI report:
   
   * a0ae769663e7b237021fbad03ebc309928dd2445 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17317)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] todd5167 commented on a change in pull request #15755: [FLINK-22318][table] Support RENAME column name for ALTER TABLE state…



todd5167 commented on a change in pull request #15755:
URL: https://github.com/apache/flink/pull/15755#discussion_r621814045



##
File path: 
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/OperationConverterUtils.java
##
@@ -144,6 +149,116 @@ public static Operation convertChangeColumn(
 // TODO: handle watermark and constraints
 }
 
+public static Operation convertRenameColumn(
+ObjectIdentifier tableIdentifier,
+String originColumnName,
+String newColumnName,
+CatalogTable catalogTable) {
+
+Schema modifiedTableSchema = catalogTable.getUnresolvedSchema();
+validateColumnName(originColumnName, newColumnName, 
modifiedTableSchema);
+
+Schema.Builder builder = Schema.newBuilder();
+// build column
+modifiedTableSchema.getColumns().stream()
+.forEach(
+column -> {
+if (StringUtils.equals(column.getName(), 
originColumnName)) {
+buildNewColumnFromOriginColumn(builder, 
column, newColumnName);
+} else {
+buildNewColumnFromOriginColumn(builder, 
column, column.getName());
+}
+});
+// build primary key column
+List originPrimaryKeyNames =
+modifiedTableSchema
+.getPrimaryKey()
+.map(Schema.UnresolvedPrimaryKey::getColumnNames)
+.orElseGet(Lists::newArrayList);
+
+List newPrimaryKeyNames =
+originPrimaryKeyNames.stream()
+.map(
+pkName ->
+StringUtils.equals(pkName, 
originColumnName)
+? newColumnName
+: pkName)
+.collect(Collectors.toList());
+
+if (newPrimaryKeyNames.size() > 0) {
+builder.primaryKey(newPrimaryKeyNames);
+}
+// build watermark
+modifiedTableSchema.getWatermarkSpecs().stream()
+.forEach(
+watermarkSpec -> {
+String watermarkRefColumnName = 
watermarkSpec.getColumnName();
+Expression watermarkExpression = 
watermarkSpec.getWatermarkExpression();
+if (StringUtils.equals(watermarkRefColumnName, 
originColumnName)) {
+String newWatermarkExpression =
+((SqlCallExpression) 
watermarkExpression)
+.getSqlExpression()
+
.replace(watermarkRefColumnName, newColumnName);

Review comment:
   Hi  @wuchong ,
   traversing the resolved expression tree to see whether it contains renamed 
column is a better solution. the parsing process may be more complicated.
   
   In my JIRA solution, SqlValidatorException can only throw Unknown 
identifier'xxx' because the resolved expression has been validated when the 
table is created.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15788: [FLINK_22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error



flinkbot edited a comment on pull request #15788:
URL: https://github.com/apache/flink/pull/15788#issuecomment-827674618


   
   ## CI report:
   
   * a7918f2545747f468780b682c54e9c909c298f21 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17309)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17310)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17330)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15792: [hotfix][cep] Add some missing parameterization in cep module



flinkbot edited a comment on pull request #15792:
URL: https://github.com/apache/flink/pull/15792#issuecomment-828132718


   
   ## CI report:
   
   * d3defe13239c7d38525a45fd1958786f37c52314 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17329)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15760: [FLINK-19606][table-runtime-blink] Introduce WindowJoinOperator and WindowJoinOperatorBuilder.



flinkbot edited a comment on pull request #15760:
URL: https://github.com/apache/flink/pull/15760#issuecomment-826500326


   
   ## CI report:
   
   * 1d43ce7ad5aa1b1a53be0e304b1d5b9ed477c3b7 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17194)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17197)
 
   * 7812ac58b48303fd5c157ddb4d24e2226766c920 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17328)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] todd5167 commented on a change in pull request #15755: [FLINK-22318][table] Support RENAME column name for ALTER TABLE state…



todd5167 commented on a change in pull request #15755:
URL: https://github.com/apache/flink/pull/15755#discussion_r621808605



##
File path: 
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/OperationConverterUtils.java
##
@@ -144,6 +149,116 @@ public static Operation convertChangeColumn(
 // TODO: handle watermark and constraints
 }
 
+public static Operation convertRenameColumn(
+ObjectIdentifier tableIdentifier,
+String originColumnName,
+String newColumnName,
+CatalogTable catalogTable) {
+
+Schema modifiedTableSchema = catalogTable.getUnresolvedSchema();
+validateColumnName(originColumnName, newColumnName, 
modifiedTableSchema);
+
+Schema.Builder builder = Schema.newBuilder();
+// build column
+modifiedTableSchema.getColumns().stream()
+.forEach(
+column -> {
+if (StringUtils.equals(column.getName(), 
originColumnName)) {
+buildNewColumnFromOriginColumn(builder, 
column, newColumnName);
+} else {
+buildNewColumnFromOriginColumn(builder, 
column, column.getName());
+}
+});
+// build primary key column
+List originPrimaryKeyNames =
+modifiedTableSchema
+.getPrimaryKey()
+.map(Schema.UnresolvedPrimaryKey::getColumnNames)
+.orElseGet(Lists::newArrayList);
+
+List newPrimaryKeyNames =
+originPrimaryKeyNames.stream()
+.map(
+pkName ->
+StringUtils.equals(pkName, 
originColumnName)
+? newColumnName
+: pkName)
+.collect(Collectors.toList());
+
+if (newPrimaryKeyNames.size() > 0) {
+builder.primaryKey(newPrimaryKeyNames);
+}
+// build watermark
+modifiedTableSchema.getWatermarkSpecs().stream()
+.forEach(
+watermarkSpec -> {
+String watermarkRefColumnName = 
watermarkSpec.getColumnName();
+Expression watermarkExpression = 
watermarkSpec.getWatermarkExpression();
+if (StringUtils.equals(watermarkRefColumnName, 
originColumnName)) {
+String newWatermarkExpression =
+((SqlCallExpression) 
watermarkExpression)
+.getSqlExpression()
+
.replace(watermarkRefColumnName, newColumnName);

Review comment:
   @wuchong  
   Using org.apache.flink.table.catalog.DefaultSchemaResolver#resolve to 
resolve a Schema is  a good solution.
   In my JIRA solution, SqlValidatorException can only throw Unknown 
identifier'xxx' because the expression has been validated when the table is 
created.   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] githubMJ commented on pull request #15788: [FLINK_22408]Flink Table Parsr Hive Drop Partitions Syntax unparse is Error



githubMJ commented on pull request #15788:
URL: https://github.com/apache/flink/pull/15788#issuecomment-828138782


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #15792: [hotfix][cep] Add some missing parameterization in cep module



flinkbot commented on pull request #15792:
URL: https://github.com/apache/flink/pull/15792#issuecomment-828132718


   
   ## CI report:
   
   * d3defe13239c7d38525a45fd1958786f37c52314 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15791: [FLINK-22438][metrics] Add numRecordsOut metric for Async IO



flinkbot edited a comment on pull request #15791:
URL: https://github.com/apache/flink/pull/15791#issuecomment-828125920


   
   ## CI report:
   
   * 7a0f4e1a3c40f295b100a94c4772ba238929f4b0 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17327)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15790: [hotfix][python][docs] add bundling udfs section



flinkbot edited a comment on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828062007


   
   ## CI report:
   
   * 57608d10dc311491f674c9e0c1f350d127b49c4c UNKNOWN
   * 86c54e97d3657b5e572988c8987979f39983590d Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17322)
 
   * 1c2dd4a9e8e93256e53daec1825a8c30a199e0e7 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17326)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15760: [FLINK-19606][table-runtime-blink] Introduce WindowJoinOperator and WindowJoinOperatorBuilder.



flinkbot edited a comment on pull request #15760:
URL: https://github.com/apache/flink/pull/15760#issuecomment-826500326


   
   ## CI report:
   
   * 1d43ce7ad5aa1b1a53be0e304b1d5b9ed477c3b7 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17194)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17197)
 
   * 7812ac58b48303fd5c157ddb4d24e2226766c920 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15753: [FLINK-22428][docs][table] Translate timezone page into Chinese



flinkbot edited a comment on pull request #15753:
URL: https://github.com/apache/flink/pull/15753#issuecomment-826339948


   
   ## CI report:
   
   * 254a78439e2e3128167c760cab0dcdfb363a91cd Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17304)
 
   * ef16adf7941acfdf56818b000c9e7b2763138564 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17325)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-22406) Unstable test ReactiveModeITCase.testScaleDownOnTaskManagerLoss()



[ 
https://issues.apache.org/jira/browse/FLINK-22406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334439#comment-17334439
 ] 

Guowei Ma commented on FLINK-22406:
---

another case
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17316=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=5360d54c-8d94-5d85-304e-a89267eb785a=10889

> Unstable test ReactiveModeITCase.testScaleDownOnTaskManagerLoss()
> -
>
> Key: FLINK-22406
> URL: https://issues.apache.org/jira/browse/FLINK-22406
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.13.0
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
>Priority: Critical
>  Labels: test-stability
>
> The test is stalling on Azure CI.
> https://dev.azure.com/sewen0794/Flink/_build/results?buildId=292=logs=0a15d512-44ac-5ba5-97ab-13a5d066c22c=634cd701-c189-5dff-24cb-606ed884db87=4865



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #15792: [hotfix][cep] Add some missing parameterization in cep module



flinkbot commented on pull request #15792:
URL: https://github.com/apache/flink/pull/15792#issuecomment-828127793


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit d3defe13239c7d38525a45fd1958786f37c52314 (Wed Apr 28 
04:08:26 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] dengziming opened a new pull request #15792: [hotfix][cep] Add some missing parameterization in cep module



dengziming opened a new pull request #15792:
URL: https://github.com/apache/flink/pull/15792


   
   
   ## What is the purpose of the change
   
   *(For example: This pull request makes task deployment go through the blob 
server, rather than through RPC. That way we avoid re-transferring them on each 
deployment (during recovery).)*
   
   
   ## Brief change log
   
   *(for example:)*
 - *The TaskInfo is stored in the blob store on job creation time as a 
persistent artifact*
 - *Deployments RPC transmits only the blob storage reference*
 - *TaskManagers retrieve the TaskInfo from the blob cache*
   
   
   ## Verifying this change
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(100MB)*
 - *Extended integration test for recovery after master (JobManager) 
failure*
 - *Added test that validates that TaskInfo is transferred only once across 
recoveries*
 - *Manually verified the change by running a 4 node cluser with 2 
JobManagers and 4 TaskManagers, a stateful streaming program, and killing one 
JobManager and two TaskManagers during the execution, verifying that recovery 
happens correctly.*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / no)
 - The serializers: (yes / no / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / no / 
don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't 
know)
 - The S3 file system connector: (yes / no / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / no)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-22498) cast the primary key for source table that has a decimal primary key as string, and then insert into a kudu table that has a string primary key throw the exception : U

2021-04-27 Thread Jark Wu (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334438#comment-17334438
 ] 

Jark Wu commented on FLINK-22498:
-

This is a problem on Kudu sink, the Kudu sink implementation should upgrade to 
new {{DynamicTableSink}}. 

> cast the primary key for source table that has a decimal primary key as 
> string, and then insert into a kudu table that has a string primary key throw 
> the exception : UpsertStreamTableSink requires that Table has a full primary 
> keys if it is updated
> 
>
> Key: FLINK-22498
> URL: https://issues.apache.org/jira/browse/FLINK-22498
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Affects Versions: 1.12.1
> Environment: flink 1.12.1
> jdk 1.8
> hive 2.1.1
> kudu 1.10.0
> kafka 2.0.0
>Reporter: Carl
>Priority: Critical
> Attachments: bug.rar
>
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> *1. source table：*
> CREATE TABLE ddl_source (
>  appl_seq DECIMAL(16,2),
>  name STRING,
>  PRIMARY KEY(appl_seq) NOT ENFORCED
> ) WITH (
>  'connector' = 'kafka',
>  'topic' = 'ogg-json-03',
>  'properties.bootstrap.servers' = ':9092',
>  'value.format' = 'canal-json'
> )
> *2. sink table：*create the table use impala
> create table rt_dwd.test_bug( 
>     pk       string  ,
>     name      string  ,
>     primary key (pk)
> ) partition by hash (pk) partitions 5 stored as kudu 
> TBLPROPERTIES  ('kudu.master_addresses' = ':7051');
> *3. execute sql：*use blink planner
> insert into kuducatalog.default_database.`rt_dwd.test_bug`
> select CAST(appl_seq AS STRING), name  from ddl_source
>  
> *throw an exception :*
> Exception in thread "main" org.apache.flink.table.api.TableException: 
> UpsertStreamTableSink requires that Table has a full primary keys if it is 
> updated.Exception in thread "main" org.apache.flink.table.api.TableException: 
> UpsertStreamTableSink requires that Table has a full primary keys if it is 
> updated. at 
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlanInternal(StreamExecLegacySink.scala:93)
>  at 
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlanInternal(StreamExecLegacySink.scala:48)
>  at 
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:59)
>  at 
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlan(StreamExecLegacySink.scala:48)
>  at 
> org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:66)
>  at 
> org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:65)
>  at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>  at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:891) at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at 
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at 
> scala.collection.AbstractTraversable.map(Traversable.scala:104) at 
> org.apache.flink.table.planner.delegation.StreamPlanner.translateToPlan(StreamPlanner.scala:65)
>  at 
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:167)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1329)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:676)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:767)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666)
>  
> *case A：if we use source table as follows, it will not throw the exception :*
> CREATE TABLE ddl_source (
>  appl_seq STRING,
>  name STRING,
>  PRIMARY KEY(appl_seq) NOT ENFORCED
> ) WITH (
>  'connector' = 'kafka',
>  'topic' = 'ogg-json-03',
>  'properties.bootstrap.servers' = ':9092',
>  'value.format' = 'canal-json'
> )
>  
> *case B：or we ddl kudu table,and use sql as follows,  it will not throw the 
> exception :*
> _DDL：_
> create table rt_dwd.test_bug( 
>     pk       decimal(16,2),
>     name      string  ,
>     primary key (pk)
> ) partition by hash (pk) partitions 5 stored as kudu

[jira] [Closed] (FLINK-22498) cast the primary key for source table that has a decimal primary key as string, and then insert into a kudu table that has a string primary key throw the exception : Upse

2021-04-27 Thread Jark Wu (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu closed FLINK-22498.
---
Resolution: Not A Problem

> cast the primary key for source table that has a decimal primary key as 
> string, and then insert into a kudu table that has a string primary key throw 
> the exception : UpsertStreamTableSink requires that Table has a full primary 
> keys if it is updated
> 
>
> Key: FLINK-22498
> URL: https://issues.apache.org/jira/browse/FLINK-22498
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Affects Versions: 1.12.1
> Environment: flink 1.12.1
> jdk 1.8
> hive 2.1.1
> kudu 1.10.0
> kafka 2.0.0
>Reporter: Carl
>Priority: Critical
> Attachments: bug.rar
>
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> *1. source table：*
> CREATE TABLE ddl_source (
>  appl_seq DECIMAL(16,2),
>  name STRING,
>  PRIMARY KEY(appl_seq) NOT ENFORCED
> ) WITH (
>  'connector' = 'kafka',
>  'topic' = 'ogg-json-03',
>  'properties.bootstrap.servers' = ':9092',
>  'value.format' = 'canal-json'
> )
> *2. sink table：*create the table use impala
> create table rt_dwd.test_bug( 
>     pk       string  ,
>     name      string  ,
>     primary key (pk)
> ) partition by hash (pk) partitions 5 stored as kudu 
> TBLPROPERTIES  ('kudu.master_addresses' = ':7051');
> *3. execute sql：*use blink planner
> insert into kuducatalog.default_database.`rt_dwd.test_bug`
> select CAST(appl_seq AS STRING), name  from ddl_source
>  
> *throw an exception :*
> Exception in thread "main" org.apache.flink.table.api.TableException: 
> UpsertStreamTableSink requires that Table has a full primary keys if it is 
> updated.Exception in thread "main" org.apache.flink.table.api.TableException: 
> UpsertStreamTableSink requires that Table has a full primary keys if it is 
> updated. at 
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlanInternal(StreamExecLegacySink.scala:93)
>  at 
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlanInternal(StreamExecLegacySink.scala:48)
>  at 
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:59)
>  at 
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlan(StreamExecLegacySink.scala:48)
>  at 
> org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:66)
>  at 
> org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:65)
>  at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>  at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:891) at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at 
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at 
> scala.collection.AbstractTraversable.map(Traversable.scala:104) at 
> org.apache.flink.table.planner.delegation.StreamPlanner.translateToPlan(StreamPlanner.scala:65)
>  at 
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:167)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1329)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:676)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:767)
>  at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666)
>  
> *case A：if we use source table as follows, it will not throw the exception :*
> CREATE TABLE ddl_source (
>  appl_seq STRING,
>  name STRING,
>  PRIMARY KEY(appl_seq) NOT ENFORCED
> ) WITH (
>  'connector' = 'kafka',
>  'topic' = 'ogg-json-03',
>  'properties.bootstrap.servers' = ':9092',
>  'value.format' = 'canal-json'
> )
>  
> *case B：or we ddl kudu table,and use sql as follows,  it will not throw the 
> exception :*
> _DDL：_
> create table rt_dwd.test_bug( 
>     pk       decimal(16,2),
>     name      string  ,
>     primary key (pk)
> ) partition by hash (pk) partitions 5 stored as kudu 
> TBLPROPERTIES  ('kudu.master_addresses' = ':7051');
> _DML：_
> insert into

[GitHub] [flink] flinkbot commented on pull request #15791: [FLINK-22438][metrics] Add numRecordsOut metric for Async IO



flinkbot commented on pull request #15791:
URL: https://github.com/apache/flink/pull/15791#issuecomment-828125920


   
   ## CI report:
   
   * 7a0f4e1a3c40f295b100a94c4772ba238929f4b0 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15753: [FLINK-22428][docs][table] Translate timezone page into Chinese



flinkbot edited a comment on pull request #15753:
URL: https://github.com/apache/flink/pull/15753#issuecomment-826339948


   
   ## CI report:
   
   * 254a78439e2e3128167c760cab0dcdfb363a91cd Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17304)
 
   * ef16adf7941acfdf56818b000c9e7b2763138564 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf



flinkbot edited a comment on pull request #14376:
URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765


   
   ## CI report:
   
   * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN
   * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN
   * e3f953d91778ef96ad5fe222dd9da229dc2c5619 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17323)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-22499) JDBC sink table-api support "sink.parallelism" ?

2021-04-27 Thread ranqiqiang (Jira)

ranqiqiang created FLINK-22499:
--

 Summary: JDBC  sink table-api  support "sink.parallelism" ?
 Key: FLINK-22499
 URL: https://issues.apache.org/jira/browse/FLINK-22499
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / JDBC
Affects Versions: 1.12.1
Reporter: ranqiqiang


I receive message from kafka, then sink to mysql/tidb across the sql table-api,

How can I change the sink parallelism ? 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #15790: [hotfix][python][docs] add bundling udfs section



flinkbot edited a comment on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828062007


   
   ## CI report:
   
   * 40ff67a835fa6304cf3fe72e642e4ac6a3af76a8 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17319)
 
   * 57608d10dc311491f674c9e0c1f350d127b49c4c UNKNOWN
   * 86c54e97d3657b5e572988c8987979f39983590d Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17322)
 
   * 1c2dd4a9e8e93256e53daec1825a8c30a199e0e7 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #15791: [FLINK-22438][metrics] Add numRecordsOut metric for Async IO



flinkbot commented on pull request #15791:
URL: https://github.com/apache/flink/pull/15791#issuecomment-828119027


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 7a0f4e1a3c40f295b100a94c4772ba238929f4b0 (Wed Apr 28 
03:42:41 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15121: The method $(String) is undefined for the type TableExample



flinkbot edited a comment on pull request #15121:
URL: https://github.com/apache/flink/pull/15121#issuecomment-793480240


   
   ## CI report:
   
   * cf26ce895a7956d258acc073817f578558e78227 UNKNOWN
   * 709c4110370b845f42d916a06e56c6026cf2fac8 UNKNOWN
   * 4ea99332e1997eebca1f3f0a9d9229b8265fe32c UNKNOWN
   * 9a256ec6b644a0f795376076f4e3ff4e85aaad1e UNKNOWN
   * b7dca63a2e98f09c8c780a21091b5268d897b220 UNKNOWN
   * 2b7bb05a4dbbb0883db566589dfe1975fdbb279d UNKNOWN
   * 23ab25410bbff3d717302c9a2549d05ab2aae330 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17305)
 
   * 60b3b4f14b78889b827c5e573b1d4ef992791d32 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17324)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-22498) cast the primary key for source table that has a decimal primary key as string, and then insert into a kudu table that has a string primary key throw the exception : Ups

2021-04-27 Thread Carl (Jira)

Carl created FLINK-22498:


 Summary: cast the primary key for source table that has a decimal 
primary key as string, and then insert into a kudu table that has a string 
primary key throw the exception : UpsertStreamTableSink requires that Table has 
a full primary keys if it is updated
 Key: FLINK-22498
 URL: https://issues.apache.org/jira/browse/FLINK-22498
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.12.1
 Environment: flink 1.12.1

jdk 1.8

hive 2.1.1

kudu 1.10.0

kafka 2.0.0
Reporter: Carl
 Attachments: bug.rar

*1. source table：*

CREATE TABLE ddl_source (
 appl_seq DECIMAL(16,2),
 name STRING,
 PRIMARY KEY(appl_seq) NOT ENFORCED
) WITH (
 'connector' = 'kafka',
 'topic' = 'ogg-json-03',
 'properties.bootstrap.servers' = ':9092',
 'value.format' = 'canal-json'
)

*2. sink table：*create the table use impala

create table rt_dwd.test_bug( 

    pk       string  ,

    name      string  ,

    primary key (pk)

) partition by hash (pk) partitions 5 stored as kudu 

TBLPROPERTIES  ('kudu.master_addresses' = ':7051');

*3. execute sql：*use blink planner

insert into kuducatalog.default_database.`rt_dwd.test_bug`
select CAST(appl_seq AS STRING), name  from ddl_source

 

*throw an exception :*

Exception in thread "main" org.apache.flink.table.api.TableException: 
UpsertStreamTableSink requires that Table has a full primary keys if it is 
updated.Exception in thread "main" org.apache.flink.table.api.TableException: 
UpsertStreamTableSink requires that Table has a full primary keys if it is 
updated. at 
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlanInternal(StreamExecLegacySink.scala:93)
 at 
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlanInternal(StreamExecLegacySink.scala:48)
 at 
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:59)
 at 
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLegacySink.translateToPlan(StreamExecLegacySink.scala:48)
 at 
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:66)
 at 
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:65)
 at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
 at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
 at scala.collection.Iterator$class.foreach(Iterator.scala:891) at 
scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at 
scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at 
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at 
scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at 
scala.collection.AbstractTraversable.map(Traversable.scala:104) at 
org.apache.flink.table.planner.delegation.StreamPlanner.translateToPlan(StreamPlanner.scala:65)
 at 
org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:167)
 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1329)
 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:676)
 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:767)
 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666)

 

*case A：if we use source table as follows, it will not throw the exception :*

CREATE TABLE ddl_source (
 appl_seq STRING,
 name STRING,
 PRIMARY KEY(appl_seq) NOT ENFORCED
) WITH (
 'connector' = 'kafka',
 'topic' = 'ogg-json-03',
 'properties.bootstrap.servers' = ':9092',
 'value.format' = 'canal-json'
)

 

*case B：or we ddl kudu table,and use sql as follows,  it will not throw the 
exception :*

_DDL：_

create table rt_dwd.test_bug( 

    pk       decimal(16,2),

    name      string  ,

    primary key (pk)

) partition by hash (pk) partitions 5 stored as kudu 

TBLPROPERTIES  ('kudu.master_addresses' = ':7051');

_DML：_

insert into kuducatalog.default_database.`rt_dwd.test_bug`
select  appl_seq, name  from ddl_source

 

*When debugging the source code, it may be related to SQL parsing engine*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-22497) When using DefaultRollingPolicy in StreamingFileSink, the file will be finished delayed



 [ 
https://issues.apache.org/jira/browse/FLINK-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChangjiGuo updated FLINK-22497:
---
Attachment: (was: 企业微信截图_43e36204-0a2f-4acd-ae17-56aa4d7661e4.png)

> When using DefaultRollingPolicy in StreamingFileSink, the file will be 
> finished delayed
> ---
>
> Key: FLINK-22497
> URL: https://issues.apache.org/jira/browse/FLINK-22497
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Affects Versions: 1.11.2
> Environment: hadoop-2.8.4
>Reporter: ChangjiGuo
>Priority: Major
> Attachments: 1.png
>
>
> I had a doubt when testing StreamingFileSink：
> The default 60s rolling interval in DefaultRollingPolicy is detected by 
> procTimeService. If the rolling interval is not met this time, it will be 
> delayed to the next timer trigger point (after 60s), so this is not 
> real-time. For example, if the checkpoint period is set to 60s, the file 
> should be converted to finished at the second checkpoint, but it will be 
> delayed to the third checkpoint.
> You can refer to the attached picture for detail.
> If we add rollingPolicy.shouldRollOnProcessingTime to the if condition of 
> Bucket.write method, the file will be set to finished in the second 
> checkpoint.
> {code:java}
> void write(IN element, long currentTime) throws IOException {
> if (inProgressPart == null || 
> rollingPolicy.shouldRollOnEvent(inProgressPart, element)
>   ||rollingPolicy.shouldRollOnProcessingTime(inProgressPart, 
> currentTime)) {
>   if (LOG.isDebugEnabled()) {
>   LOG.info("Subtask {} closing in-progress part file for 
> bucket id={} due to element {}.",
>   subtaskIndex, bucketId, element);
>   }
>   rollPartFile(currentTime);
>   }
>   inProgressPart.write(element, currentTime);
> }
> {code}
>  
> Is my understanding correct? 
> Thanks！ ^_^



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-22497) When using DefaultRollingPolicy in StreamingFileSink, the file will be finished delayed



 [ 
https://issues.apache.org/jira/browse/FLINK-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChangjiGuo updated FLINK-22497:
---
Attachment: 1.png

> When using DefaultRollingPolicy in StreamingFileSink, the file will be 
> finished delayed
> ---
>
> Key: FLINK-22497
> URL: https://issues.apache.org/jira/browse/FLINK-22497
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Affects Versions: 1.11.2
> Environment: hadoop-2.8.4
>Reporter: ChangjiGuo
>Priority: Major
> Attachments: 1.png
>
>
> I had a doubt when testing StreamingFileSink：
> The default 60s rolling interval in DefaultRollingPolicy is detected by 
> procTimeService. If the rolling interval is not met this time, it will be 
> delayed to the next timer trigger point (after 60s), so this is not 
> real-time. For example, if the checkpoint period is set to 60s, the file 
> should be converted to finished at the second checkpoint, but it will be 
> delayed to the third checkpoint.
> You can refer to the attached picture for detail.
> If we add rollingPolicy.shouldRollOnProcessingTime to the if condition of 
> Bucket.write method, the file will be set to finished in the second 
> checkpoint.
> {code:java}
> void write(IN element, long currentTime) throws IOException {
> if (inProgressPart == null || 
> rollingPolicy.shouldRollOnEvent(inProgressPart, element)
>   ||rollingPolicy.shouldRollOnProcessingTime(inProgressPart, 
> currentTime)) {
>   if (LOG.isDebugEnabled()) {
>   LOG.info("Subtask {} closing in-progress part file for 
> bucket id={} due to element {}.",
>   subtaskIndex, bucketId, element);
>   }
>   rollPartFile(currentTime);
>   }
>   inProgressPart.write(element, currentTime);
> }
> {code}
>  
> Is my understanding correct? 
> Thanks！ ^_^



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-22497) When using DefaultRollingPolicy in StreamingFileSink, the file will be finished delayed



 [ 
https://issues.apache.org/jira/browse/FLINK-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChangjiGuo updated FLINK-22497:
---
Attachment: 企业微信截图_43e36204-0a2f-4acd-ae17-56aa4d7661e4.png

> When using DefaultRollingPolicy in StreamingFileSink, the file will be 
> finished delayed
> ---
>
> Key: FLINK-22497
> URL: https://issues.apache.org/jira/browse/FLINK-22497
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Affects Versions: 1.11.2
> Environment: hadoop-2.8.4
>Reporter: ChangjiGuo
>Priority: Major
> Attachments: 1.png
>
>
> I had a doubt when testing StreamingFileSink：
> The default 60s rolling interval in DefaultRollingPolicy is detected by 
> procTimeService. If the rolling interval is not met this time, it will be 
> delayed to the next timer trigger point (after 60s), so this is not 
> real-time. For example, if the checkpoint period is set to 60s, the file 
> should be converted to finished at the second checkpoint, but it will be 
> delayed to the third checkpoint.
> You can refer to the attached picture for detail.
> If we add rollingPolicy.shouldRollOnProcessingTime to the if condition of 
> Bucket.write method, the file will be set to finished in the second 
> checkpoint.
> {code:java}
> void write(IN element, long currentTime) throws IOException {
> if (inProgressPart == null || 
> rollingPolicy.shouldRollOnEvent(inProgressPart, element)
>   ||rollingPolicy.shouldRollOnProcessingTime(inProgressPart, 
> currentTime)) {
>   if (LOG.isDebugEnabled()) {
>   LOG.info("Subtask {} closing in-progress part file for 
> bucket id={} due to element {}.",
>   subtaskIndex, bucketId, element);
>   }
>   rollPartFile(currentTime);
>   }
>   inProgressPart.write(element, currentTime);
> }
> {code}
>  
> Is my understanding correct? 
> Thanks！ ^_^



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (FLINK-10644) Batch Job: Speculative execution

2021-04-27 Thread wangwj (Jira)

[
https://issues.apache.org/jira/browse/FLINK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334427#comment-17334427
]

wangwj edited comment on FLINK-10644 at 4/28/21, 3:31 AM:
--

[~trohrmann]
Hi Till.

I am come from the search and recommendation department of Alibaba in China.
Our big data processing platform uses Flink Batch to process extremely huge
data every day.
There are many long-tail tasks every day, we can only manually go to the
machine to kill the process and seriously affect the experience of users. So I
made up my mind to solve this problem.

First of all, I think that speculative execution means that two executions in a
ExecutionVertex run at the same time. While failover means that two tasks run
at different times. Based on this theory, I think this feature(speculative
execution) is definitely achievable. Finally, the facts proved that my idea was
right.

So I have implemented a speculative execution for batch job based on Blink, and
it has a very significant effect in our product cluster. My approach is as
follows, happy to discuss them.

（1）Which kind of ExecutionJobVertex is suitable enable speculative execution
feature in a batch job?
Because of the speculative execution feature involves the implementation
details of the region failover. After research, I have decided a
ExecutionJobVertex will enable speculative execution feature only if all input
edges and output edges of this ExecutionJobVertex are blocking.

（2）How to distinguish long tail task？
I distinguish long tail task based on current time and the execution first
create/deploying time before it failover.
For ExecutionJobVertex that meets the condition (1)
When a configurable percentage of executions have been finished in an
ExecutionJobVertex, the speculative execution thread starts to really work.
In an ExecutionJobVertex, when the running time of a execution is a
configurable multiple of the median running time of other finished executions,
this execution is judged as long tail execution.

（3）How to make the speculative execution algorithm more precise?
Baesd on the speculative execution algorithm in (2), In our product cluster, I
can completely solve the long tail problem.
In the next step, we maybe add the throughput of the task to the speculative
execution algorithm through the heartbeat of TaskManager with JobManager.

（4）How schedule another execution in an same ExecutionVertex?
We have changed the currentExecution in ExecutionVertex to a list, which means
that there can be multiple executions in an ExecutionVertex at the same time.
Then we reuse the current scheduling logic to schedule the speculative
execution execution.

（5）How to make the speculative task runs on a different machine from the
original task.
We have implemented a machine-dimensional blacklist，and add the machine ip in
the blacklist when a execution is a long tail execution base on speculative
execution algorithm in (2). The blacklist has the ability of timed out.
When schedule executions we will add blacklist information to yarn
PlacementConstraint.
In this way, I can ensure that the yarn container is not on the machines in the
blacklist.

（6）How to avoid errors when multiple executions finish at the same time in an
ExecutionVertex?
In ExecutionVertex executionFinished() method, I have done multi-thread
synchronization, to ensure that an ExecutionVertex will eventually have only
one execution successfully finished, and other executions will all go to the
cancellation logic.

（7）How to deal with multiple sink files in one ExecutionVertex when the job is
sink to files?
When batch job will sink to file, we will add an executionAttemptID suffix to
the file name.
Finally in finalizeOnMaster() I will delete or rename files.
Here we should pay attention to the situation of flink stream job processing
bounded data sets.

（8）In batch job with all-to-all shuffle, how do we let the downstream original
execution and speculative execution know which ResultSubPartition to read of
upstream task?
Two executions of a upstream ExecutionVertex will produce two ResultPartition.
When upstream ExecutionVertex have finished we will update the input channel of
down stream execution to the fastest finished execution of upstream.
Here we should pay attention to the situation that the down stream execution
when meet DataConsumptionException. It will restarts with the upstream
execution that has been finished.

（9）How to display information about speculative task on the Flink web ui.
After I have implemented this feature. When speculative execution runs faster
then original execution, the flink ui will show that this task has been
cancelled. But the result of the job is correct, which is in full compliance
with our expectations.
I don’t know much about the web, I will ask my colleague for help.

[~trohrmann]
My implementation has

[jira] [Created] (FLINK-22497) When using DefaultRollingPolicy in StreamingFileSink, the file will be finished delayed

ChangjiGuo created FLINK-22497:
--

 Summary: When using DefaultRollingPolicy in StreamingFileSink, the 
file will be finished delayed
 Key: FLINK-22497
 URL: https://issues.apache.org/jira/browse/FLINK-22497
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Affects Versions: 1.11.2
 Environment: hadoop-2.8.4
Reporter: ChangjiGuo


I had a doubt when testing StreamingFileSink：

The default 60s rolling interval in DefaultRollingPolicy is detected by 
procTimeService. If the rolling interval is not met this time, it will be 
delayed to the next timer trigger point (after 60s), so this is not real-time. 
For example, if the checkpoint period is set to 60s, the file should be 
converted to finished at the second checkpoint, but it will be delayed to the 
third checkpoint.

You can refer to the attached picture for detail.

If we add rollingPolicy.shouldRollOnProcessingTime to the if condition of 
Bucket.write method, the file will be set to finished in the second checkpoint.
{code:java}
void write(IN element, long currentTime) throws IOException {
if (inProgressPart == null || 
rollingPolicy.shouldRollOnEvent(inProgressPart, element)
||rollingPolicy.shouldRollOnProcessingTime(inProgressPart, 
currentTime)) {
if (LOG.isDebugEnabled()) {
LOG.info("Subtask {} closing in-progress part file for 
bucket id={} due to element {}.",
subtaskIndex, bucketId, element);
}
rollPartFile(currentTime);
}
inProgressPart.write(element, currentTime);
}
{code}
 
Is my understanding correct? 
Thanks！ ^_^



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-22438) add numRecordsOut metric for Async IO

2021-04-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-22438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-22438:
---
Labels: pull-request-available  (was: )

> add numRecordsOut metric for Async IO
> -
>
> Key: FLINK-22438
> URL: https://issues.apache.org/jira/browse/FLINK-22438
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Metrics, Runtime / Task
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Zhengqi Zhang
>Assignee: Zhengqi Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.12.3, 1.13.1
>
> Attachments: QQ截图20210424004201.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In Flink WebUI,there is no numRecordsOut metric,and the class 
> AsyncWaitOperator did't have this metric in fact.Other operators have this 
> metric, which makes it difficult to monitor Async IO operator and can cause 
> confusion for users.
> I think we can directly  use the wrapping output class CountingOutput to 
> update numRecordsOut metric.CountingOutput is used in super class of 
> AsyncWaitOperator(AbstractStreamOperator).
> Here is my commit,And I have run a test, it work.
> [my  
> commit|https://github.com/onyourhead/flink/commit/58a8ac27b292280696639caa2e311637cd631a00]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] onyourhead opened a new pull request #15791: [FLINK-22438][metrics] Add numRecordsOut metric for Async IO



onyourhead opened a new pull request #15791:
URL: https://github.com/apache/flink/pull/15791


   ## What is the purpose of the change
   
   *In Flink WebUI,there is no numRecordsOut metric in Async IO operator while 
other operator have, this makes it difficult to monitor Async IO operator and 
can cause confusion for users.The commit is about to fix it. see 
https://issues.apache.org/jira/browse/FLINK-22438*
   
   ## Brief change log
   
 - *Use the `output` field in the super class of `AsyncWaitOperator` to 
update numRecordsOut metric.*
   
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-10644) Batch Job: Speculative execution

2021-04-27 Thread wangwj (Jira)

[
https://issues.apache.org/jira/browse/FLINK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334427#comment-17334427
]

wangwj edited comment on FLINK-10644 at 4/28/21, 3:21 AM:
--

[~trohrmann]
Hi Till.

So I have implemented a speculative execution for batch job based on Blink, and
it has a very significant effect in our product cluster. My approach is as
follows, happy to discuss them.

[~trohrmann]
I am very interested in

[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf



flinkbot edited a comment on pull request #14376:
URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765


   
   ## CI report:
   
   * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN
   * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN
   * bd27dabd13c11fd65b98423595e9d93234538444 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=16330)
 
   * e3f953d91778ef96ad5fe222dd9da229dc2c5619 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] maosuhan commented on pull request #14376: [FLINK-18202][PB format] New Format of protobuf



maosuhan commented on pull request #14376:
URL: https://github.com/apache/flink/pull/14376#issuecomment-828109372


   @sv3ndk Sorry for late reply.
   We had a discussion offline and we think that the rules that you proposed 
are more strict but introduce many different cases. We plan to deliver a more 
simple version of protobuf format in first release. So we will still use 
original 'read-default-values' options.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-10644) Batch Job: Speculative execution

2021-04-27 Thread wangwj (Jira)

[
https://issues.apache.org/jira/browse/FLINK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334427#comment-17334427
]

wangwj commented on FLINK-10644:

[~trohrmann]
Hi Till.

So I have implemented a speculative execution for batch job based on Blink, and
it has a very significant effect in our product cluster. My approach is as
follows, happy to discuss them.

（3）How to make the speculative execution algorithm more precise?
Baesd on the speculative execution algorithm in (2), In our product cluster, I
can completely solve the long tail problem.
In the next step, we will add the throughput of the task to the speculative
execution algorithm through the heartbeat of TaskManager with JobManager.

（4）How schedule another Execution in an same ExecutionVertex?
We have changed the currentExecution in ExecutionVertex to a list, which means
that there can be multiple executions in an ExecutionVertex at the same time.
Then we reuse the current scheduling logic to schedule the speculative
execution execution.

（7）How to deal with multiple sink files in one ExecutionVertex when the job is
sink to files?
When batch job will sink to file, we will add an executionAttemptID suffix to
the file name.
Finally in finalizeOnMaster() I will delete or rename them.
Here we should pay attention to the situation of flink stream job processing
bounded data sets.

（8）In batch job with all-to-all shuffle, how do we let the downstream original
execution and speculative execution know which ResultSubPartition to read of
upstream task?
Two executions of a upstream ExecutionVertex will produce two
ResultPartition。When edge is blocking between upstream and down stream. When
upstream ExecutionVertex have finished we will update the input channel of down
stream execution to the fastest finished execution of upstream.
Here we should pay attention to the situation that the down stream execution
when meet DataConsumptionException. It will restarts with the upstream
execution that has been finished.
Here we should pay attention to the situation of flink stream job processing
bounded data sets.

[~trohrmann]
I am very interested in this issue, and my implementation has played a big role
in our product

[GitHub] [flink-web] xintongsong commented on a change in pull request #436: Add Apache Flink release 1.13.0



xintongsong commented on a change in pull request #436:
URL: https://github.com/apache/flink-web/pull/436#discussion_r621776893



##
File path: _posts/2021-04-22-release-1.13.0.md
##
@@ -0,0 +1,483 @@
+---
+layout: post 
+title:  "Apache Flink 1.13.0 Release Announcement"
+date: 2021-04-22T08:00:00.000Z 
+categories: news 
+authors:
+- stephan:
+  name: "Stephan Ewen"
+  twitter: "StephanEwen"
+- dwysakowicz:
+  name: "Dawid Wysakowicz"
+  twitter: "dwysakowicz"
+
+excerpt: The Apache Flink community is excited to announce the release of 
Flink 1.13.0! Close to xxx contributors worked on over xxx threads to bring 
significant improvements to usability and observability as well as new features 
that improve elasticity of Flink’s Application-style deployments.
+---
+
+
+The Apache Flink community is excited to announce the release of Flink 1.13.0! 
More than 200
+contributors worked on over 1k threads to bring significant improvements to 
usability and
+observability as well as new features that improve elasticity of Flink’s 
Application-style
+deployments.
+
+This release brings us a big step forward in one of our major efforts: Making 
Stream Processing
+Applications as natural and as simple to manage as any other application. The 
new reactive scaling
+mode means that scaling streaming applications in and out now works like in 
any other application,
+by just changing the number of parallel processes.
+
+We also added a series of improvements that help users better understand the 
performance of
+applications. When the streams don't flow as fast as you’d hope, these can 
help you to understand
+why: Load and backpressure visualization to identify bottlenecks, CPU flame 
graphs to identify hot
+code paths in your application, and State Access Latencies to see how the 
State Backends are keeping
+up.
+
+This blog post describes all major new features and improvements, important 
changes to be aware of
+and what to expect moving forward.
+
+{% toc %}
+
+We encourage you to [download the 
release](https://flink.apache.org/downloads.html) and share your
+feedback with the community through
+the [Flink mailing 
lists](https://flink.apache.org/community.html#mailing-lists)
+or [JIRA](https://issues.apache.org/jira/projects/FLINK/summary).
+
+## Notable Features and Improvements
+
+### Reactive mode
+
+The Reactive Mode is the latest piece in Flink's initiative for making Stream 
Processing
+Applications as natural and as simple to manage as any other application.
+
+Flink has a dual nature when it comes to resource management and deployments: 
You can deploy
+clusters onto Resource Managers like Kubernetes or Yarn in such a way that 
Flink actively manages
+the resource, and allocates and releases workers as needed. That is especially 
useful for jobs and
+applications that rapidly change their required resources, like batch 
applications and ad-hoc SQL
+queries. The application parallelism rules, the number of workers follows. We 
call this active
+scaling.
+
+For long running streaming applications, it is often a nicer model to just 
deploy them like any
+other long-running application: The application doesn't really need to know 
that it runs on K8s,
+EKS, Yarn, etc. and doesn't try to acquire a specific amount of workers; 
instead, it just uses the
+number of workers that is given to it. The number of workers rules, the 
application parallelism
+adjusts to that. We call that re-active scaling.
+
+The Application Deployment Mode started this effort, making deployments 
application-like (avoiding
+having to separate deployment steps to (1) start cluster and (2) submit 
application). The reactive
+scheduler completes this, and you now don't have to use extra tools (scripts 
or a K8s operator) any
+more to keep the number of workers and the application parallelism settings in 
sync.
+
+You can now put an auto-scaler around Flink applications like around other 
typical applications — as
+long as you are mindful when configuring the autoscaler that stateful 
applications still spend
+effort in moving state around when scaling.
+
+
+### Bottleneck detection, Backpressure and Idleness Monitoring
+
+One of the most important metrics to investigate when a job does not consume 
records as fast as you
+would expect is the backpressure ratio. It lets you track down bottlenecks in 
your pipelines. The
+current mechanism had two limitations:
+It was heavy, because it worked by repeatedly taking stack trace samples of 
your running tasks. It
+was difficult to find out which vertex was the source of backpressure. In 
Flink 1.13, we reworked
+the mechanism to include new metrics for the time tasks spend being 
backpressured, along with a
+reworked graphical representation of the job (including a percentage of time 
particular vertices are
+backpressured).
+
+
+
+  
+
+
+### Support for CPU flame graphs in Web UI
+
+It is desirable to provide better visibility into the distribution of CPU 
resources while

[GitHub] [flink] flinkbot edited a comment on pull request #15790: [hotfix][python][docs] add bundling udfs section



flinkbot edited a comment on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828062007


   
   ## CI report:
   
   * 40ff67a835fa6304cf3fe72e642e4ac6a3af76a8 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17319)
 
   * 57608d10dc311491f674c9e0c1f350d127b49c4c UNKNOWN
   * 86c54e97d3657b5e572988c8987979f39983590d UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15745: [FLINK-22304][table] Refactor some interfaces for TVF based window to…



flinkbot edited a comment on pull request #15745:
URL: https://github.com/apache/flink/pull/15745#issuecomment-826249839


   
   ## CI report:
   
   * 588bfb357b6bb193b446af68f1ff48f10252da1d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17287)
 
   * 9e7da4cae78b1e44abaf51805fa0f89f3c81fe2d Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17321)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-20950) SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with "AssertionError"



[ 
https://issues.apache.org/jira/browse/FLINK-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334423#comment-17334423
 ] 

Guowei Ma commented on FLINK-20950:
---

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17315=logs=119bbba7-f5e3-5e08-e72d-09f1529665de=7dc1f5a9-54e1-502e-8b02-c7df69073cfc=4238

> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with 
> "AssertionError"
> ---
>
> Key: FLINK-20950
> URL: https://issues.apache.org/jira/browse/FLINK-20950
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Huang Xingbo
>Priority: Critical
>  Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11940=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2]
> {code:java}
> 2021-01-12T16:47:00.7579536Z [ERROR] Failures: 
> 2021-01-12T16:47:00.7580061Z [ERROR]   
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode:218 
> 2021-01-12T16:47:00.7587186Z Expected: iterable over 
> ["(895,null,-9223372036854775808)", "(895,null,-9223372036854775808)", 
> "(127,null,-9223372036854775808)", "(127,null,-9223372036854775808)", 
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)", 
> "(161,null,-9223372036854775808)", "(161,null,-9223372036854775808)", 
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)", 
> "(662,null,-9223372036854775808)", "(662,null,-9223372036854775808)", 
> "(822,null,-9223372036854775808)", "(822,null,-9223372036854775808)", 
> "(491,null,-9223372036854775808)", "(491,null,-9223372036854775808)", 
> "(275,null,-9223372036854775808)", "(275,null,-9223372036854775808)", 
> "(122,null,-9223372036854775808)", "(122,null,-9223372036854775808)", 
> "(850,null,-9223372036854775808)", "(850,null,-9223372036854775808)", 
> "(630,null,-9223372036854775808)", "(630,null,-9223372036854775808)", 
> "(682,null,-9223372036854775808)", "(682,null,-9223372036854775808)", 
> "(765,null,-9223372036854775808)", "(765,null,-9223372036854775808)", 
> "(434,null,-9223372036854775808)", "(434,null,-9223372036854775808)", 
> "(970,null,-9223372036854775808)", "(970,null,-9223372036854775808)", 
> "(714,null,-9223372036854775808)", "(714,null,-9223372036854775808)", 
> "(795,null,-9223372036854775808)", "(795,null,-9223372036854775808)", 
> "(288,null,-9223372036854775808)", "(288,null,-9223372036854775808)", 
> "(422,null,-9223372036854775808)", "(422,null,-9223372036854775808)"] in any 
> order
> 2021-01-12T16:47:00.7591663Z  but: Not matched: "end of input"
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-20950) SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with "AssertionError"



 [ 
https://issues.apache.org/jira/browse/FLINK-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guowei Ma reassigned FLINK-20950:
-

Assignee: Guowei Ma

> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with 
> "AssertionError"
> ---
>
> Key: FLINK-20950
> URL: https://issues.apache.org/jira/browse/FLINK-20950
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Huang Xingbo
>Assignee: Guowei Ma
>Priority: Critical
>  Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11940=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2]
> {code:java}
> 2021-01-12T16:47:00.7579536Z [ERROR] Failures: 
> 2021-01-12T16:47:00.7580061Z [ERROR]   
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode:218 
> 2021-01-12T16:47:00.7587186Z Expected: iterable over 
> ["(895,null,-9223372036854775808)", "(895,null,-9223372036854775808)", 
> "(127,null,-9223372036854775808)", "(127,null,-9223372036854775808)", 
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)", 
> "(161,null,-9223372036854775808)", "(161,null,-9223372036854775808)", 
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)", 
> "(662,null,-9223372036854775808)", "(662,null,-9223372036854775808)", 
> "(822,null,-9223372036854775808)", "(822,null,-9223372036854775808)", 
> "(491,null,-9223372036854775808)", "(491,null,-9223372036854775808)", 
> "(275,null,-9223372036854775808)", "(275,null,-9223372036854775808)", 
> "(122,null,-9223372036854775808)", "(122,null,-9223372036854775808)", 
> "(850,null,-9223372036854775808)", "(850,null,-9223372036854775808)", 
> "(630,null,-9223372036854775808)", "(630,null,-9223372036854775808)", 
> "(682,null,-9223372036854775808)", "(682,null,-9223372036854775808)", 
> "(765,null,-9223372036854775808)", "(765,null,-9223372036854775808)", 
> "(434,null,-9223372036854775808)", "(434,null,-9223372036854775808)", 
> "(970,null,-9223372036854775808)", "(970,null,-9223372036854775808)", 
> "(714,null,-9223372036854775808)", "(714,null,-9223372036854775808)", 
> "(795,null,-9223372036854775808)", "(795,null,-9223372036854775808)", 
> "(288,null,-9223372036854775808)", "(288,null,-9223372036854775808)", 
> "(422,null,-9223372036854775808)", "(422,null,-9223372036854775808)"] in any 
> order
> 2021-01-12T16:47:00.7591663Z  but: Not matched: "end of input"
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-20950) SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with "AssertionError"



 [ 
https://issues.apache.org/jira/browse/FLINK-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guowei Ma updated FLINK-20950:
--
Labels: test-stability  (was: auto-unassigned test-stability)

> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with 
> "AssertionError"
> ---
>
> Key: FLINK-20950
> URL: https://issues.apache.org/jira/browse/FLINK-20950
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Huang Xingbo
>Priority: Critical
>  Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11940=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2]
> {code:java}
> 2021-01-12T16:47:00.7579536Z [ERROR] Failures: 
> 2021-01-12T16:47:00.7580061Z [ERROR]   
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode:218 
> 2021-01-12T16:47:00.7587186Z Expected: iterable over 
> ["(895,null,-9223372036854775808)", "(895,null,-9223372036854775808)", 
> "(127,null,-9223372036854775808)", "(127,null,-9223372036854775808)", 
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)", 
> "(161,null,-9223372036854775808)", "(161,null,-9223372036854775808)", 
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)", 
> "(662,null,-9223372036854775808)", "(662,null,-9223372036854775808)", 
> "(822,null,-9223372036854775808)", "(822,null,-9223372036854775808)", 
> "(491,null,-9223372036854775808)", "(491,null,-9223372036854775808)", 
> "(275,null,-9223372036854775808)", "(275,null,-9223372036854775808)", 
> "(122,null,-9223372036854775808)", "(122,null,-9223372036854775808)", 
> "(850,null,-9223372036854775808)", "(850,null,-9223372036854775808)", 
> "(630,null,-9223372036854775808)", "(630,null,-9223372036854775808)", 
> "(682,null,-9223372036854775808)", "(682,null,-9223372036854775808)", 
> "(765,null,-9223372036854775808)", "(765,null,-9223372036854775808)", 
> "(434,null,-9223372036854775808)", "(434,null,-9223372036854775808)", 
> "(970,null,-9223372036854775808)", "(970,null,-9223372036854775808)", 
> "(714,null,-9223372036854775808)", "(714,null,-9223372036854775808)", 
> "(795,null,-9223372036854775808)", "(795,null,-9223372036854775808)", 
> "(288,null,-9223372036854775808)", "(288,null,-9223372036854775808)", 
> "(422,null,-9223372036854775808)", "(422,null,-9223372036854775808)"] in any 
> order
> 2021-01-12T16:47:00.7591663Z  but: Not matched: "end of input"
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] YikSanChan commented on pull request #15790: [hotfix][python][docs] add bundling udfs section



YikSanChan commented on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828105045


   @dianfu thanks for pointing out, fixed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] xintongsong commented on a change in pull request #436: Add Apache Flink release 1.13.0



xintongsong commented on a change in pull request #436:
URL: https://github.com/apache/flink-web/pull/436#discussion_r621774146



##
File path: _posts/2021-04-22-release-1.13.0.md
##
@@ -0,0 +1,483 @@
+---
+layout: post 
+title:  "Apache Flink 1.13.0 Release Announcement"
+date: 2021-04-22T08:00:00.000Z 
+categories: news 
+authors:
+- stephan:
+  name: "Stephan Ewen"
+  twitter: "StephanEwen"
+- dwysakowicz:
+  name: "Dawid Wysakowicz"
+  twitter: "dwysakowicz"
+
+excerpt: The Apache Flink community is excited to announce the release of 
Flink 1.13.0! Close to xxx contributors worked on over xxx threads to bring 
significant improvements to usability and observability as well as new features 
that improve elasticity of Flink’s Application-style deployments.
+---
+
+
+The Apache Flink community is excited to announce the release of Flink 1.13.0! 
More than 200
+contributors worked on over 1k threads to bring significant improvements to 
usability and
+observability as well as new features that improve elasticity of Flink’s 
Application-style
+deployments.
+
+This release brings us a big step forward in one of our major efforts: Making 
Stream Processing
+Applications as natural and as simple to manage as any other application. The 
new reactive scaling
+mode means that scaling streaming applications in and out now works like in 
any other application,
+by just changing the number of parallel processes.
+
+We also added a series of improvements that help users better understand the 
performance of
+applications. When the streams don't flow as fast as you’d hope, these can 
help you to understand
+why: Load and backpressure visualization to identify bottlenecks, CPU flame 
graphs to identify hot
+code paths in your application, and State Access Latencies to see how the 
State Backends are keeping
+up.
+
+This blog post describes all major new features and improvements, important 
changes to be aware of
+and what to expect moving forward.
+
+{% toc %}
+
+We encourage you to [download the 
release](https://flink.apache.org/downloads.html) and share your
+feedback with the community through
+the [Flink mailing 
lists](https://flink.apache.org/community.html#mailing-lists)
+or [JIRA](https://issues.apache.org/jira/projects/FLINK/summary).
+
+## Notable Features and Improvements
+
+### Reactive mode
+
+The Reactive Mode is the latest piece in Flink's initiative for making Stream 
Processing
+Applications as natural and as simple to manage as any other application.
+
+Flink has a dual nature when it comes to resource management and deployments: 
You can deploy
+clusters onto Resource Managers like Kubernetes or Yarn in such a way that 
Flink actively manages
+the resource, and allocates and releases workers as needed. That is especially 
useful for jobs and
+applications that rapidly change their required resources, like batch 
applications and ad-hoc SQL
+queries. The application parallelism rules, the number of workers follows. We 
call this active
+scaling.
+
+For long running streaming applications, it is often a nicer model to just 
deploy them like any
+other long-running application: The application doesn't really need to know 
that it runs on K8s,
+EKS, Yarn, etc. and doesn't try to acquire a specific amount of workers; 
instead, it just uses the
+number of workers that is given to it. The number of workers rules, the 
application parallelism
+adjusts to that. We call that re-active scaling.
+
+The Application Deployment Mode started this effort, making deployments 
application-like (avoiding
+having to separate deployment steps to (1) start cluster and (2) submit 
application). The reactive
+scheduler completes this, and you now don't have to use extra tools (scripts 
or a K8s operator) any
+more to keep the number of workers and the application parallelism settings in 
sync.
+
+You can now put an auto-scaler around Flink applications like around other 
typical applications — as
+long as you are mindful when configuring the autoscaler that stateful 
applications still spend
+effort in moving state around when scaling.
+
+
+### Bottleneck detection, Backpressure and Idleness Monitoring
+
+One of the most important metrics to investigate when a job does not consume 
records as fast as you
+would expect is the backpressure ratio. It lets you track down bottlenecks in 
your pipelines. The
+current mechanism had two limitations:
+It was heavy, because it worked by repeatedly taking stack trace samples of 
your running tasks. It
+was difficult to find out which vertex was the source of backpressure. In 
Flink 1.13, we reworked
+the mechanism to include new metrics for the time tasks spend being 
backpressured, along with a
+reworked graphical representation of the job (including a percentage of time 
particular vertices are
+backpressured).
+
+
+
+  
+
+
+### Support for CPU flame graphs in Web UI
+
+It is desirable to provide better visibility into the distribution of CPU 
resources while

[GitHub] [flink] dianfu commented on a change in pull request #15790: [hotfix][python][docs] add bundling udfs section



dianfu commented on a change in pull request #15790:
URL: https://github.com/apache/flink/pull/15790#discussion_r621771601



##
File path: docs/content.zh/docs/dev/python/table/udfs/python_udfs.md
##
@@ -554,3 +554,11 @@ class 
ListViewConcatTableAggregateFunction(TableAggregateFunction):
 def get_result_type(self):
 return DataTypes.ROW([DataTypes.FIELD("a", DataTypes.STRING())])
 ```
+
+## 打包 UDFs
+
+To run Python UDFs (as well as Pandas UDFs) in any non-local mode, it is 
strongly recommended to bundle your Python UDF definitions using the config 
option [`python-files`]({{< ref "docs/dev/python/python_config" 
>}}#python-files), if your Python UDFs live outside of the file where the 
`main()` function is defined.

Review comment:
   It's not necessary to keep the English version if you have already 
translated it to Chinese. The paragraph should be removed.

##
File path: docs/content.zh/docs/dev/python/table/udfs/python_udfs.md
##
@@ -554,3 +554,11 @@ class 
ListViewConcatTableAggregateFunction(TableAggregateFunction):
 def get_result_type(self):
 return DataTypes.ROW([DataTypes.FIELD("a", DataTypes.STRING())])
 ```
+
+## 打包 UDFs
+
+To run Python UDFs (as well as Pandas UDFs) in any non-local mode, it is 
strongly recommended to bundle your Python UDF definitions using the config 
option [`python-files`]({{< ref "docs/dev/python/python_config" 
>}}#python-files), if your Python UDFs live outside of the file where the 
`main()` function is defined.
+Otherwise, you may run into `ModuleNotFoundError: No module named 'my_udf'` if 
you define Python UDFs in a file called `my_udf.py`.
+
+如果你在非 local 模式下运行 Python UDFs 和 Pandas UDFs，且 Python UDFs 没有定义在含 `main()` 入口的 
Python 主文件中，我们强烈建议你通过 [`python-files`]({{< ref "docs/dev/python/python_config" 
>}}#python-files) 配置项将 Python UDF 的定义打包起来。

Review comment:
   ```suggestion
   如果你在非 local 模式下运行 Python UDFs 和 Pandas UDFs，且 Python UDFs 没有定义在含 `main()` 
入口的 Python 主文件中，强烈建议你通过 [`python-files`]({{< ref 
"docs/dev/python/python_config" >}}#python-files) 配置项指定 Python UDF 的定义。
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-19028) Translate the "application_parameters.zh.md" into Chinese

2021-04-27 Thread jjiey (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-19028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334419#comment-17334419
 ] 

jjiey commented on FLINK-19028:
---

hi, [~jark], I am willing to do it. Can you assign it to me? thank you.

> Translate the "application_parameters.zh.md"  into Chinese
> --
>
> Key: FLINK-19028
> URL: https://issues.apache.org/jira/browse/FLINK-19028
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Reporter: pp
>Priority: Major
>  Labels: pull-request-available, stale-major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #15790: [hotfix][python][docs] add bundling udfs section



flinkbot edited a comment on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828062007


   
   ## CI report:
   
   * 40ff67a835fa6304cf3fe72e642e4ac6a3af76a8 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17319)
 
   * 57608d10dc311491f674c9e0c1f350d127b49c4c UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-22496) ClusterEntrypointTest.testCloseAsyncShouldBeExecutedInShutdownHook failed

Guowei Ma created FLINK-22496:
-

 Summary: 
ClusterEntrypointTest.testCloseAsyncShouldBeExecutedInShutdownHook failed
 Key: FLINK-22496
 URL: https://issues.apache.org/jira/browse/FLINK-22496
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17313=logs=21408240-6569-5a01-c099-3adfe83ce651=b2761bb8-3852-5a0d-bc43-6a1d327b63cb=6207


{code:java}
52 [ERROR] 
testCloseAsyncShouldBeExecutedInShutdownHook(org.apache.flink.runtime.entrypoint.ClusterEntrypointTest)
  Time elapsed: 9.83 s  <<< FAILURE!
Apr 27 21:20:52 java.lang.AssertionError: 
Apr 27 21:20:52 Process 843 does not exit within 3000 ms
Apr 27 21:20:52 Expected: is 
Apr 27 21:20:52  but: was 
Apr 27 21:20:52 at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
Apr 27 21:20:52 at org.junit.Assert.assertThat(Assert.java:956)
Apr 27 21:20:52 at 
org.apache.flink.runtime.entrypoint.ClusterEntrypointTest.testCloseAsyncShouldBeExecutedInShutdownHook(ClusterEntrypointTest.java:224)
Apr 27 21:20:52 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Apr 27 21:20:52 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 27 21:20:52 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 27 21:20:52 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 27 21:20:52 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 27 21:20:52 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 27 21:20:52 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 27 21:20:52 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Apr 27 21:20:52 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Apr 27 21:20:52 at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Apr 27 21:20:52 at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
Apr 27 21:20:52 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
Apr 27 21:20:52 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
Apr 27 21:20:52 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
Apr 27 21:20:52 at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Apr 27 21:20:52 at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Apr 27 21:20:52 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #15745: [FLINK-22304][table] Refactor some interfaces for TVF based window to…



flinkbot edited a comment on pull request #15745:
URL: https://github.com/apache/flink/pull/15745#issuecomment-826249839


   
   ## CI report:
   
   * 588bfb357b6bb193b446af68f1ff48f10252da1d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17287)
 
   * 9e7da4cae78b1e44abaf51805fa0f89f3c81fe2d UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Reopened] (FLINK-21121) TaggedOperatorSubtaskState is missing when creating a new savepoint using state processor api

2021-04-27 Thread binguo (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-21121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

binguo reopened FLINK-21121:


We are not sure whether adding the state-processor-api to the job running the 
manipulated savepoint is something intended? If it is, it should be mentioned 
in the docs.

> TaggedOperatorSubtaskState is missing when creating a new savepoint using 
> state processor api
> -
>
> Key: FLINK-21121
> URL: https://issues.apache.org/jira/browse/FLINK-21121
> Project: Flink
>  Issue Type: Bug
>  Components: API / State Processor
>Affects Versions: 1.11.0
>Reporter: binguo
>Priority: Major
>
> I am getting an exception when using the Flink State Processor API to write a 
> new SavePoint, which is:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at 
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:204)
> at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
> at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.FlinkException: Could not restore operator 
> state backend for StreamSource_e8ea6e352a1a627513ffbd4573fa1628_(1/1) from 
> any of the 1 provided restore options.
> at 
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> at 
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.operatorStateBackend(StreamTaskStateInitializerImpl.java:265)
> at 
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:152)
> ... 9 more
> Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed 
> when trying to restore operator state backend
> at 
> org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder.build(DefaultOperatorStateBackendBuilder.java:86)
> at 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createOperatorStateBackend(RocksDBStateBackend.java:552)
> at 
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$operatorStateBackend$0(StreamTaskStateInitializerImpl.java:256)
> at 
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> at 
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> ... 11 more
> Caused by: java.lang.IllegalStateException: Missing value for the key 
> 'org.apache.flink.state.api.output.TaggedOperatorSubtaskState'
> at 
> org.apache.flink.util.LinkedOptionalMap.unwrapOptionals(LinkedOptionalMap.java:190)
> at 
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializerSnapshot.restoreSerializer(KryoSerializerSnapshot.java:86)
> at 
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at 
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
> at 
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
> at 
> org.apache.flink.api.common.typeutils.NestedSerializersSnapshotDelegate.snapshotsToRestoreSerializers(NestedSerializersSnapshotDelegate.java:225)
> at 
> org.apache.flink.api.common.typeutils.NestedSerializersSnapshotDelegate.getRestoredNestedSerializers(NestedSerializersSnapshotDelegate.java:83)
> at 
> org.apache.flink.api.common.typeutils.CompositeTypeSerializerSnapshot.restoreSerializer(CompositeTypeSerializerSnapshot.java:204)
>

[jira] [Commented] (FLINK-17274) Maven: Premature end of Content-Length delimited message body



[ 
https://issues.apache.org/jira/browse/FLINK-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334413#comment-17334413
 ] 

Guowei Ma commented on FLINK-17274:
---

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17313=logs=3e60b793-4158-5027-ac6d-4cdc51dffe1e=d5ed4970-7667-5f7e-2ece-62e410f74748=10292

> Maven: Premature end of Content-Length delimited message body
> -
>
> Key: FLINK-17274
> URL: https://issues.apache.org/jira/browse/FLINK-17274
> Project: Flink
>  Issue Type: Bug
>  Components: Build System / Azure Pipelines
>Reporter: Robert Metzger
>Priority: Critical
>  Labels: test-stability
>
> CI: 
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=7786=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb
> {code}
> [ERROR] Failed to execute goal on project 
> flink-connector-elasticsearch7_2.11: Could not resolve dependencies for 
> project 
> org.apache.flink:flink-connector-elasticsearch7_2.11:jar:1.11-SNAPSHOT: Could 
> not transfer artifact org.apache.lucene:lucene-sandbox:jar:8.3.0 from/to 
> alicloud-mvn-mirror 
> (http://mavenmirror.alicloud.dak8s.net:/repository/maven-central/): GET 
> request of: org/apache/lucene/lucene-sandbox/8.3.0/lucene-sandbox-8.3.0.jar 
> from alicloud-mvn-mirror failed: Premature end of Content-Length delimited 
> message body (expected: 289920; received: 239832 -> [Help 1]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-22438) add numRecordsOut metric for Async IO

2021-04-27 Thread Zhengqi Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-22438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334412#comment-17334412
 ] 

Zhengqi Zhang commented on FLINK-22438:
---

Yes,I'll submit a PR later

> add numRecordsOut metric for Async IO
> -
>
> Key: FLINK-22438
> URL: https://issues.apache.org/jira/browse/FLINK-22438
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Metrics, Runtime / Task
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Zhengqi Zhang
>Assignee: Zhengqi Zhang
>Priority: Major
> Fix For: 1.14.0, 1.12.3, 1.13.1
>
> Attachments: QQ截图20210424004201.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In Flink WebUI,there is no numRecordsOut metric,and the class 
> AsyncWaitOperator did't have this metric in fact.Other operators have this 
> metric, which makes it difficult to monitor Async IO operator and can cause 
> confusion for users.
> I think we can directly  use the wrapping output class CountingOutput to 
> update numRecordsOut metric.CountingOutput is used in super class of 
> AsyncWaitOperator(AbstractStreamOperator).
> Here is my commit,And I have run a test, it work.
> [my  
> commit|https://github.com/onyourhead/flink/commit/58a8ac27b292280696639caa2e311637cd631a00]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-18444) KafkaITCase failing with "Failed to send data to Kafka: This server does not host this topic-partition"



[ 
https://issues.apache.org/jira/browse/FLINK-18444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334411#comment-17334411
 ] 

Guowei Ma commented on FLINK-18444:
---

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17314=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=eb5f4d19-2d2d-5856-a4ce-acf5f904a994=11854

> KafkaITCase failing with "Failed to send data to Kafka: This server does not 
> host this topic-partition"
> ---
>
> Key: FLINK-18444
> URL: https://issues.apache.org/jira/browse/FLINK-18444
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Critical
>  Labels: stale-critical, test-stability
>
> Instance on master: 
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4092=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f
> {code}
> 2020-06-28T21:37:54.8113215Z [ERROR] 
> testMultipleSourcesOnePartition(org.apache.flink.streaming.connectors.kafka.KafkaITCase)
>   Time elapsed: 5.079 s  <<< ERROR!
> 2020-06-28T21:37:54.8113885Z 
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2020-06-28T21:37:54.8114418Z  at 
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> 2020-06-28T21:37:54.8114905Z  at 
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:677)
> 2020-06-28T21:37:54.8115397Z  at 
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:81)
> 2020-06-28T21:37:54.8116254Z  at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1699)
> 2020-06-28T21:37:54.8116857Z  at 
> org.apache.flink.streaming.connectors.kafka.testutils.DataGenerators.generateRandomizedIntegerSequence(DataGenerators.java:120)
> 2020-06-28T21:37:54.8117715Z  at 
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:933)
> 2020-06-28T21:37:54.8118327Z  at 
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testMultipleSourcesOnePartition(KafkaITCase.java:107)
> 2020-06-28T21:37:54.8118805Z  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-06-28T21:37:54.8119859Z  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-06-28T21:37:54.8120861Z  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-06-28T21:37:54.8121436Z  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2020-06-28T21:37:54.8121899Z  at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-06-28T21:37:54.8122424Z  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-06-28T21:37:54.8122942Z  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-06-28T21:37:54.8123406Z  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-06-28T21:37:54.8123899Z  at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2020-06-28T21:37:54.8124507Z  at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2020-06-28T21:37:54.8124978Z  at 
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 2020-06-28T21:37:54.8125332Z  at 
> java.base/java.lang.Thread.run(Thread.java:834)
> 2020-06-28T21:37:54.8125743Z Caused by: 
> org.apache.flink.runtime.JobException: Recovery is suppressed by 
> NoRestartBackoffTimeStrategy
> 2020-06-28T21:37:54.8126305Z  at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
> 2020-06-28T21:37:54.8126961Z  at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
> 2020-06-28T21:37:54.8127766Z  at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
> 2020-06-28T21:37:54.8128570Z  at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185)
> 2020-06-28T21:37:54.8129140Z  at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179)
> 2020-06-28T21:37:54.8129686Z  at 
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:506)
> 2020-06-28T21:37:54.8130174Z  at 
>

[jira] [Updated] (FLINK-18444) KafkaITCase failing with "Failed to send data to Kafka: This server does not host this topic-partition"



 [ 
https://issues.apache.org/jira/browse/FLINK-18444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guowei Ma updated FLINK-18444:
--
Affects Version/s: (was: 1.11.0)
   1.11.3

> KafkaITCase failing with "Failed to send data to Kafka: This server does not 
> host this topic-partition"
> ---
>
> Key: FLINK-18444
> URL: https://issues.apache.org/jira/browse/FLINK-18444
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.3, 1.12.0
>Reporter: Dian Fu
>Priority: Critical
>  Labels: stale-critical, test-stability
>
> Instance on master: 
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4092=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f
> {code}
> 2020-06-28T21:37:54.8113215Z [ERROR] 
> testMultipleSourcesOnePartition(org.apache.flink.streaming.connectors.kafka.KafkaITCase)
>   Time elapsed: 5.079 s  <<< ERROR!
> 2020-06-28T21:37:54.8113885Z 
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2020-06-28T21:37:54.8114418Z  at 
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> 2020-06-28T21:37:54.8114905Z  at 
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:677)
> 2020-06-28T21:37:54.8115397Z  at 
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:81)
> 2020-06-28T21:37:54.8116254Z  at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1699)
> 2020-06-28T21:37:54.8116857Z  at 
> org.apache.flink.streaming.connectors.kafka.testutils.DataGenerators.generateRandomizedIntegerSequence(DataGenerators.java:120)
> 2020-06-28T21:37:54.8117715Z  at 
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:933)
> 2020-06-28T21:37:54.8118327Z  at 
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testMultipleSourcesOnePartition(KafkaITCase.java:107)
> 2020-06-28T21:37:54.8118805Z  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-06-28T21:37:54.8119859Z  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-06-28T21:37:54.8120861Z  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-06-28T21:37:54.8121436Z  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2020-06-28T21:37:54.8121899Z  at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-06-28T21:37:54.8122424Z  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-06-28T21:37:54.8122942Z  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-06-28T21:37:54.8123406Z  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-06-28T21:37:54.8123899Z  at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2020-06-28T21:37:54.8124507Z  at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2020-06-28T21:37:54.8124978Z  at 
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 2020-06-28T21:37:54.8125332Z  at 
> java.base/java.lang.Thread.run(Thread.java:834)
> 2020-06-28T21:37:54.8125743Z Caused by: 
> org.apache.flink.runtime.JobException: Recovery is suppressed by 
> NoRestartBackoffTimeStrategy
> 2020-06-28T21:37:54.8126305Z  at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
> 2020-06-28T21:37:54.8126961Z  at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
> 2020-06-28T21:37:54.8127766Z  at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
> 2020-06-28T21:37:54.8128570Z  at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185)
> 2020-06-28T21:37:54.8129140Z  at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179)
> 2020-06-28T21:37:54.8129686Z  at 
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:506)
> 2020-06-28T21:37:54.8130174Z  at 
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:386)
> 2020-06-28T21:37:54.8130717Z  at 
>

[jira] [Resolved] (FLINK-19879) Support "IF NOT EXISTS" in DDL for CREATE TABLE

2021-04-27 Thread Leonard Xu (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-19879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Leonard Xu resolved FLINK-19879.

Fix Version/s: 1.12.0
   Resolution: Fixed

> Support "IF NOT EXISTS" in DDL for CREATE TABLE
> ---
>
> Key: FLINK-19879
> URL: https://issues.apache.org/jira/browse/FLINK-19879
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / API
>Affects Versions: 1.11.2
>Reporter: Ingo Bürk
>Priority: Minor
>  Labels: stale-minor
> Fix For: 1.12.0
>
>
> Catalog#createTable has a ignoreIfExists flag, but the corresponding DDL does 
> not yet seem to support "IF NOT EXISTS"¹. It would be nice if this could be 
> made available in SQL as well.
>  
> ¹ 
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] YikSanChan commented on pull request #15790: [hotfix][python][docs] add bundling udfs section



YikSanChan commented on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828094763


   @dianfu added zh docs as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] leonardBang removed a comment on pull request #15785: [hotfix][doc] Fix invalid sql comments and words in timezone.md



leonardBang removed a comment on pull request #15785:
URL: https://github.com/apache/flink/pull/15785#issuecomment-828076672


   Thanks @hackergin for the contribution, could we fix this one another 
translation PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] leonardBang commented on a change in pull request #15753: [FLINK-22428][docs][table] Translate timezone page into Chinese



leonardBang commented on a change in pull request #15753:
URL: https://github.com/apache/flink/pull/15753#discussion_r621757583



##
File path: docs/content.zh/docs/dev/table/concepts/timezone.md
##
@@ -23,29 +23,29 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-## Overview
+## 概述
 
-Flink provides rich data types for Date and Time, including `DATE`, `TIME`, 
`TIMESTAMP`, `TIMESTAMP_LTZ`, `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO 
SECOND` (please see [Date and Time]({{< ref "docs/dev/table/types" 
>}}#date-and-time) for detailed information).
-Flink supports setting time zone in session level (please see 
[table.local-time-zone]({{< ref 
"docs/dev/table/config">}}#table-local-time-zone) for detailed information).
-These timestamp data types and time zone support of Flink make it easy to 
process business data across time zones.
+Flink 为日期和时间提供了丰富的数据类型， 包括 `DATE`， `TIME`， `TIMESTAMP`， `TIMESTAMP_LTZ`， 
`INTERVAL YEAR TO MONTH`， `INTERVAL DAY TO SECOND` (更多详情请参考 [Date and Time]({{< 
ref "docs/dev/table/types" >}}#date-and-time))。
+Flink 支持在 session （会话）级别设置时区（更多详情请参考 [table.local-time-zone]({{< ref 
"docs/dev/table/config">}}#table-local-time-zone)）。
+Flink 对多种时间类型和时区的支持使得跨时区的数据处理变得非常容易。
 
 ## TIMESTAMP vs TIMESTAMP_LTZ
 
-### TIMESTAMP type
- - `TIMESTAMP(p)` is an abbreviation for `TIMESTAMP(p) WITHOUT TIME ZONE`, the 
precision `p` supports range is from 0 to 9, 6 by default.
- - `TIMESTAMP` describes a timestamp represents year, month, day, hour, 
minute, second and fractional seconds.
- - `TIMESTAMP` can be specified from a string literal, e.g.
+### TIMESTAMP 类型
+ - `TIMESTAMP(p)` 是 `TIMESTAMP(p) WITHOUT TIME ZONE` 的简写， 精度 `p` 支持的范围是0-9， 
默认是6。
+ - `TIMESTAMP` 用于描述年， 月， 日， 小时， 分钟， 秒 和 小数秒对应的时间戳。
+ - `TIMESTAMP` 可以通过一个字符串来指定，例如：
  ```sql
 Flink SQL> SELECT TIMESTAMP '1970-01-01 00:00:04.001';
 +-+
 | 1970-01-01 00:00:04.001 |
 +-+
 ```
 
-### TIMESTAMP_LTZ type
- - `TIMESTAMP_LTZ(p)` is an abbreviation for `TIMESTAMP(p) WITH LOCAL TIME 
ZONE`, the precision `p` supports range is from 0 to 9, 6 by default.
- - `TIMESTAMP_LTZ` describes an absolute time point on the time-line, it 
stores a long value representing epoch-milliseconds and an int representing 
nanosecond-of-millisecond. The epoch time is measured from the standard Java 
epoch of `1970-01-01T00:00:00Z`. Every datum of `TIMESTAMP_LTZ` type is 
interpreted in the local time zone configured in the current session for 
computation and visualization.
- - `TIMESTAMP_LTZ` has no literal representation and thus can not specify from 
literal, it can derives from a long epoch time(e.g. The long time produced by 
Java `System.currentTimeMillis()`)
+### TIMESTAMP_LTZ 类型
+ - `TIMESTAMP_LTZ(p)` 是 `TIMESTAMP(p) WITH LOCAL TIME ZONE` 的简写， 精度 `p` 
支持的范围是0-9， 默认是6。
+ - `TIMESTAMP_LTZ` 用于描述时间线上的绝对时间点， 使用 long 保存从 epoch 至今的毫秒数， 使用int保存毫秒中的纳秒数。 
epoch 时间是从 java 的标准 epoch 时间 `1970-01-01T00:00:00Z` 开始计算。 在计算和可视化时， 每个 
`TIMESTAMP_LTZ` 类型的数据都是使用的 session （会话）中配置的时区。
+ - `TIMESTAMP_LTZ` 没有字符串表达形式因此无法通过字符串来指定， 可以通过一个 long 类型的 epoch 时间来转化(例如: 
通过Java来产生一个long类型epoch时间 `System.currentTimeMillis()`)

Review comment:
   ```suggestion
- `TIMESTAMP_LTZ` 没有字符串表达形式因此无法通过字符串来指定， 可以通过一个 long 类型的 epoch 时间来转化(例如: 通过 
Java 来产生一个 long 类型的 epoch 时间 `System.currentTimeMillis()`)
   ```

##
File path: docs/content.zh/docs/dev/table/concepts/timezone.md
##
@@ -539,35 +538,34 @@ Returns the different window start, window end and window 
rowtime compared to ca
 
+-+-+-+--+---+
 ```
 
-## Daylight Saving Time Support
-Flink SQL supports defining time attributes on TIMESTAMP_LTZ column, base on 
this, Flink SQL gracefully uses TIMESTAMP and TIMESTAMP_LTZ type in window 
processing to support the Daylight Saving Time.
+## 夏令时支持
+Flink SQL支持在 `TIMESTAMP_LTZ`列上定义时间属性， 基于这一特征，Flink SQL 在窗口中使用 `TIMESTAMP` 和 
`TIMESTAMP_LTZ` 类型优雅地支持了夏令时。

-  
-Flink use timestamp literal to split the window and assigns window to data 
according to the epoch time of the each row. It means Flink uses `TIMESTAMP` 
type for window start and window end (e.g. `TUMBLE_START` and `TUMBLE_END`), 
uses `TIMESTAMP_LTZ` for window time attribute (e.g. `TUMBLE_PROCTIME`, 
`TUMBLE_ROWTIME`).
-Given a example of tumble window, the DaylightTime in Los_Angele starts at 
time 2021-03-14 02:00:00:
+Flink 使用时间的字符格式来分割窗口并通过 row 的 epoch 时间来分配窗口。 这意味着 Flink 窗口开始时间和窗口结束时间使用的是 
`TIMESTAMP` 类型（例如: `TUMBLE_START` 和 `TUMBLE_END`）， 将 `TIMESTAMP_LTZ` 
类型用于窗口的时间属性（例如: `TUMBLE_PROCTIME`， `TUMBLE_ROWTIME`）。
+给定一个tumble window示例， 在Los_Angele时区下夏令时从 `2021-03-14 02:00:00` 开始：
 ```
 long epoch1 = 161570880L; // 2021-03-14 00:00:00
 long epoch2 = 161571240L; // 2021-03-14 01:00:00
 long epoch3 = 161571600L; // 2021-03-14 03:00:00, skip one hour 
(2021-03-14 02:00:00)
 long epoch4 = 161571960L; // 2021-03-14 04:00:00

[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement

2021-04-27 Thread Jark Wu (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334405#comment-17334405
 ] 

Jark Wu commented on FLINK-22318:
-

[~todd5167] 
1. The {{SqlValidatorException}} may be not because of non-existent fields, 
maybe invalid expression/udfs, etc... If we want to throw an accurate 
exception, a better solution can be using 
{{org.apache.flink.table.catalog.DefaultSchemaResolver#resolve}} to resolve a 
{{Schema}}, and then you should be able to traverse the resolved expression 
tree to see whether it contains renamed column.  
2. I'm fine with this. 

Besides, we can continue the disucssion in PR, don't need to swtich between 
JIRA and PR. 

> Support RENAME column name for ALTER TABLE statement
> 
>
> Key: FLINK-22318
> URL: https://issues.apache.org/jira/browse/FLINK-22318
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / API
>Reporter: Jark Wu
>Priority: Major
>  Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #11786: [FLINK-17208][network] Reduce redundant data available notifications of PipelinedSubpartition



flinkbot edited a comment on pull request #11786:
URL: https://github.com/apache/flink/pull/11786#issuecomment-615087816


   
   ## CI report:
   
   * f5b89a2a4df3cb5c31ac5297359dddbb5d32282b Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5032)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] YikSanChan commented on pull request #15790: [hotfix][python][docs] add bundling udfs section



YikSanChan commented on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828090279


   @dianfu Resolved


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-22380) Pub/Sub Lite Connector for Flink

2021-04-27 Thread Evan Palmer (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evan Palmer updated FLINK-22380:

Component/s: Connectors / Common

> Pub/Sub Lite Connector for Flink
> 
>
> Key: FLINK-22380
> URL: https://issues.apache.org/jira/browse/FLINK-22380
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors / Common, Connectors / Google Cloud PubSub
>Reporter: Evan Palmer
>Priority: Major
>
> Hello,
> I'm an engineer at Google working on [Pub/Sub 
> Lite|https://cloud.google.com/pubsub/lite/docs]. Pub/Sub Lite is zonal 
> partition based messaging service which is meant to be a cheaper alternative 
> to Cloud Pub/Sub.
> We're interesting in writing a Flink connector so that users can read/write 
> to Pub/Sub Lite in flink pipelines. I'm wondering if we can contribute this 
> connector to the flink repo. Perhaps somewhere near 
> [https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-gcp-pubsub|https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-gcp-pubsub.].
>  
> Thanks!
> Evan
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #15121: The method $(String) is undefined for the type TableExample



flinkbot edited a comment on pull request #15121:
URL: https://github.com/apache/flink/pull/15121#issuecomment-793480240


   
   ## CI report:
   
   * cf26ce895a7956d258acc073817f578558e78227 UNKNOWN
   * 709c4110370b845f42d916a06e56c6026cf2fac8 UNKNOWN
   * 4ea99332e1997eebca1f3f0a9d9229b8265fe32c UNKNOWN
   * 9a256ec6b644a0f795376076f4e3ff4e85aaad1e UNKNOWN
   * b7dca63a2e98f09c8c780a21091b5268d897b220 UNKNOWN
   * 2b7bb05a4dbbb0883db566589dfe1975fdbb279d UNKNOWN
   * 23ab25410bbff3d717302c9a2549d05ab2aae330 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17305)
 
   * 60b3b4f14b78889b827c5e573b1d4ef992791d32 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] dianfu commented on a change in pull request #15790: [hotfix][python][docs] add bundling udfs section



dianfu commented on a change in pull request #15790:
URL: https://github.com/apache/flink/pull/15790#discussion_r621750740



##
File path: docs/content/docs/dev/python/table/udfs/python_udfs.md
##
@@ -30,6 +30,10 @@ User-defined functions are important features, because they 
significantly extend
 
 **NOTE:** Python UDF execution requires Python version (3.6, 3.7 or 3.8) with 
PyFlink installed. It's required on both the client side and the cluster side. 
 
+## Bundling UDFs

Review comment:
   What about move to the end of this file?

##
File path: docs/content/docs/dev/python/table/udfs/python_udfs.md
##
@@ -30,6 +30,10 @@ User-defined functions are important features, because they 
significantly extend
 
 **NOTE:** Python UDF execution requires Python version (3.6, 3.7 or 3.8) with 
PyFlink installed. It's required on both the client side and the cluster side. 
 
+## Bundling UDFs
+
+**NOTE:** To run Python UDFs (as well as Pandas UDF) in any non-local mode, it 
is strongly recommended to bundle your UDF definitions using the config option 
[`python-files`]({{< ref "docs/dev/python/python_config" >}}#python-files), if 
your UDFs live outside of the file where the `main()` function is defined. 
Otherwise, you may run into `ModuleNotFoundError: No module named 'my_udf'` if 
you define UDFs in a file called `my_udf.py`.

Review comment:
   ```suggestion
   To run Python UDFs (as well as Pandas UDFs) in any non-local mode, it is 
strongly recommended to bundle your Python UDF definitions using the config 
option [`python-files`]({{< ref "docs/dev/python/python_config" 
>}}#python-files), if your Python UDFs live outside of the file where the 
`main()` function is defined. Otherwise, you may run into `ModuleNotFoundError: 
No module named 'my_udf'` if you define Python UDFs in a file called 
`my_udf.py`.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] cshuo commented on pull request #15745: [FLINK-22304][table] Refactor some interfaces for TVF based window to…



cshuo commented on pull request #15745:
URL: https://github.com/apache/flink/pull/15745#issuecomment-828081952


   The failed test is python docs, which is already fixed. I will rebase and 
trigger test again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-18952) Add 10 minutes to DataStream API documentation

2021-04-27 Thread Dian Fu (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu closed FLINK-18952.
---
Fix Version/s: 1.12.4
   Resolution: Fixed

Merged to
- master via 981bed0706005e1c02975733141da69a7260f58c
- release-1.13 via 07faab241efe7c2203fdc428dfde6633ae2c59ff
- release-1.12 via 60b3b4f14b78889b827c5e573b1d4ef992791d32

> Add 10 minutes to DataStream API documentation
> --
>
> Key: FLINK-18952
> URL: https://issues.apache.org/jira/browse/FLINK-18952
> Project: Flink
>  Issue Type: Sub-task
>  Components: API / Python
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>  Labels: pull-request-available, stale-assigned
> Fix For: 1.13.0, 1.12.4
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-18952) Add 10 minutes to DataStream API documentation

2021-04-27 Thread Dian Fu (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu updated FLINK-18952:

Fix Version/s: (was: 1.14.0)
   1.13.0

> Add 10 minutes to DataStream API documentation
> --
>
> Key: FLINK-18952
> URL: https://issues.apache.org/jira/browse/FLINK-18952
> Project: Flink
>  Issue Type: Sub-task
>  Components: API / Python
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>  Labels: pull-request-available, stale-assigned
> Fix For: 1.13.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] leonardBang commented on a change in pull request #15753: [FLINK-22428][docs][table] Translate timezone page into Chinese



leonardBang commented on a change in pull request #15753:
URL: https://github.com/apache/flink/pull/15753#discussion_r621748251



##
File path: docs/content.zh/docs/dev/table/concepts/timezone.md
##
@@ -528,7 +527,7 @@ Flink SQL> SET table.local-time-zone=Asia/Shanghai;
 Flink SQL> SELECT * FROM MyView5;
 ```
 
-Returns the different window start, window end and window rowtime compared to 
calculation in UTC timezone.   
+返回和 UTC 时区下计算时的不同的窗口开始时间， 窗口结束时间和窗口的 rowtime 。

Review comment:
   ```suggestion
   相比在 UTC 时区下的计算结果，在 Asia/Shanghai 时区计算的窗口开始时间， 窗口结束时间和窗口的 rowtime 是不同的。
   ```
   相似的这几句，我们这样翻译会好点吗？




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] leonardBang commented on pull request #15785: [hotfix][doc] Fix invalid sql comments and words in timezone.md



leonardBang commented on pull request #15785:
URL: https://github.com/apache/flink/pull/15785#issuecomment-828076672


   Thanks @hackergin for the contribution, could we fix this one another 
translation PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15790: [hotfix][python][docs] add bundling udfs section



flinkbot edited a comment on pull request #15790:
URL: https://github.com/apache/flink/pull/15790#issuecomment-828062007


   
   ## CI report:
   
   * 40ff67a835fa6304cf3fe72e642e4ac6a3af76a8 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17319)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #15783: [hotfix][python][docs] Fix python.archives docs



flinkbot edited a comment on pull request #15783:
URL: https://github.com/apache/flink/pull/15783#issuecomment-827482345


   
   ## CI report:
   
   * 4db5f1c0dce58bd0ee5aaa1d8fca4d60635e90c8 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17290)
 
   * f913eb003f86af890e05edfcddb00c2c65b2aca1 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17318)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] YikSanChan opened a new pull request #15790: [hotfix][python][docs] add bundling udfs section



YikSanChan opened a new pull request #15790:
URL: https://github.com/apache/flink/pull/15790


   
   
   ## What is the purpose of the change
   
   Remind people that they need to include their udf definition in python.files 
config option. Follow up to 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ModuleNotFound-when-loading-udf-from-another-py-file-td43275.html
   
   ## Brief change log
   
   Add docs.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? not applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-10954) Hardlink from files of previous local stored state might cross devices

2021-04-27 Thread Dan Hill (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334369#comment-17334369
 ] 

Dan Hill commented on FLINK-10954:
--

[~yunta]- thanks Yun!

> Hardlink from files of previous local stored state might cross devices
> --
>
> Key: FLINK-10954
> URL: https://issues.apache.org/jira/browse/FLINK-10954
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Affects Versions: 1.6.2
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Critical
>
> Currently, local recovery's base directories is initialized from 
> '{{io.tmp.dirs}}' if parameter '{{taskmanager.state.local.root-dirs}}' is not 
> set. For Yarn environment, the tmp dirs is replaced by its '{{LOCAL_DIRS}}', 
> which might consist of directories from different devices, such as 
> /dump/1/nm-local-dir, /dump/2/nm-local-dir. The local directory for RocksDB 
> is initialized from IOManager's spillingDirectories, which might located in 
> different device from local recovery's folder. However, hard-link between 
> different devices is not allowed, it will throw exception below:
> {code:java}
> java.nio.file.FileSystemException: target -> souce: Invalid cross-device link
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-6605) Allow users to specify a default name for processing time in Table API



 [ 
https://issues.apache.org/jira/browse/FLINK-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot reassigned FLINK-6605:
-

Assignee: (was: Haohui Mai)

> Allow users to specify a default name for processing time in Table API
> --
>
> Key: FLINK-6605
> URL: https://issues.apache.org/jira/browse/FLINK-6605
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / API
>Reporter: Haohui Mai
>Priority: Major
>  Labels: auto-unassigned
>
> FLINK-5884 enables users to specify column names for both processing time and 
> event time. FLINK-6595 and FLINK-6584 breaks as chained / nested queries will 
> no longer have an attribute of processing time / event time.
> This jira proposes to add a default name for the processing time in order to 
> unbreak FLINK-6595 and FLINK-6584.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-7016) Move inputFormat to InputFormatVertex from TaskConfig



 [ 
https://issues.apache.org/jira/browse/FLINK-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot reassigned FLINK-7016:
-

Assignee: (was: Xu Pingyong)

> Move inputFormat to InputFormatVertex from TaskConfig
> -
>
> Key: FLINK-7016
> URL: https://issues.apache.org/jira/browse/FLINK-7016
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataSet, API / DataStream
>Reporter: Xu Pingyong
>Priority: Major
>  Labels: auto-unassigned
>
> On batch case, InputFormat is put into TaskConfig, batch task gets it to read 
> data and job manager uses it to split splits from TaskConfig.
> On streaming case, all configs are put into StreamConfig, but this 
> inputFormat is put into TaskConfig.
> We can put InputFormat into InputFormatVertex, and batch task still gets 
> InputFormat from TaskConfig. It will be clear.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-6477) The first time to click Taskmanager cannot get the actual data



 [ 
https://issues.apache.org/jira/browse/FLINK-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot reassigned FLINK-6477:
-

Assignee: (was: zhihao chen)

> The first time to click Taskmanager cannot get the actual data
> --
>
> Key: FLINK-6477
> URL: https://issues.apache.org/jira/browse/FLINK-6477
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Web Frontend
>Affects Versions: 1.2.0
>Reporter: zhihao chen
>Priority: Major
>  Labels: auto-unassigned, pull-request-available
> Attachments: errDisplay.jpg
>
>
> Flink web page first click Taskmanager to get less than the actual data, when 
> the parameter “jobmanager.web.refresh-interval” is set to a larger value, eg: 
> 180, if you do not manually refresh the page you need to wait time after 
> the timeout normal display



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-22344) KafkaTableITCase.testKafkaSourceSinkWithKeyAndFullValue times out



[ 
https://issues.apache.org/jira/browse/FLINK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334361#comment-17334361
 ] 

Flink Jira Bot commented on FLINK-22344:


This critical issue is unassigned and itself and all of its Sub-Tasks have not 
been updated for 7 days. So, it has been labeled "stale-critical". If this 
ticket is indeed critical, please either assign yourself or give an update. 
Afterwards, please remove the label. In 7 days the issue will be deprioritized.

> KafkaTableITCase.testKafkaSourceSinkWithKeyAndFullValue times out
> -
>
> Key: FLINK-22344
> URL: https://issues.apache.org/jira/browse/FLINK-22344
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
>  Labels: stale-critical, test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16731=logs=72d4811f-9f0d-5fd0-014a-0bc26b72b642=c1d93a6a-ba91-515d-3196-2ee8019fbda7=6619
> {code}
> Apr 18 21:49:28 [ERROR] testKafkaSourceSinkWithKeyAndFullValue[legacy = 
> false, format = 
> csv](org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase)  
> Time elapsed: 30.121 s  <<< ERROR!
> Apr 18 21:49:28 org.junit.runners.model.TestTimedOutException: test timed out 
> after 30 seconds
> Apr 18 21:49:28   at java.lang.Thread.sleep(Native Method)
> Apr 18 21:49:28   at 
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> Apr 18 21:49:28   at 
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> Apr 18 21:49:28   at 
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> Apr 18 21:49:28   at 
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> Apr 18 21:49:28   at 
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:351)
> Apr 18 21:49:28   at 
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.collectRows(KafkaTableTestUtils.java:52)
> Apr 18 21:49:28   at 
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testKafkaSourceSinkWithKeyAndFullValue(KafkaTableITCase.java:538)
> Apr 18 21:49:28   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
> Apr 18 21:49:28   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 18 21:49:28   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 18 21:49:28   at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 18 21:49:28   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Apr 18 21:49:28   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Apr 18 21:49:28   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Apr 18 21:49:28   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Apr 18 21:49:28   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Apr 18 21:49:28   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> Apr 18 21:49:28   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Apr 18 21:49:28   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Apr 18 21:49:28   at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Apr 18 21:49:28   at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-19705) Avro Confluent Schema Registry nightly end-to-end test: curl: (56) Recv failure: Connection reset by peer



[ 
https://issues.apache.org/jira/browse/FLINK-19705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334356#comment-17334356
 ] 

Flink Jira Bot commented on FLINK-19705:


This major issue is unassigned and itself and all of its Sub-Tasks have not 
been updated for 30 days. So, it has been labeled "stale-major". If this ticket 
is indeed "major", please either assign yourself or give an update. Afterwards, 
please remove the label. In 7 days the issue will be deprioritized.

> Avro Confluent Schema Registry nightly end-to-end test: curl: (56) Recv 
> failure: Connection reset by peer
> -
>
> Key: FLINK-19705
> URL: https://issues.apache.org/jira/browse/FLINK-19705
> Project: Flink
>  Issue Type: Bug
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7826=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> 2020-10-19T06:08:30.2781766Z Downloading confluent from 
> http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz
> 2020-10-19T06:08:30.2849354Z   % Total% Received % Xferd  Average Speed   
> TimeTime Time  Current
> 2020-10-19T06:08:30.2850032Z  Dload  Upload   
> Total   SpentLeft  Speed
> 2020-10-19T06:08:30.2850669Z 
> 2020-10-19T06:08:30.3457235Z   0 00 00 0  0  0 
> --:--:-- --:--:-- --:--:-- 0
> 2020-10-19T06:08:31.3295810Z   0 00 00 0  0  0 
> --:--:-- --:--:-- --:--:-- 0
> 2020-10-19T06:08:32.3309796Z  22  296M   22 67.1M0 0  64.1M  0  
> 0:00:04  0:00:01  0:00:03 64.1M
> 2020-10-19T06:08:33.3293737Z  47  296M   47  141M0 0  69.1M  0  
> 0:00:04  0:00:02  0:00:02 69.1M
> 2020-10-19T06:08:33.8972228Z  78  296M   78  232M0 0  76.1M  0  
> 0:00:03  0:00:03 --:--:-- 76.1M
> 2020-10-19T06:08:33.8973866Z  97  296M   97  290M0 0  80.3M  0  
> 0:00:03  0:00:03 --:--:-- 80.3M
> 2020-10-19T06:08:33.8974595Z curl: (56) Recv failure: Connection reset by peer
> 2020-10-19T06:08:33.9000256Z Stopping job timeout watchdog (with pid=111423)
> 2020-10-19T06:08:33.9007709Z 
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/kafka-common.sh: line 
> 209: 
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-26263816822/confluent-5.0.0/bin/schema-registry-stop:
>  No such file or directory
> 2020-10-19T06:08:33.9010767Z [FAIL] Test script contains errors.
> 2020-10-19T06:08:33.9021624Z Checking for errors...
> 2020-10-19T06:08:33.9231098Z No errors in log files.
> 2020-10-19T06:08:33.9231651Z Checking for exceptions...
> 2020-10-19T06:08:33.9470201Z No exceptions in log files.
> 2020-10-19T06:08:33.9471568Z Checking for non-empty .out files...
> 2020-10-19T06:08:33.9489148Z grep: 
> /home/vsts/work/1/s/flink-dist/target/flink-1.12-SNAPSHOT-bin/flink-1.12-SNAPSHOT/log/*.out:
>  No such file or directory
> 2020-10-19T06:08:33.9493300Z No non-empty .out files.
> 2020-10-19T06:08:33.9493708Z 
> 2020-10-19T06:08:33.9494586Z [FAIL] 'Avro Confluent Schema Registry nightly 
> end-to-end test' failed after 0 minutes and 7 seconds! Test exited with exit 
> code 1
> 2020-10-19T06:08:33.9495050Z 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-3857) Add reconnect attempt to Elasticsearch host



 [ 
https://issues.apache.org/jira/browse/FLINK-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot reassigned FLINK-3857:
-

Assignee: (was: Subhobrata Dey)

> Add reconnect attempt to Elasticsearch host
> ---
>
> Key: FLINK-3857
> URL: https://issues.apache.org/jira/browse/FLINK-3857
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common
>Affects Versions: 1.0.2, 1.1.0
>Reporter: Fabian Hueske
>Priority: Major
>  Labels: auto-unassigned, pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, the connection to the Elasticsearch host is opened in 
> {{ElasticsearchSink.open()}}. In case the connection is lost (maybe due to a 
> changed DNS entry), the sink fails.
> I propose to catch the Exception for lost connections in the {{invoke()}} 
> method and try to re-open the connection for a configurable number of times 
> with a certain delay.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-4045) Test AMQP connector using Apache ActiveMQ emebdded broker



[ 
https://issues.apache.org/jira/browse/FLINK-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334320#comment-17334320
 ] 

Flink Jira Bot commented on FLINK-4045:
---

This issue was marked "stale-assigned" and has not received an update in 7 
days. It is now automatically unassigned. If you are still working on it, you 
can assign it to yourself again. Please also give an update about the status of 
the work.

> Test AMQP connector using Apache ActiveMQ emebdded broker
> -
>
> Key: FLINK-4045
> URL: https://issues.apache.org/jira/browse/FLINK-4045
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Robert Metzger
>Assignee: Subhankar Biswas
>Priority: Major
>  Labels: stale-assigned
>
> Currently, our (RabbitMQ) AMQP client is not tested in any integration tests.
> Apache ActiveMQ implements an AMQP broker and they provide an embedded 
> implementation we can use for the integration tests: 
> http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-21993) Remove LocalCollectionOutputFormat from code base



[ 
https://issues.apache.org/jira/browse/FLINK-21993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334342#comment-17334342
 ] 

Flink Jira Bot commented on FLINK-21993:


This major issue is unassigned and itself and all of its Sub-Tasks have not 
been updated for 30 days. So, it has been labeled "stale-major". If this ticket 
is indeed "major", please either assign yourself or give an update. Afterwards, 
please remove the label. In 7 days the issue will be deprioritized.

> Remove LocalCollectionOutputFormat from code base
> -
>
> Key: FLINK-21993
> URL: https://issues.apache.org/jira/browse/FLINK-21993
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataSet, Library / Graph Processing (Gelly), Table 
> SQL / Planner
>Reporter: Till Rohrmann
>Priority: Major
>  Labels: stale-major
>
> The {{LocalCollectionOutputFormat}} is not safe wrt to failovers. The problem 
> is that it records also partial results from previous attempts. I would 
> suggest to remove this class. See FLINK-21609 for an example problem.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-6346) Migrate from Java serialization for GenericWriteAheadSink's state



 [ 
https://issues.apache.org/jira/browse/FLINK-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot updated FLINK-6346:
--
Labels: auto-unassigned  (was: stale-assigned)

> Migrate from Java serialization for GenericWriteAheadSink's state
> -
>
> Key: FLINK-6346
> URL: https://issues.apache.org/jira/browse/FLINK-6346
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Fang Yong
>Priority: Major
>  Labels: auto-unassigned
>
> See umbrella JIRA FLINK-6343 for details. This subtask tracks the migration 
> for {{GenericWriteAheadSink}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-6931) Support custom compression formats for checkpoints (+Upgrade/Compatibility)



 [ 
https://issues.apache.org/jira/browse/FLINK-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot updated FLINK-6931:
--
Labels: auto-unassigned  (was: stale-assigned)

> Support custom compression formats for checkpoints (+Upgrade/Compatibility)
> ---
>
> Key: FLINK-6931
> URL: https://issues.apache.org/jira/browse/FLINK-6931
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Reporter: Stefan Richter
>Assignee: Yun Tang
>Priority: Major
>  Labels: auto-unassigned
>
> With FLINK-6773, we introduced optional snappy compression for keyed state in 
> full checkpoints and savepoints. We should offer users a way to register 
> their own compression formats with the {{ExecutionConfig}}. For this, we 
> should also have a compatibility story, very similar to what 
> {{TypeSerializerConfigSnapshot}} doesfor type serializers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-22355) Simple Task Manager Memory Model image does not show up



[ 
https://issues.apache.org/jira/browse/FLINK-22355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334360#comment-17334360
 ] 

Flink Jira Bot commented on FLINK-22355:


This critical issue is unassigned and itself and all of its Sub-Tasks have not 
been updated for 7 days. So, it has been labeled "stale-critical". If this 
ticket is indeed critical, please either assign yourself or give an update. 
Afterwards, please remove the label. In 7 days the issue will be deprioritized.

> Simple Task Manager Memory Model image does not show up
> ---
>
> Key: FLINK-22355
> URL: https://issues.apache.org/jira/browse/FLINK-22355
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
>  Labels: stale-critical, starter
> Fix For: 1.13.0
>
> Attachments: gnome-shell-screenshot-AMVR10.png
>
>
> An image does not show up in 
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_setup_tm/
>  after migrating to Hugo.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-19879) Support "IF NOT EXISTS" in DDL for CREATE TABLE



[ 
https://issues.apache.org/jira/browse/FLINK-19879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334341#comment-17334341
 ] 

Flink Jira Bot commented on FLINK-19879:


This issue and all of its Sub-Tasks have not been updated for 180 days. So, it 
has been labeled "stale-minor". If you are still affected by this bug or are 
still interested in this issue, please give an update and remove the label. In 
7 days the issue will be closed automatically.

> Support "IF NOT EXISTS" in DDL for CREATE TABLE
> ---
>
> Key: FLINK-19879
> URL: https://issues.apache.org/jira/browse/FLINK-19879
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / API
>Affects Versions: 1.11.2
>Reporter: Ingo Bürk
>Priority: Minor
>  Labels: stale-minor
>
> Catalog#createTable has a ignoreIfExists flag, but the corresponding DDL does 
> not yet seem to support "IF NOT EXISTS"¹. It would be nice if this could be 
> made available in SQL as well.
>  
> ¹ 
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-11170) Support DISTINCT aggregates in MATCH_RECOGNIZE



 [ 
https://issues.apache.org/jira/browse/FLINK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot reassigned FLINK-11170:
--

Assignee: (was: Hequn Cheng)

> Support DISTINCT aggregates in MATCH_RECOGNIZE
> --
>
> Key: FLINK-11170
> URL: https://issues.apache.org/jira/browse/FLINK-11170
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / API
>Affects Versions: 1.8.0
>Reporter: Dawid Wysakowicz
>Priority: Major
>  Labels: auto-unassigned
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-7607) Web Frontend Hangs with Large Numbers of Tasks



[ 
https://issues.apache.org/jira/browse/FLINK-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334226#comment-17334226
 ] 

Flink Jira Bot commented on FLINK-7607:
---

This issue was marked "stale-assigned" and has not received an update in 7 
days. It is now automatically unassigned. If you are still working on it, you 
can assign it to yourself again. Please also give an update about the status of 
the work.

> Web Frontend Hangs with Large Numbers of Tasks
> --
>
> Key: FLINK-7607
> URL: https://issues.apache.org/jira/browse/FLINK-7607
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Web Frontend
>Affects Versions: 1.3.2
> Environment: Attempted to load the web frontend on a MacBook Pro 15" 
> (late 2016) with 16 GB of memory using both Chrome 60.0 and Safari 10.1.2.
>Reporter: Joshua Griffith
>Assignee: Steven Langbroek
>Priority: Major
>  Labels: performance, stale-assigned
>
> Viewing a job with a high number of tasks in the web front-end causes the 
> page to hang, consuming 100% CPU on a core. At 200 tasks the page slows 
> noticeably and scrolling results in long, non-responsive pauses. At 400 tasks 
> the page only updates once per minute and is almost entirely non-responsive.
> Initially, I thought this was caused by rendering a complex job graph but 
> opening the inspector and deleting the canvas did not improve page 
> performance. Further inspection indicated that the page was redrawing every 
> DOM element in the task list on every update.
> A possible solution is to use an approach similar to 
> [react-list|https://github.com/orgsync/react-list] and only request 
> data/render list items that are in view and only update DOM nodes that have 
> changed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-7373) Using URLClassLoader to load UDF triggers HepPlanner unexpected ClassNotFoundException



 [ 
https://issues.apache.org/jira/browse/FLINK-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot updated FLINK-7373:
--
Labels: auto-unassigned  (was: stale-assigned)

> Using URLClassLoader to load UDF triggers HepPlanner unexpected 
> ClassNotFoundException
> --
>
> Key: FLINK-7373
> URL: https://issues.apache.org/jira/browse/FLINK-7373
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / API
>Affects Versions: 1.3.1
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>  Labels: auto-unassigned
>
> Using URLClassLoader to load, say from Artifactory, and instantiate UDF 
> instances will cause some Rule failed during runHepPlanner or 
> runVolcanoPlanner.
> One example could add an ITCase in:
> {code:title=flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/batch/table/CalcITCase.scala}
>   @Test
>   def testUserDefinedFunctionDynamicClassloader() {
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tableEnv = TableEnvironment.getTableEnvironment(env, config)
> val jarFileURI = "file://org/apache/flink/table/udf/HelloWorld.jar"
> val udfClassLoader: ClassLoader = new URLClassLoader(List(new 
> URI(jarFileURI).toURL).toArray)
> val clazz = 
> udfClassLoader.loadClass("org.apache.flink.table.udf.HelloWorld")
> val helloWorldUDF: ScalarFunction = 
> clazz.newInstance().asInstanceOf[ScalarFunction]
> tableEnv.registerFunction("helloWorld", helloWorldUDF)
> val table = env.fromElements("a", "b", "c").toTable(tableEnv, 'text)
> val result = table.select("text.helloWorld()")
> val results = result.toDataSet[Row].collect()
> val expected = "Hello World"
> TestBaseUtils.compareResultAsText(results.asJava, expected)
>   }
> {code}
> where
> {code:title=HelloWorld.java}
> package org.apache.flink.table.udf;
> import org.apache.flink.table.functions.ScalarFunction;
> public class HelloWorld extends ScalarFunction {
>   public String eval() {
> return "Hello World";
>   }
> }
> {code}
> This triggers the following Exception:
> {panel:title=Exception}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
>   at 
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.compile(FlatMapRunner.scala:31)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.open(FlatMapRunner.scala:45)
>   at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
>   at 
> org.apache.flink.api.common.operators.base.FlatMapOperatorBase.executeOnCollections(FlatMapOperatorBase.java:62)
> ..
> Caused by: org.codehaus.commons.compiler.CompileException: Line 6, Column 22: 
> Cannot determine simple type name "org"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11672)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6416)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6177)
> ..
> {panel}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-21025) SQLClientHBaseITCase fails when untarring HBase



[ 
https://issues.apache.org/jira/browse/FLINK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334365#comment-17334365
 ] 

Flink Jira Bot commented on FLINK-21025:


This critical issue is unassigned and itself and all of its Sub-Tasks have not 
been updated for 7 days. So, it has been labeled "stale-critical". If this 
ticket is indeed critical, please either assign yourself or give an update. 
Afterwards, please remove the label. In 7 days the issue will be deprioritized.

> SQLClientHBaseITCase fails when untarring HBase
> ---
>
> Key: FLINK-21025
> URL: https://issues.apache.org/jira/browse/FLINK-21025
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / HBase, Table SQL / Client, Tests
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
>  Labels: stale-critical, test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12210=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 908.614 s <<< FAILURE! - in 
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase
> Jan 19 08:19:36 [ERROR] testHBase[1: 
> hbase-version:2.2.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase)  
> Time elapsed: 615.099 s  <<< ERROR!
> Jan 19 08:19:36 java.io.IOException: 
> Jan 19 08:19:36 Process execution failed due error. Error output:
> Jan 19 08:19:36   at 
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:133)
> Jan 19 08:19:36   at 
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:108)
> Jan 19 08:19:36   at 
> org.apache.flink.tests.util.AutoClosableProcess.runBlocking(AutoClosableProcess.java:70)
> Jan 19 08:19:36   at 
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.setupHBaseDist(LocalStandaloneHBaseResource.java:86)
> Jan 19 08:19:36   at 
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.before(LocalStandaloneHBaseResource.java:76)
> Jan 19 08:19:36   at 
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
> Jan 19 08:19:36   at 
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Jan 19 08:19:36   at 
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> Jan 19 08:19:36   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Jan 19 08:19:36   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Jan 19 08:19:36   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Jan 19 08:19:36   at org.junit.runners.Suite.runChild(Suite.java:128)
> Jan 19 08:19:36   at org.junit.runners.Suite.runChild(Suite.java:27)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Jan 19 08:19:36   at 
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Jan 19 08:19:36   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Jan 19 08:19:36   at org.junit.runners.Suite.runChild(Suite.java:128)
> Jan 19 08:19:36   at org.junit.runners.Suite.runChild(Suite.java:27)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36   at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36   at 
>

[jira] [Commented] (FLINK-7373) Using URLClassLoader to load UDF triggers HepPlanner unexpected ClassNotFoundException



[ 
https://issues.apache.org/jira/browse/FLINK-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334238#comment-17334238
 ] 

Flink Jira Bot commented on FLINK-7373:
---

This issue was marked "stale-assigned" and has not received an update in 7 
days. It is now automatically unassigned. If you are still working on it, you 
can assign it to yourself again. Please also give an update about the status of 
the work.

> Using URLClassLoader to load UDF triggers HepPlanner unexpected 
> ClassNotFoundException
> --
>
> Key: FLINK-7373
> URL: https://issues.apache.org/jira/browse/FLINK-7373
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / API
>Affects Versions: 1.3.1
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>  Labels: stale-assigned
>
> Using URLClassLoader to load, say from Artifactory, and instantiate UDF 
> instances will cause some Rule failed during runHepPlanner or 
> runVolcanoPlanner.
> One example could add an ITCase in:
> {code:title=flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/batch/table/CalcITCase.scala}
>   @Test
>   def testUserDefinedFunctionDynamicClassloader() {
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tableEnv = TableEnvironment.getTableEnvironment(env, config)
> val jarFileURI = "file://org/apache/flink/table/udf/HelloWorld.jar"
> val udfClassLoader: ClassLoader = new URLClassLoader(List(new 
> URI(jarFileURI).toURL).toArray)
> val clazz = 
> udfClassLoader.loadClass("org.apache.flink.table.udf.HelloWorld")
> val helloWorldUDF: ScalarFunction = 
> clazz.newInstance().asInstanceOf[ScalarFunction]
> tableEnv.registerFunction("helloWorld", helloWorldUDF)
> val table = env.fromElements("a", "b", "c").toTable(tableEnv, 'text)
> val result = table.select("text.helloWorld()")
> val results = result.toDataSet[Row].collect()
> val expected = "Hello World"
> TestBaseUtils.compareResultAsText(results.asJava, expected)
>   }
> {code}
> where
> {code:title=HelloWorld.java}
> package org.apache.flink.table.udf;
> import org.apache.flink.table.functions.ScalarFunction;
> public class HelloWorld extends ScalarFunction {
>   public String eval() {
> return "Hello World";
>   }
> }
> {code}
> This triggers the following Exception:
> {panel:title=Exception}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
>   at 
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.compile(FlatMapRunner.scala:31)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.open(FlatMapRunner.scala:45)
>   at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
>   at 
> org.apache.flink.api.common.operators.base.FlatMapOperatorBase.executeOnCollections(FlatMapOperatorBase.java:62)
> ..
> Caused by: org.codehaus.commons.compiler.CompileException: Line 6, Column 22: 
> Cannot determine simple type name "org"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11672)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6416)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6177)
> ..
> {panel}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-5865) Throw original exception in states



[ 
https://issues.apache.org/jira/browse/FLINK-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334282#comment-17334282
 ] 

Flink Jira Bot commented on FLINK-5865:
---

This issue was marked "stale-assigned" and has not received an update in 7 
days. It is now automatically unassigned. If you are still working on it, you 
can assign it to yourself again. Please also give an update about the status of 
the work.

> Throw original exception in states
> --
>
> Key: FLINK-5865
> URL: https://issues.apache.org/jira/browse/FLINK-5865
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / State Backends
>Affects Versions: 1.3.0
>Reporter: Xiaogang Shi
>Assignee: Xiaogang Shi
>Priority: Major
>  Labels: pull-request-available, stale-assigned
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Now all exception thrown in RocksDB states are converted to 
> {{RuntimeException}}. It's unnecessary and will print useless stacks in the 
> log.
> I think it's better to throw the original exception, without any wrapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-6370) FileAlreadyExistsException on startup



 [ 
https://issues.apache.org/jira/browse/FLINK-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot reassigned FLINK-6370:
-

Assignee: (was: Chesnay Schepler)

> FileAlreadyExistsException on startup
> -
>
> Key: FLINK-6370
> URL: https://issues.apache.org/jira/browse/FLINK-6370
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Web Frontend
>Affects Versions: 1.2.0
>Reporter: Andrey
>Priority: Major
>  Labels: auto-unassigned
>
> Currently static web resources are lazily cached onto disk during first 
> request. However if 2 concurrent requests will be executed, then 
> FileAlreadyExistsException will be in logs.
> {code}
> 2017-04-24 14:00:58,075 ERROR 
> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler  - error 
> while responding [nioEventLoopGroup-3-2]
> java.nio.file.FileAlreadyExistsException: 
> /flink/web/flink-web-528f8cb8-dd60-433c-8f6c-df49ad0b79e0/index.html
>   at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>   at 
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
>   at 
> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
>   at java.nio.file.Files.newOutputStream(Files.java:216)
>   at java.nio.file.Files.copy(Files.java:3016)
>   at 
> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler.respondAsLeader(StaticFileServerHandler.java:238)
>   at 
> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler.channelRead0(StaticFileServerHandler.java:197)
>   at 
> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler.channelRead0(StaticFileServerHandler.java:99)
>   at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at io.netty.handler.codec.http.router.Handler.routed(Handler.java:62)
> {code}
> Expect: 
> * extract all static resources on startup in main thread and before opening 
> http port.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #15789: [WIP][FLINK-21181][runtime] Wait for Invokable cancellation before releasing network resources