[jira] [Commented] (FLINK-18433) From the end-to-end performance test results, 1.11 has a regression
[ https://issues.apache.org/jira/browse/FLINK-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147221#comment-17147221 ] Aihua Li commented on FLINK-18433: -- Thanks for all the above analysis.I add some test notes: 1. The last commit corresponding to the test package is e13146f80114266aa34c9fe9f3dc27e87f7a7649, [~liyu] You can check whether your pr is included 2. Result acquisition: The comparison result data is the job's tps. In order to avoid the impact of resource allocation, the TPS will be obtained through the restful API(jobs/$jobId/vertices/$verticeId/subtasks/metrics?get=*numBuffersOutPerSecond) after the job is submitted for 2 minutes (the job has been actually scheduled at this time). Each job will get 10 tps, the interval between the two tps is 10s, and then average the 10 tps. Each scene will submit 5 jobs to find the average tps as the finally result. 3. parallism is 1: I will adjust the machine to 1 and run again, then update the data > From the end-to-end performance test results, 1.11 has a regression > --- > > Key: FLINK-18433 > URL: https://issues.apache.org/jira/browse/FLINK-18433 > Project: Flink > Issue Type: Bug > Components: API / Core, API / DataStream >Affects Versions: 1.11.0 > Environment: 3 machines > [|https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations_1.11/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java] >Reporter: Aihua Li >Priority: Major > Attachments: flink_11.log.gz > > > > I ran end-to-end performance tests between the Release-1.10 and Release-1.11. > the results were as follows: > |scenarioName|release-1.10|release-1.11| | > |OneInput_Broadcast_LazyFromSource_ExactlyOnce_10_rocksdb|46.175|43.8133|-5.11%| > |OneInput_Rescale_LazyFromSource_ExactlyOnce_100_heap|211.835|200.355|-5.42%| > |OneInput_Rebalance_LazyFromSource_ExactlyOnce_1024_rocksdb|1721.041667|1618.32|-5.97%| > |OneInput_KeyBy_LazyFromSource_ExactlyOnce_10_heap|46|43.615|-5.18%| > |OneInput_Broadcast_Eager_ExactlyOnce_100_rocksdb|212.105|199.688|-5.85%| > |OneInput_Rescale_Eager_ExactlyOnce_1024_heap|1754.64|1600.12|-8.81%| > |OneInput_Rebalance_Eager_ExactlyOnce_10_rocksdb|45.9167|43.0983|-6.14%| > |OneInput_KeyBy_Eager_ExactlyOnce_100_heap|212.0816667|200.727|-5.35%| > |OneInput_Broadcast_LazyFromSource_AtLeastOnce_1024_rocksdb|1718.245|1614.381667|-6.04%| > |OneInput_Rescale_LazyFromSource_AtLeastOnce_10_heap|46.12|43.5517|-5.57%| > |OneInput_Rebalance_LazyFromSource_AtLeastOnce_100_rocksdb|212.038|200.388|-5.49%| > |OneInput_KeyBy_LazyFromSource_AtLeastOnce_1024_heap|1762.048333|1606.408333|-8.83%| > |OneInput_Broadcast_Eager_AtLeastOnce_10_rocksdb|46.0583|43.4967|-5.56%| > |OneInput_Rescale_Eager_AtLeastOnce_100_heap|212.233|201.188|-5.20%| > |OneInput_Rebalance_Eager_AtLeastOnce_1024_rocksdb|1720.66|1616.85|-6.03%| > |OneInput_KeyBy_Eager_AtLeastOnce_10_heap|46.14|43.6233|-5.45%| > |TwoInputs_Broadcast_LazyFromSource_ExactlyOnce_100_rocksdb|156.918|152.957|-2.52%| > |TwoInputs_Rescale_LazyFromSource_ExactlyOnce_1024_heap|1415.511667|1300.1|-8.15%| > |TwoInputs_Rebalance_LazyFromSource_ExactlyOnce_10_rocksdb|34.2967|34.1667|-0.38%| > |TwoInputs_KeyBy_LazyFromSource_ExactlyOnce_100_heap|158.353|151.848|-4.11%| > |TwoInputs_Broadcast_Eager_ExactlyOnce_1024_rocksdb|1373.406667|1300.056667|-5.34%| > |TwoInputs_Rescale_Eager_ExactlyOnce_10_heap|34.5717|32.0967|-7.16%| > |TwoInputs_Rebalance_Eager_ExactlyOnce_100_rocksdb|158.655|147.44|-7.07%| > |TwoInputs_KeyBy_Eager_ExactlyOnce_1024_heap|1356.611667|1292.386667|-4.73%| > |TwoInputs_Broadcast_LazyFromSource_AtLeastOnce_10_rocksdb|34.01|33.205|-2.37%| > |TwoInputs_Rescale_LazyFromSource_AtLeastOnce_100_heap|149.588|145.997|-2.40%| > |TwoInputs_Rebalance_LazyFromSource_AtLeastOnce_1024_rocksdb|1359.74|1299.156667|-4.46%| > |TwoInputs_KeyBy_LazyFromSource_AtLeastOnce_10_heap|34.025|29.6833|-12.76%| > |TwoInputs_Broadcast_Eager_AtLeastOnce_100_rocksdb|157.303|151.4616667|-3.71%| > |TwoInputs_Rescale_Eager_AtLeastOnce_1024_heap|1368.74|1293.238333|-5.52%| > |TwoInputs_Rebalance_Eager_AtLeastOnce_10_rocksdb|34.325|33.285|-3.03%| > |TwoInputs_KeyBy_Eager_AtLeastOnce_100_heap|162.5116667|134.375|-17.31%| > It can be seen that the performance of 1.11 has a regression, basically > around 5%, and the maximum regression is 17%. This needs to be checked. > the test code: > flink-1.10.0: > [https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java] > flink-1.11.0: >
[GitHub] [flink] libenchao commented on pull request #12759: [FLINK-18397][docs-zh] Translate "Table & SQL Connectors Overview" page into Chinese
libenchao commented on pull request #12759: URL: https://github.com/apache/flink/pull/12759#issuecomment-650701241 @wuchong Sure, I'll give it a review later today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Luxios22 commented on pull request #12772: [FLINK-18394][docs-zh] Translate "Parquet Format" page into Chinese
Luxios22 commented on pull request #12772: URL: https://github.com/apache/flink/pull/12772#issuecomment-650700603 Thanks for your review, I've submitted the requested change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Luxios22 commented on a change in pull request #12772: [FLINK-18394][docs-zh] Translate "Parquet Format" page into Chinese
Luxios22 commented on a change in pull request #12772: URL: https://github.com/apache/flink/pull/12772#discussion_r446603593 ## File path: docs/dev/table/connectors/formats/parquet.zh.md ## @@ -186,4 +185,4 @@ The following table lists the type mapping from Flink type to Parquet type. -Attention Composite data type: Array, Map and Row are not supported. +注意 复合数据类型（Array、Map 与 Row）不被支持。 Review comment: Thanks for your review, I've submitted the requested change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18440) ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
[ https://issues.apache.org/jira/browse/FLINK-18440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147219#comment-17147219 ] LakeShen commented on FLINK-18440: -- Hi [~lsy], I find that in the end my sql would translate into like this : SELECT `test`.`name`, `test`.`eat`, `test`.`cnt`, ROW_NUMBER() OVER (PARTITION BY `test`.`name` ORDER BY `test`.`cnt` DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS `row_num` FROM `default_catalog`.`default_database`.`test` AS `test`. And at the same time , I add the ROWS BETWEEN unbounded PRECEDING AND CURRENT ROW into my sql , It aslo throw the exception like this : Caused by: org.apache.calcite.sql.validate.SqlValidatorException: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions > ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or > ROW_NUMBER functions > > > Key: FLINK-18440 > URL: https://issues.apache.org/jira/browse/FLINK-18440 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: LakeShen >Priority: Major > Fix For: 1.11.1 > > > When I run flink sql ,the flink sql like this: > create view test as select name, eat ,sum(age) as cnt from test_source group > by name,eat; > create view results as select *, ROW_NUMBER() OVER (PARTITION BY name > ORDER BY cnt DESC) as row_num from test; > create table sink ( > name varchar, > eat varchar, > cnt bigint > ) > with( > 'connector' = 'print' > ); > insert into sink select name,eat , cnt from results where row_num <= 3 ; > The same sql code I could run success in flink 1.10, now I change the flink > version into flink 1.11, it throw the exception. > Exception in thread "main" org.apache.flink.table.api.ValidationException: > SQL validation failed. From line 1, column 124 to line 1, column 127: > ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:146) > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:204) > at > org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52) > at > org.apache.flink.table.planner.catalog.SqlCatalogViewTable.convertToRel(SqlCatalogViewTable.java:58) > at > org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:59) > at > org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:55) > at > org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:661) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:642) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3345) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:568) > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:164) > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:151) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:773) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:745) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlInsert(SqlToOperationConverter.java:527) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12777: [FLINK-18262][python][e2e] Fix the unstable e2e tests of pyflink.
flinkbot edited a comment on pull request #12777: URL: https://github.com/apache/flink/pull/12777#issuecomment-650680478 ## CI report: * ef52a75fdbcddde7ac55eea60d1b6db3bbcd6056 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4066) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18440) ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
[
https://issues.apache.org/jira/browse/FLINK-18440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147217#comment-17147217
]
dalongliu edited comment on FLINK-18440 at 6/28/20, 5:34 AM:
-
[~shenlang]，I also encountered this problem when create view use row_number
function, I have study calcite source code, it 'SqlWindow#validate' method
throw this error, so I guess calcite don't support create view use row_ number
over partition by order by desc grammar, because there's no explicit lower
bound and upper bound. if you use following sql:
{code:java}
create view order_source as select sum(amount) over (partition by order_id,
order_goods_id order by proctime ROWS BETWEEN unbounded PRECEDING AND CURRENT
ROW) from dm_trd_order_goods
{code}
it will not throw exception, I think this is calcite's problem.
CC [~jark], what do you think about it?
was (Author: lsy):
[~shenlang]，I also encountered this problem when create view use row_number
function, I have study calcite source code, it 'SqlWindow#validate' method
throw this error, so I guess calcite don't support create view use row_ number
over partition by order by desc grammar, because there's no explicit lower
bound and upper bound. if you use following sql:
{code:java}
create view order_source as select sum(amount) over (partition by order_id,
order_goods_id order by proctime ROWS BETWEEN unbounded PRECEDING AND CURRENT
ROW) from dm_trd_order_goods
{code}
it will not throw exception, I think this is calcite's problem.
> ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or
> ROW_NUMBER functions
>
>
> Key: FLINK-18440
> URL: https://issues.apache.org/jira/browse/FLINK-18440
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: LakeShen
>Priority: Major
> Fix For: 1.11.1
>
>
> When I run flink sql ,the flink sql like this:
> create view test as select name, eat ,sum(age) as cnt from test_source group
> by name,eat;
> create view results as select *, ROW_NUMBER() OVER (PARTITION BY name
> ORDER BY cnt DESC) as row_num from test;
> create table sink (
> name varchar,
> eat varchar,
> cnt bigint
> )
> with(
> 'connector' = 'print'
> );
> insert into sink select name,eat , cnt from results where row_num <= 3 ;
> The same sql code I could run success in flink 1.10, now I change the flink
> version into flink 1.11, it throw the exception.
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 1, column 124 to line 1, column 127:
> ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:146)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:204)
> at
> org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52)
> at
> org.apache.flink.table.planner.catalog.SqlCatalogViewTable.convertToRel(SqlCatalogViewTable.java:58)
> at
> org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:59)
> at
> org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:55)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:661)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:642)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3345)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:568)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:164)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:151)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:773)
> at
>
[jira] [Commented] (FLINK-18440) ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
[
https://issues.apache.org/jira/browse/FLINK-18440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147217#comment-17147217
]
dalongliu commented on FLINK-18440:
---
[~shenlang]，I also encountered this problem when create view use row_number
function, I have study calcite source code, it 'SqlWindow#validate' method
throw this error, so I guess calcite don't support create view use row_ number
over partition by order by desc grammar, because there's no explicit lower
bound and upper bound. if you use following sql:
{code:java}
create view order_source as select sum(amount) over (partition by order_id,
order_goods_id order by proctime ROWS BETWEEN unbounded PRECEDING AND CURRENT
ROW) from dm_trd_order_goods
{code}
it will not throw exception, I think this is calcite's problem.
> ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or
> ROW_NUMBER functions
>
>
> Key: FLINK-18440
> URL: https://issues.apache.org/jira/browse/FLINK-18440
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: LakeShen
>Priority: Major
> Fix For: 1.11.1
>
>
> When I run flink sql ,the flink sql like this:
> create view test as select name, eat ,sum(age) as cnt from test_source group
> by name,eat;
> create view results as select *, ROW_NUMBER() OVER (PARTITION BY name
> ORDER BY cnt DESC) as row_num from test;
> create table sink (
> name varchar,
> eat varchar,
> cnt bigint
> )
> with(
> 'connector' = 'print'
> );
> insert into sink select name,eat , cnt from results where row_num <= 3 ;
> The same sql code I could run success in flink 1.10, now I change the flink
> version into flink 1.11, it throw the exception.
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 1, column 124 to line 1, column 127:
> ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:146)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:204)
> at
> org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52)
> at
> org.apache.flink.table.planner.catalog.SqlCatalogViewTable.convertToRel(SqlCatalogViewTable.java:58)
> at
> org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:59)
> at
> org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:55)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:661)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:642)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3345)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:568)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:164)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:151)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:773)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:745)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlInsert(SqlToOperationConverter.java:527)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18423) Fix Prefer tag in document "Detecting Patterns" page of "Streaming Concepts"
[ https://issues.apache.org/jira/browse/FLINK-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-18423. --- Fix Version/s: 1.11.0 Resolution: Fixed - master (1.12.0): a5c2f26a7b799473cfa43acb53d94d24cd33464e - 1.11.0: 02695f4f2c853fd40fae993a61c40bd1cfc078ed > Fix Prefer tag in document "Detecting Patterns" page of "Streaming Concepts" > > > Key: FLINK-18423 > URL: https://issues.apache.org/jira/browse/FLINK-18423 > Project: Flink > Issue Type: Improvement > Components: Documentation, Documentation / Training >Affects Versions: 1.10.0, 1.10.1 >Reporter: Roc Marshal >Assignee: Roc Marshal >Priority: Minor > Labels: document, easyfix, pull-request-available > Fix For: 1.11.0 > > > Update Prefer tag in documentation "Detecting Patterns" page of "Streaming > Concepts" according to > [Prefer|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Reminder-Prefer-link-tag-in-documentation-td42362.html]. > > The markdown file location is: > flink/docs/dev/table/streaming/match_recognize.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #12764: [FLINK-18423][docs] Fix Prefer tag in document "Detecting Patterns" age of "Streaming Concepts"
wuchong merged pull request #12764: URL: https://github.com/apache/flink/pull/12764 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #12759: [FLINK-18397][docs-zh] Translate "Table & SQL Connectors Overview" page into Chinese
wuchong commented on pull request #12759: URL: https://github.com/apache/flink/pull/12759#issuecomment-650699331 Hi @libenchao could you help to review this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18395) Translate "ORC Format" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18395: --- Labels: pull-request-available (was: ) > Translate "ORC Format" page into Chinese > > > Key: FLINK-18395 > URL: https://issues.apache.org/jira/browse/FLINK-18395 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: pengweibo >Priority: Major > Labels: pull-request-available > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/formats/orc.html > The markdown file is located in > flink/docs/dev/table/connectors/formats/orc.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #12774: [FLINK-18395][FLINK-18388][docs-zh] Translate "ORC Format" and "CSV Format" page into Chinese
wuchong commented on a change in pull request #12774:
URL: https://github.com/apache/flink/pull/12774#discussion_r446597955
##
File path: docs/dev/table/connectors/formats/csv.zh.md
##
@@ -1,8 +1,8 @@
---
-title: "CSV Format"
-nav-title: CSV
-nav-parent_id: sql-formats
-nav-pos: 1
+标题: "CSV 格式"
+源标题: CSV
+父id: sql 格式
+位置: 1
Review comment:
Do not translate the meta fields. You can have a look
https://flink.apache.org/contributing/docs-style.html about the "Front Matter".
##
File path: docs/dev/table/connectors/formats/csv.zh.md
##
@@ -23,27 +23,29 @@ specific language governing permissions and limitations
under the License.
-->
-Format: Serialization Schema
-Format: Deserialization Schema
+格式: 序列化结构
+格式: 反序列化结构
-* This will be replaced by the TOC
+* 此处被TOC替换
Review comment:
Do not translate this.
##
File path: docs/dev/table/connectors/formats/csv.zh.md
##
@@ -67,116 +69,112 @@ CREATE TABLE user_behavior (
-Format Options
+格式选项
-Option
-Required
-Default
-Type
-Description
+选项
+要求
+默认
+类型
+描述
format
- required
+ 必须
(none)
String
- Specify what format to use, here should be 'csv'.
+ 详细描述使用什么格式，这里应该是 'csv'。
csv.field-delimiter
- optional
+ 可选
,
String
- Field delimiter character (',' by default).
+ 分隔方式 (默认',')。
csv.line-delimiter
- optional
+ 可选
\n
String
- Line delimiter, \n by default. Note the \n
and \r are invisible special characters, you have to use unicode
to specify them in plain SQL.
+ 行分隔符, 默认\n. \n 和 \r
是不可见的特殊符号, 在显式的sql语句中必须使用unicode编码。
- e.g. 'csv.line-delimiter' = U&'\\000D' specifies
the to use carriage return \r as line delimiter.
- e.g. 'csv.line-delimiter' = U&'\\000A' specifies
the to use line feed \n as line delimiter.
+ 例如 'csv.line-delimiter' = U&'\\000D' 使用换行符号
\r 作为行分隔符。
+ 例如 'csv.line-delimiter' = U&'\\000A' 使用换行符号
\n 作为行分隔符。
csv.disable-quote-character
- optional
+ 可选
false
Boolean
- Disabled quote character for enclosing field values (false by
default).
- If true, option 'csv.quote-character' must be set.
+ 关闭对引用的值使用引号 (默认是 false).如果允许，选项 'csv.quote-character'
必须被设置。
csv.quote-character
- optional
+ 可选
"
String
- Quote character for enclosing field values (" by
default).
+ 引用内容的符号设置 (默认").
csv.allow-comments
- optional
+ 可选
false
Boolean
- Ignore comment lines that start with '#' (disabled by
default).
- If enabled, make sure to also ignore parse errors to allow empty
rows.
+ 对注释的行使用 '#' (默认关闭).如果开启, 忽略解析错误来允许空行的存在.
+
csv.ignore-parse-errors
- optional
+ 可选
false
Boolean
- Skip fields and rows with parse errors instead of failing.
- Fields are set to null in case of errors.
+ 对于解析失败的字段和行直接忽略.解析失败的字段直接设置为null.
Review comment:
```suggestion
当解析异常时，是跳过当前字段或行，还是抛出错误失败（默认为
false，即抛出错误失败）。如果忽略字段的解析异常，则会将该字段值设置为null。
```
##
File path: docs/dev/table/connectors/formats/csv.zh.md
##
@@ -67,116 +69,112 @@ CREATE TABLE user_behavior (
-Format Options
+格式选项
-Option
-Required
-Default
-Type
-Description
+选项
+要求
+默认
+类型
+描述
format
- required
+ 必须
Review comment:
```suggestion
必选
```
##
File path: docs/dev/table/connectors/formats/orc.zh.md
##
@@ -23,28 +23,28 @@ specific language governing permissions and limitations
under the License.
-->
-Format: Serialization Schema
-Format: Deserialization Schema
+格式: 序列化结构
+格式: 反序列化结构
-* This will be replaced by the TOC
+* 这里将会被TOC替换
Review comment:
Do not translate.
##
File path: docs/dev/table/connectors/formats/orc.zh.md
##
@@ -65,17 +65,17 @@ CREATE TABLE user_behavior (
-Format Options
+格式选项
-Option
-Required
-Default
-Type
-Description
+选项
+要求
Review comment:
是否必选
##
File path: docs/dev/table/connectors/formats/csv.zh.md
##
@@ -67,116 +69,112 @@ CREATE TABLE user_behavior (
-Format Options
+格式选项
-Option
-Required
-Default
-Type
-Description
+选项
+要求
+默认
+类型
+描述
format
- required
+ 必须
(none)
String
-
[GitHub] [flink] flinkbot edited a comment on pull request #12778: [FLINK-18422][docs] Update Prefer tag in documentation 'Fault Tolerance training lesson'
flinkbot edited a comment on pull request #12778: URL: https://github.com/apache/flink/pull/12778#issuecomment-650692418 ## CI report: * 32c8e1a9920e030977d39a0b1e5caf9ac52cd840 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4074) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18440) ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
[ https://issues.apache.org/jira/browse/FLINK-18440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LakeShen updated FLINK-18440: - Description: When I run flink sql ,the flink sql like this: create view test as select name, eat ,sum(age) as cnt from test_source group by name,eat; create view results as select *, ROW_NUMBER() OVER (PARTITION BY name ORDER BY cnt DESC) as row_num from test; create table sink ( name varchar, eat varchar, cnt bigint ) with( 'connector' = 'print' ); insert into sink select name,eat , cnt from results where row_num <= 3 ; The same sql code I could run success in flink 1.10, now I change the flink version into flink 1.11, it throw the exception. Exception in thread "main" org.apache.flink.table.api.ValidationException: SQL validation failed. From line 1, column 124 to line 1, column 127: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:146) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:204) at org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52) at org.apache.flink.table.planner.catalog.SqlCatalogViewTable.convertToRel(SqlCatalogViewTable.java:58) at org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:59) at org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:55) at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492) at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:661) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:642) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3345) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:568) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:164) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:151) at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:773) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:745) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlInsert(SqlToOperationConverter.java:527) was: When I run flink sql ,the flink sql like this: create view test as select name, eat ,sum(age) as cnt from test_source group by name,eat; create view results as select *, ROW_NUMBER() OVER (PARTITION BY name ORDER BY cnt DESC) as row_num from test; The same sql code I could run success in flink 1.10, now I change the flink version into flink 1.11, it throw the exception. Exception in thread "main" org.apache.flink.table.api.ValidationException: SQL validation failed. From line 1, column 124 to line 1, column 127: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:146) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:204) at org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52) at org.apache.flink.table.planner.catalog.SqlCatalogViewTable.convertToRel(SqlCatalogViewTable.java:58) at org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:59) at org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:55) at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492) at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051)
[GitHub] [flink] flinkbot commented on pull request #12778: [FLINK-18422][docs] Update Prefer tag in documentation 'Fault Tolerance training lesson'
flinkbot commented on pull request #12778: URL: https://github.com/apache/flink/pull/12778#issuecomment-650692418 ## CI report: * 32c8e1a9920e030977d39a0b1e5caf9ac52cd840 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18198) Translate "HBase SQL Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-18198. --- Fix Version/s: 1.11.0 Resolution: Fixed - master (1.12.0): 2ab3468c33b1b4e6d86c63cff2009209a23f873e - 1.11.0: 2c55f967d2598b2e2781bd5516b80ba3885da353 > Translate "HBase SQL Connector" page into Chinese > - > > Key: FLINK-18198 > URL: https://issues.apache.org/jira/browse/FLINK-18198 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: YiQiang Zeng >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/hbase.html > The markdown file is located in flink/docs/dev/table/connectors/hbase.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] RocMarshal commented on pull request #12778: [FLINK-18422][docs] Update Prefer tag in documentation 'Fault Tolerance training lesson'
RocMarshal commented on pull request #12778: URL: https://github.com/apache/flink/pull/12778#issuecomment-650690595 @klion26 Could you help me to review this PR(https://github.com/apache/flink/pull/12778) Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18164) null <> 'str' should be true
[ https://issues.apache.org/jira/browse/FLINK-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147198#comment-17147198 ] Leonard Xu commented on FLINK-18164: HI, [~libenchao] I think our code gen logic that set the default bool expression value to false is the implementation of unknown as false mode by design. And my concern is that you closed this issue without any comments..., so I'm not sure your next plan of this issue. > null <> 'str' should be true > > > Key: FLINK-18164 > URL: https://issues.apache.org/jira/browse/FLINK-18164 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Reporter: Benchao Li >Priority: Major > > Currently, if we compare null with other literals, the result will always be > false. > It's because the code gen always gives a default value (false) for the > result. And I think it's a bug if `null <> 'str'` is false. > It's reported from user-zh: > http://apache-flink.147419.n8.nabble.com/flink-sql-null-false-td3640.html > CC [~jark] [~ykt836] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12778: [FLINK-18422][docs] Update Prefer tag in documentation 'Fault Tolerance training lesson'
flinkbot commented on pull request #12778: URL: https://github.com/apache/flink/pull/12778#issuecomment-650690513 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 32c8e1a9920e030977d39a0b1e5caf9ac52cd840 (Sun Jun 28 04:12:00 UTC 2020) **Warnings:** * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18422) Update Prefer tag in documentation 'Fault Tolerance training lesson'
[ https://issues.apache.org/jira/browse/FLINK-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18422: --- Labels: document easyfix pull-request-available (was: document easyfix) > Update Prefer tag in documentation 'Fault Tolerance training lesson' > > > Key: FLINK-18422 > URL: https://issues.apache.org/jira/browse/FLINK-18422 > Project: Flink > Issue Type: Improvement > Components: Documentation, Documentation / Training >Affects Versions: 1.10.0, 1.10.1 >Reporter: Roc Marshal >Assignee: Roc Marshal >Priority: Minor > Labels: document, easyfix, pull-request-available > Attachments: current_prefer_mode.png > > Original Estimate: 48h > Remaining Estimate: 48h > > Update Prefer tag in documentation 'Fault Tolerance training lesson' > according to > [Prefer|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Reminder-Prefer-link-tag-in-documentation-td42362.html]. > > The location is: docs/learn-flink/fault_tolerance.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #12644: [FLINK-18198][docs-zh]Translate "HBase SQL Connector" page into Chinese
wuchong closed pull request #12644: URL: https://github.com/apache/flink/pull/12644 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal opened a new pull request #12778: [FLINK-18422][docs] Update Prefer tag in documentation 'Fault Tolerance training lesson'
RocMarshal opened a new pull request #12778: URL: https://github.com/apache/flink/pull/12778 ## What is the purpose of the change Update Prefer tag in documentation 'Fault Tolerance training lesson' according to [Prefer Reminder](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Reminder-Prefer-link-tag-in-documentation-td42362.html) .The markdown file location is: flink/docs/learn-flink/fault_tolerance.md ## Brief change log - *Update Prefer tag in documentation 'flink/docs/learn-flink/fault_tolerance.md'* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #12644: [FLINK-18198][docs-zh]Translate "HBase SQL Connector" page into Chinese
wuchong commented on a change in pull request #12644:
URL: https://github.com/apache/flink/pull/12644#discussion_r446596940
##
File path: docs/dev/table/connectors/hbase.zh.md
##
@@ -66,123 +66,115 @@ CREATE TABLE hTable (
'zookeeper.quorum' = 'localhost:2181'
);
--- use ROW(...) construction function construct column families and write data
into the HBase table.
--- assuming the schema of "T" is [rowkey, f1q1, f2q2, f2q3, f3q4, f3q5, f3q6]
+-- 用 ROW(...) 构造函数构造列簇，并往 HBase 表写数据。
+-- 假设 "T" 的表结构是 [rowkey, f1q1, f2q2, f2q3, f3q4, f3q5, f3q6]
INSERT INTO hTable
SELECT rowkey, ROW(f1q1), ROW(f2q2, f2q3), ROW(f3q4, f3q5, f3q6) FROM T;
--- scan data from the HBase table
+-- 从 HBase 表扫描数据
SELECT rowkey, family1, family3.q4, family3.q6 FROM hTable;
--- temporal join the HBase table as a dimension table
+-- temporal join HBase 表，将 HBase 表作为维表
SELECT * FROM myTopic
LEFT JOIN hTable FOR SYSTEM_TIME AS OF myTopic.proctime
ON myTopic.key = hTable.rowkey;
{% endhighlight %}
-Connector Options
+连接器参数选项
-Option
-Required
-Default
-Type
-Description
+参数选项
+是否必选
+默认值
+数据类型
+参数描述
connector
- required
- (none)
+ 必选
+ none
Review comment:
Keep the original bracket `( )`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-18440) ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
LakeShen created FLINK-18440: Summary: ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions Key: FLINK-18440 URL: https://issues.apache.org/jira/browse/FLINK-18440 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.11.0 Reporter: LakeShen Fix For: 1.11.1 When I run flink sql ,the flink sql like this: create view test as select name, eat ,sum(age) as cnt from test_source group by name,eat; create view results as select *, ROW_NUMBER() OVER (PARTITION BY name ORDER BY cnt DESC) as row_num from test; The same sql code I could run success in flink 1.10, now I change the flink version into flink 1.11, it throw the exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18440) ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
[ https://issues.apache.org/jira/browse/FLINK-18440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LakeShen updated FLINK-18440: - Description: When I run flink sql ,the flink sql like this: create view test as select name, eat ,sum(age) as cnt from test_source group by name,eat; create view results as select *, ROW_NUMBER() OVER (PARTITION BY name ORDER BY cnt DESC) as row_num from test; The same sql code I could run success in flink 1.10, now I change the flink version into flink 1.11, it throw the exception. Exception in thread "main" org.apache.flink.table.api.ValidationException: SQL validation failed. From line 1, column 124 to line 1, column 127: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:146) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:204) at org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52) at org.apache.flink.table.planner.catalog.SqlCatalogViewTable.convertToRel(SqlCatalogViewTable.java:58) at org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:59) at org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:55) at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492) at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:661) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:642) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3345) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:568) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:164) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:151) at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:773) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:745) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlInsert(SqlToOperationConverter.java:527) was: When I run flink sql ,the flink sql like this: create view test as select name, eat ,sum(age) as cnt from test_source group by name,eat; create view results as select *, ROW_NUMBER() OVER (PARTITION BY name ORDER BY cnt DESC) as row_num from test; The same sql code I could run success in flink 1.10, now I change the flink version into flink 1.11, it throw the exception. > ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or > ROW_NUMBER functions > > > Key: FLINK-18440 > URL: https://issues.apache.org/jira/browse/FLINK-18440 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: LakeShen >Priority: Major > Fix For: 1.11.1 > > > When I run flink sql ,the flink sql like this: > create view test as select name, eat ,sum(age) as cnt from test_source group > by name,eat; > create view results as select *, ROW_NUMBER() OVER (PARTITION BY name > ORDER BY cnt DESC) as row_num from test; > The same sql code I could run success in flink 1.10, now I change the flink > version into flink 1.11, it throw the exception. > Exception in thread "main" org.apache.flink.table.api.ValidationException: > SQL validation failed. From line 1, column 124 to line 1, column 127: > ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:146) > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:204) > at >
[jira] [Issue Comment Deleted] (FLINK-18422) Update Prefer tag in documentation 'Fault Tolerance training lesson'
[
https://issues.apache.org/jira/browse/FLINK-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roc Marshal updated FLINK-18422:
Comment: was deleted
(was: [~alpinegizmo] It doesn't matter.
And I still have a question as follows:
Should *"\{{ site.baseurl }}/fig/stream_barriers.svg"* be written as *"\{%
link fig/stream_barriers.svg %}"* or *"/fig/stream_barriers.svg"* according to
*Visual Elements* in
[docs-style|https://flink.apache.org/zh/contributing/docs-style.html]?
Could you give me some suggestions about this ?
Thank you.)
> Update Prefer tag in documentation 'Fault Tolerance training lesson'
>
>
> Key: FLINK-18422
> URL: https://issues.apache.org/jira/browse/FLINK-18422
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Documentation / Training
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Roc Marshal
>Assignee: Roc Marshal
>Priority: Minor
> Labels: document, easyfix
> Attachments: current_prefer_mode.png
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Update Prefer tag in documentation 'Fault Tolerance training lesson'
> according to
> [Prefer|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Reminder-Prefer-link-tag-in-documentation-td42362.html].
>
> The location is: docs/learn-flink/fault_tolerance.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] RocMarshal commented on a change in pull request #12727: [FLINK-17292][docs] Translate Fault Tolerance training lesson to Chinese
RocMarshal commented on a change in pull request #12727:
URL: https://github.com/apache/flink/pull/12727#discussion_r446596356
##
File path: docs/learn-flink/fault_tolerance.zh.md
##
@@ -29,180 +29,156 @@ under the License.
## State Backends
-The keyed state managed by Flink is a sort of sharded, key/value store, and
the working copy of each
-item of keyed state is kept somewhere local to the taskmanager responsible for
that key. Operator
-state is also local to the machine(s) that need(s) it. Flink periodically
takes persistent snapshots
-of all the state and copies these snapshots somewhere more durable, such as a
distributed file
-system.
+由 Flink 管理的 keyed state 是一种分片的键/值存储，并且 keyed state 的每一项的工作副本都保存在负责该键的
taskmanager 本地的某个地方。
+Operator state 对于需要它的机器节点来说也是本地的。Flink 定期获取所有状态的连续快照，并将这些快照复制到持久化的地方，例如分布式文件系统。
-In the event of the failure, Flink can restore the complete state of your
application and resume
-processing as though nothing had gone wrong.
+如果发生故障，Flink 可以恢复应用程序的完整状态并恢复处理，就如同没有出现过异常一样。
-This state that Flink manages is stored in a _state backend_. Two
implementations of state backends
-are available -- one based on RocksDB, an embedded key/value store that keeps
its working state on
-disk, and another heap-based state backend that keeps its working state in
memory, on the Java heap.
-This heap-based state backend comes in two flavors: the FsStateBackend that
persists its state
-snapshots to a distributed file system, and the MemoryStateBackend that uses
the JobManager's heap.
+Flink 管理的状态存储在 _state backend_。
+state backends 的两种实现 -- 一种是基于 RocksDB 内嵌 key/value 存储将其工作状态保存在磁盘上的，另一种基于堆的
state backend，将其工作状态保存在 Java 的 堆内存中。
+这种基于堆的 state backend 有两种方式：保存其状态快照到分布式文件系统的 FsStateBackend，以及使用 JobManager 堆的
MemoryStateBackend。
- Name
+ 名称
Working State
- State Backup
- Snapshotting
+ 状态备份
+ 快照
RocksDBStateBackend
- Local disk (tmp dir)
- Distributed file system
- Full / Incremental
+ 本地磁盘（tmp dir）
+ 分布式文件系统
+ 全量 / 增量
- Supports state larger than available memory
- Rule of thumb: 10x slower than heap-based backends
+ 支持大于内存大小的状态
+ 经验法则：比基于堆的后端慢10倍
FsStateBackend
JVM Heap
- Distributed file system
- Full
+ 分布式文件系统
+ 全量
- Fast, requires large heap
- Subject to GC
+ 快速，需要大的堆内存
+ 受限制于 GC
MemoryStateBackend
JVM Heap
JobManager JVM Heap
- Full
+ 全量
- Good for testing and experimentation with small state
(locally)
+ 适用于小状态（本地）的测试和实验
-When working with state kept in a heap-based state backend, accesses and
updates involve reading and
-writing objects on the heap. But for objects kept in the
`RocksDBStateBackend`, accesses and updates
-involve serialization and deserialization, and so are much more expensive. But
the amount of state
-you can have with RocksDB is limited only by the size of the local disk. Note
also that only the
-`RocksDBStateBackend` is able to do incremental snapshotting, which is a
significant benefit for
-applications with large amounts of slowly changing state.
+当使用基于堆的 state backend 保存状态时，访问和更新涉及在堆上读写对象。
+但是对于保存在 `RocksDBStateBackend` 中的对象，访问和更新涉及序列化和反序列化，所以会有更加巨大的开销。
+但 RocksDB 的状态量仅受本地磁盘大小的限制。
+还要注意，只有 `RocksDBStateBackend` 能够进行增量快照，这对于具有大量变化缓慢状态的应用程序来说是大有裨益的。
-All of these state backends are able to do asynchronous snapshotting, meaning
that they can take a
-snapshot without impeding the ongoing stream processing.
+所有这些 state backends 都能够异步执行快照，这意味着它们可以在不妨碍正在进行的流处理的情况下执行快照。
{% top %}
-## State Snapshots
+## 状态快照
-### Definitions
+### 定义
-* _Snapshot_ -- a generic term referring to a global, consistent image of the
state of a Flink job.
- A snapshot includes a pointer into each of the data sources (e.g., an offset
into a file or Kafka
- partition), as well as a copy of the state from each of the job's stateful
operators that resulted
- from having processed all of the events up to those positions in the sources.
-* _Checkpoint_ -- a snapshot taken automatically by Flink for the purpose of
being able to recover
- from faults. Checkpoints can be incremental, and are optimized for being
restored quickly.
-* _Externalized Checkpoint_ -- normally checkpoints are not intended to be
manipulated by users.
- Flink retains only the _n_-most-recent checkpoints (_n_ being configurable)
while a job is
- running, and deletes them when a job is cancelled. But you can configure
them to be retained
- instead, in which case you can manually resume from them.
-* _Savepoint_ -- a snapshot triggered manually by a user (or an API call) for
some operational
- purpose, such as a stateful
[jira] [Comment Edited] (FLINK-18422) Update Prefer tag in documentation 'Fault Tolerance training lesson'
[
https://issues.apache.org/jira/browse/FLINK-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147194#comment-17147194
]
Roc Marshal edited comment on FLINK-18422 at 6/28/20, 3:48 AM:
---
[~alpinegizmo] It doesn't matter.
And I still have a question as follows:
Should *"\{{ site.baseurl }}/fig/stream_barriers.svg"* be written as *"\{%
link fig/stream_barriers.svg %}"* or *"/fig/stream_barriers.svg"* according to
*Visual Elements* in
[docs-style|https://flink.apache.org/zh/contributing/docs-style.html]?
Could you give me some suggestions about this ?
Thank you.
was (Author: rocmarshal):
[~alpinegizmo] It doesn't matter.
And I still have a question as follows:
Should *"\{{ site.baseurl }}/fig/stream_barriers.svg"* be written as *"\{%
link fig/stream_barriers.svg %}"* or *"/fig/stream_barriers.svg"* according to
*Visual Elements* in
[docs-style|https://flink.apache.org/zh/contributing/docs-style.html]?
[https://github.com/apache/flink/pull/12727#discussion_r446595140]
Could you give me some suggestions about this ?
Thank you.
> Update Prefer tag in documentation 'Fault Tolerance training lesson'
>
>
> Key: FLINK-18422
> URL: https://issues.apache.org/jira/browse/FLINK-18422
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Documentation / Training
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Roc Marshal
>Assignee: Roc Marshal
>Priority: Minor
> Labels: document, easyfix
> Attachments: current_prefer_mode.png
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Update Prefer tag in documentation 'Fault Tolerance training lesson'
> according to
> [Prefer|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Reminder-Prefer-link-tag-in-documentation-td42362.html].
>
> The location is: docs/learn-flink/fault_tolerance.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18422) Update Prefer tag in documentation 'Fault Tolerance training lesson'
[
https://issues.apache.org/jira/browse/FLINK-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147194#comment-17147194
]
Roc Marshal commented on FLINK-18422:
-
[~alpinegizmo] It doesn't matter.
And I still have a question as follows:
Should *"\{{ site.baseurl }}/fig/stream_barriers.svg"* be written as *"\{%
link fig/stream_barriers.svg %}"* or *"/fig/stream_barriers.svg"* according to
*Visual Elements* in
[docs-style|https://flink.apache.org/zh/contributing/docs-style.html]?
[https://github.com/apache/flink/pull/12727#discussion_r446595140]
Could you give me some suggestions about this ?
Thank you.
> Update Prefer tag in documentation 'Fault Tolerance training lesson'
>
>
> Key: FLINK-18422
> URL: https://issues.apache.org/jira/browse/FLINK-18422
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Documentation / Training
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Roc Marshal
>Assignee: Roc Marshal
>Priority: Minor
> Labels: document, easyfix
> Attachments: current_prefer_mode.png
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Update Prefer tag in documentation 'Fault Tolerance training lesson'
> according to
> [Prefer|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Reminder-Prefer-link-tag-in-documentation-td42362.html].
>
> The location is: docs/learn-flink/fault_tolerance.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18437) Error message is not correct when using UNNEST
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147193#comment-17147193
]
mzz commented on FLINK-18437:
-
*Schema:*
.withSchema(new Schema()
.field("ip", Types.STRING())
.field("ts", Types.STRING())
.field("environment", schemaEnvironment)
.field("advs", ObjectArrayTypeInfo.getInfoFor(new
Array[Row](0).getClass, Types.ROW(Array("count","sid", "eventid","params"),
Array[TypeInformation[_]](Types.STRING(),Types.STRING(),
Types.STRING(),Types.ROW(Array("adid","adtype","ecpm"),Array[TypeInformation[_]](Types.STRING(),Types.STRING(),Types.STRING()))
)
when execute this sql:
{code:java}
val sql =
"""
|SELECT
|ip,
|ts,
|`count`,
|environment.access,
|environment.brand,
|sid,
|params.adid,
|eventid
|FROM aggs_test
|CROSS JOIN UNNEST(advs) AS t (`count`,sid, eventid,params)
|""".stripMargin
{code}
*log：*
11:31:54,379 WARN org.apache.flink.metrics.MetricGroup
- The operator name select: (ip, ts, count, environment.access AS access,
environment.brand AS brand, sid, params.adid AS adid, eventid) exceeded the 80
characters length limit and was truncated.
*Which parameter should I modify。THX。
*
> Error message is not correct when using UNNEST
> --
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Major
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
>
[jira] [Closed] (FLINK-17292) Translate Fault Tolerance training lesson to Chinese
[ https://issues.apache.org/jira/browse/FLINK-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-17292. --- Fix Version/s: 1.11.0 Resolution: Fixed - master (1.12.0): 81d34b9a0d3ff85ac1ae687c085d8b43fc2a468f - 1.11.0: d236f1790251e2a716cc5b000612789bebd79ed5 > Translate Fault Tolerance training lesson to Chinese > > > Key: FLINK-17292 > URL: https://issues.apache.org/jira/browse/FLINK-17292 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation / Training >Reporter: David Anderson >Assignee: Roc Marshal >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > This ticket is about translating the new tutorial in > docs/training/fault_tolerance.zh.md. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18202) Introduce Protobuf format
[ https://issues.apache.org/jira/browse/FLINK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147191#comment-17147191 ] liufangliang commented on FLINK-18202: -- Hi [~libenchao] , Currently protobuf only supports C++ dynamic compilation , clickhouse supports parsing input data according to proto file format . [https://clickhouse.tech/docs/en/interfaces/formats/#protobuf |https://clickhouse.tech/docs/en/interfaces/formats/#protobuf] You can see if it has reference value > Introduce Protobuf format > - > > Key: FLINK-18202 > URL: https://issues.apache.org/jira/browse/FLINK-18202 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Benchao Li >Priority: Major > Attachments: image-2020-06-15-17-18-03-182.png > > > PB[1] is a very famous and wildly used (de)serialization framework. The ML[2] > also has some discussions about this. It's a useful feature. > This issue maybe needs some designs, or a FLIP. > [1] [https://developers.google.com/protocol-buffers] > [2] [http://apache-flink.147419.n8.nabble.com/Flink-SQL-UDF-td3725.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #12727: [FLINK-17292][docs] Translate Fault Tolerance training lesson to Chinese
wuchong commented on a change in pull request #12727:
URL: https://github.com/apache/flink/pull/12727#discussion_r446595713
##
File path: docs/learn-flink/fault_tolerance.zh.md
##
@@ -29,180 +29,156 @@ under the License.
## State Backends
-The keyed state managed by Flink is a sort of sharded, key/value store, and
the working copy of each
-item of keyed state is kept somewhere local to the taskmanager responsible for
that key. Operator
-state is also local to the machine(s) that need(s) it. Flink periodically
takes persistent snapshots
-of all the state and copies these snapshots somewhere more durable, such as a
distributed file
-system.
+由 Flink 管理的 keyed state 是一种分片的键/值存储，并且 keyed state 的每一项的工作副本都保存在负责该键的
taskmanager 本地的某个地方。
+Operator state 对于需要它的机器节点来说也是本地的。Flink 定期获取所有状态的连续快照，并将这些快照复制到持久化的地方，例如分布式文件系统。
-In the event of the failure, Flink can restore the complete state of your
application and resume
-processing as though nothing had gone wrong.
+如果发生故障，Flink 可以恢复应用程序的完整状态并恢复处理，就如同没有出现过异常一样。
-This state that Flink manages is stored in a _state backend_. Two
implementations of state backends
-are available -- one based on RocksDB, an embedded key/value store that keeps
its working state on
-disk, and another heap-based state backend that keeps its working state in
memory, on the Java heap.
-This heap-based state backend comes in two flavors: the FsStateBackend that
persists its state
-snapshots to a distributed file system, and the MemoryStateBackend that uses
the JobManager's heap.
+Flink 管理的状态存储在 _state backend_。
+state backends 的两种实现 -- 一种是基于 RocksDB 内嵌 key/value 存储将其工作状态保存在磁盘上的，另一种基于堆的
state backend，将其工作状态保存在 Java 的 堆内存中。
+这种基于堆的 state backend 有两种方式：保存其状态快照到分布式文件系统的 FsStateBackend，以及使用 JobManager 堆的
MemoryStateBackend。
- Name
+ 名称
Working State
- State Backup
- Snapshotting
+ 状态备份
+ 快照
RocksDBStateBackend
- Local disk (tmp dir)
- Distributed file system
- Full / Incremental
+ 本地磁盘（tmp dir）
+ 分布式文件系统
+ 全量 / 增量
- Supports state larger than available memory
- Rule of thumb: 10x slower than heap-based backends
+ 支持大于内存大小的状态
+ 经验法则：比基于堆的后端慢10倍
FsStateBackend
JVM Heap
- Distributed file system
- Full
+ 分布式文件系统
+ 全量
- Fast, requires large heap
- Subject to GC
+ 快速，需要大的堆内存
+ 受限制于 GC
MemoryStateBackend
JVM Heap
JobManager JVM Heap
- Full
+ 全量
- Good for testing and experimentation with small state
(locally)
+ 适用于小状态（本地）的测试和实验
-When working with state kept in a heap-based state backend, accesses and
updates involve reading and
-writing objects on the heap. But for objects kept in the
`RocksDBStateBackend`, accesses and updates
-involve serialization and deserialization, and so are much more expensive. But
the amount of state
-you can have with RocksDB is limited only by the size of the local disk. Note
also that only the
-`RocksDBStateBackend` is able to do incremental snapshotting, which is a
significant benefit for
-applications with large amounts of slowly changing state.
+当使用基于堆的 state backend 保存状态时，访问和更新涉及在堆上读写对象。
+但是对于保存在 `RocksDBStateBackend` 中的对象，访问和更新涉及序列化和反序列化，所以会有更加巨大的开销。
+但 RocksDB 的状态量仅受本地磁盘大小的限制。
+还要注意，只有 `RocksDBStateBackend` 能够进行增量快照，这对于具有大量变化缓慢状态的应用程序来说是大有裨益的。
-All of these state backends are able to do asynchronous snapshotting, meaning
that they can take a
-snapshot without impeding the ongoing stream processing.
+所有这些 state backends 都能够异步执行快照，这意味着它们可以在不妨碍正在进行的流处理的情况下执行快照。
{% top %}
-## State Snapshots
+## 状态快照
-### Definitions
+### 定义
-* _Snapshot_ -- a generic term referring to a global, consistent image of the
state of a Flink job.
- A snapshot includes a pointer into each of the data sources (e.g., an offset
into a file or Kafka
- partition), as well as a copy of the state from each of the job's stateful
operators that resulted
- from having processed all of the events up to those positions in the sources.
-* _Checkpoint_ -- a snapshot taken automatically by Flink for the purpose of
being able to recover
- from faults. Checkpoints can be incremental, and are optimized for being
restored quickly.
-* _Externalized Checkpoint_ -- normally checkpoints are not intended to be
manipulated by users.
- Flink retains only the _n_-most-recent checkpoints (_n_ being configurable)
while a job is
- running, and deletes them when a job is cancelled. But you can configure
them to be retained
- instead, in which case you can manually resume from them.
-* _Savepoint_ -- a snapshot triggered manually by a user (or an API call) for
some operational
- purpose, such as a stateful
[jira] [Commented] (FLINK-18438) TaskManager start failed
[ https://issues.apache.org/jira/browse/FLINK-18438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147184#comment-17147184 ] Xintong Song commented on FLINK-18438: -- Hi [~JohnSiro], IIUC, you are trying to start a Flink standalone cluster on your Windows machine? Please be aware that Flink no longer supports `.bat` scripts on Windows (see FLINK-16683). > TaskManager start failed > > > Key: FLINK-18438 > URL: https://issues.apache.org/jira/browse/FLINK-18438 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.10.1 > Environment: Java: java version "1.8.0_101" > Java(TM) SE Runtime Environment (build 1.8.0_101-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) > Flink: 1.10.1 (flink-1.10.1-bin-scala_2.12.tgz) > OS: Windows 10 (1903) / 64-bits >Reporter: JohnSiro >Priority: Major > > > Error: in file xxx-taskexecutor-0-xxx.out is: > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. > Improperly specified VM option 'MaxMetaspaceSize=268435456 ' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on pull request #12727: [FLINK-17292][docs] Translate Fault Tolerance training lesson to Chinese
wuchong commented on pull request #12727: URL: https://github.com/apache/flink/pull/12727#issuecomment-650685104 Thanks for the great work @RocMarshal @klion26 @XBaith. Merging... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong closed pull request #12727: [FLINK-17292][docs] Translate Fault Tolerance training lesson to Chinese
wuchong closed pull request #12727: URL: https://github.com/apache/flink/pull/12727 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18362) Shipping jdk by shipping archive
[ https://issues.apache.org/jira/browse/FLINK-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147181#comment-17147181 ] Yang Wang commented on FLINK-18362: --- Great. Thanks [~gramo] . Could you open a PR for this ticket[1]? It is more convenient to review. [~kkl0u] Could you help to assign this ticket to [~gramo]? I think this is a valid feature. [1]. [https://github.com/apache/flink/pulls] > Shipping jdk by shipping archive > > > Key: FLINK-18362 > URL: https://issues.apache.org/jira/browse/FLINK-18362 > Project: Flink > Issue Type: Wish >Affects Versions: 1.10.1 >Reporter: Noah >Priority: Minor > Attachments: support-yarn.ship-archives.patch > > > Hello, > Our team are running flink cluster on YARN, and it works so well > h4. Functional requirements > Is there any option to ship archive to YARN applications? > h4. Backgrounds > Recently, one of job has been shut down with jdk8 version related issues. > https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6675699 > It's easy problem if we could set latest jdk on > `containerized.taskmanager.env.JAVA_HOME`. > However, cluster administrator said it's difficult to install the latest jdk > on all cluster machines. > > So, we planned to run a job on latest jdk that is shipped via shared > resources. > There's an option `yarn.ship-directories` but it's quite slow because jdk has > large number of files. > If Flink supports to ship archive such as `yarn.ship-archive`, we can ship > jdk archive to remote machines and use shipped jdk location as JAVA_HOME > (using `yarn.container-start-command-template` ) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18433) From the end-to-end performance test results, 1.11 has a regression
[ https://issues.apache.org/jira/browse/FLINK-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147180#comment-17147180 ] Zhijiang commented on FLINK-18433: -- Thanks for all the above analysis. >From my side, there are three main network related changes in this release: * Buffer management for reducing in-flight data for checkpoint: this improvement is not fully merged yet, and the current merged code would only effect on exactly-once mode, so i am convinced to dismiss this feature. * Netty reuses flink's network buffer for reducing memory copy and overhead: it should not bring any additional cost in theory. If it has any side effects, it should be reflected by micro-benchmark network stacks as well. I am also convinced to dismiss this feature. * Unaligned checkpoint: there are also some additional code execution even if UC is disabled as [~AHeise] mentioned above. I also checked the channel state restore part during StreamTask#beforeInvoke to confirm no effects in practice. But I have not checked every related changes yet to give a convinced answer now. If possible, maybe we can execute and compare one of the above performance testing cases with disabling checkpoint, then we might further narrow down the scope whether there are any regressions in basic network stack to dismiss checkpoint & state effects. Another suspicion is for JVM options changes which might effect the behavior of GC in practice. > From the end-to-end performance test results, 1.11 has a regression > --- > > Key: FLINK-18433 > URL: https://issues.apache.org/jira/browse/FLINK-18433 > Project: Flink > Issue Type: Bug > Components: API / Core, API / DataStream >Affects Versions: 1.11.0 > Environment: 3 machines > [|https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations_1.11/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java] >Reporter: Aihua Li >Priority: Major > Attachments: flink_11.log.gz > > > > I ran end-to-end performance tests between the Release-1.10 and Release-1.11. > the results were as follows: > |scenarioName|release-1.10|release-1.11| | > |OneInput_Broadcast_LazyFromSource_ExactlyOnce_10_rocksdb|46.175|43.8133|-5.11%| > |OneInput_Rescale_LazyFromSource_ExactlyOnce_100_heap|211.835|200.355|-5.42%| > |OneInput_Rebalance_LazyFromSource_ExactlyOnce_1024_rocksdb|1721.041667|1618.32|-5.97%| > |OneInput_KeyBy_LazyFromSource_ExactlyOnce_10_heap|46|43.615|-5.18%| > |OneInput_Broadcast_Eager_ExactlyOnce_100_rocksdb|212.105|199.688|-5.85%| > |OneInput_Rescale_Eager_ExactlyOnce_1024_heap|1754.64|1600.12|-8.81%| > |OneInput_Rebalance_Eager_ExactlyOnce_10_rocksdb|45.9167|43.0983|-6.14%| > |OneInput_KeyBy_Eager_ExactlyOnce_100_heap|212.0816667|200.727|-5.35%| > |OneInput_Broadcast_LazyFromSource_AtLeastOnce_1024_rocksdb|1718.245|1614.381667|-6.04%| > |OneInput_Rescale_LazyFromSource_AtLeastOnce_10_heap|46.12|43.5517|-5.57%| > |OneInput_Rebalance_LazyFromSource_AtLeastOnce_100_rocksdb|212.038|200.388|-5.49%| > |OneInput_KeyBy_LazyFromSource_AtLeastOnce_1024_heap|1762.048333|1606.408333|-8.83%| > |OneInput_Broadcast_Eager_AtLeastOnce_10_rocksdb|46.0583|43.4967|-5.56%| > |OneInput_Rescale_Eager_AtLeastOnce_100_heap|212.233|201.188|-5.20%| > |OneInput_Rebalance_Eager_AtLeastOnce_1024_rocksdb|1720.66|1616.85|-6.03%| > |OneInput_KeyBy_Eager_AtLeastOnce_10_heap|46.14|43.6233|-5.45%| > |TwoInputs_Broadcast_LazyFromSource_ExactlyOnce_100_rocksdb|156.918|152.957|-2.52%| > |TwoInputs_Rescale_LazyFromSource_ExactlyOnce_1024_heap|1415.511667|1300.1|-8.15%| > |TwoInputs_Rebalance_LazyFromSource_ExactlyOnce_10_rocksdb|34.2967|34.1667|-0.38%| > |TwoInputs_KeyBy_LazyFromSource_ExactlyOnce_100_heap|158.353|151.848|-4.11%| > |TwoInputs_Broadcast_Eager_ExactlyOnce_1024_rocksdb|1373.406667|1300.056667|-5.34%| > |TwoInputs_Rescale_Eager_ExactlyOnce_10_heap|34.5717|32.0967|-7.16%| > |TwoInputs_Rebalance_Eager_ExactlyOnce_100_rocksdb|158.655|147.44|-7.07%| > |TwoInputs_KeyBy_Eager_ExactlyOnce_1024_heap|1356.611667|1292.386667|-4.73%| > |TwoInputs_Broadcast_LazyFromSource_AtLeastOnce_10_rocksdb|34.01|33.205|-2.37%| > |TwoInputs_Rescale_LazyFromSource_AtLeastOnce_100_heap|149.588|145.997|-2.40%| > |TwoInputs_Rebalance_LazyFromSource_AtLeastOnce_1024_rocksdb|1359.74|1299.156667|-4.46%| > |TwoInputs_KeyBy_LazyFromSource_AtLeastOnce_10_heap|34.025|29.6833|-12.76%| > |TwoInputs_Broadcast_Eager_AtLeastOnce_100_rocksdb|157.303|151.4616667|-3.71%| > |TwoInputs_Rescale_Eager_AtLeastOnce_1024_heap|1368.74|1293.238333|-5.52%| > |TwoInputs_Rebalance_Eager_AtLeastOnce_10_rocksdb|34.325|33.285|-3.03%| >
[GitHub] [flink] RocMarshal commented on a change in pull request #12727: [FLINK-17292][docs] Translate Fault Tolerance training lesson to Chinese
RocMarshal commented on a change in pull request #12727:
URL: https://github.com/apache/flink/pull/12727#discussion_r446595140
##
File path: docs/learn-flink/fault_tolerance.zh.md
##
@@ -29,180 +29,156 @@ under the License.
## State Backends
-The keyed state managed by Flink is a sort of sharded, key/value store, and
the working copy of each
-item of keyed state is kept somewhere local to the taskmanager responsible for
that key. Operator
-state is also local to the machine(s) that need(s) it. Flink periodically
takes persistent snapshots
-of all the state and copies these snapshots somewhere more durable, such as a
distributed file
-system.
+由 Flink 管理的 keyed state 是一种分片的键/值存储，并且 keyed state 的每一项的工作副本都保存在负责该键的
taskmanager 本地的某个地方。
+Operator state 对于需要它的机器节点来说也是本地的。Flink 定期获取所有状态的连续快照，并将这些快照复制到持久化的地方，例如分布式文件系统。
-In the event of the failure, Flink can restore the complete state of your
application and resume
-processing as though nothing had gone wrong.
+如果发生故障，Flink 可以恢复应用程序的完整状态并恢复处理，就如同没有出现过异常一样。
-This state that Flink manages is stored in a _state backend_. Two
implementations of state backends
-are available -- one based on RocksDB, an embedded key/value store that keeps
its working state on
-disk, and another heap-based state backend that keeps its working state in
memory, on the Java heap.
-This heap-based state backend comes in two flavors: the FsStateBackend that
persists its state
-snapshots to a distributed file system, and the MemoryStateBackend that uses
the JobManager's heap.
+Flink 管理的状态存储在 _state backend_。
+state backends 的两种实现 -- 一种是基于 RocksDB 内嵌 key/value 存储将其工作状态保存在磁盘上的，另一种基于堆的
state backend，将其工作状态保存在 Java 的 堆内存中。
+这种基于堆的 state backend 有两种方式：保存其状态快照到分布式文件系统的 FsStateBackend，以及使用 JobManager 堆的
MemoryStateBackend。
- Name
+ 名称
Working State
- State Backup
- Snapshotting
+ 状态备份
+ 快照
RocksDBStateBackend
- Local disk (tmp dir)
- Distributed file system
- Full / Incremental
+ 本地磁盘（tmp dir）
+ 分布式文件系统
+ 全量 / 增量
- Supports state larger than available memory
- Rule of thumb: 10x slower than heap-based backends
+ 支持大于内存大小的状态
+ 经验法则：比基于堆的后端慢10倍
FsStateBackend
JVM Heap
- Distributed file system
- Full
+ 分布式文件系统
+ 全量
- Fast, requires large heap
- Subject to GC
+ 快速，需要大的堆内存
+ 受限制于 GC
MemoryStateBackend
JVM Heap
JobManager JVM Heap
- Full
+ 全量
- Good for testing and experimentation with small state
(locally)
+ 适用于小状态（本地）的测试和实验
-When working with state kept in a heap-based state backend, accesses and
updates involve reading and
-writing objects on the heap. But for objects kept in the
`RocksDBStateBackend`, accesses and updates
-involve serialization and deserialization, and so are much more expensive. But
the amount of state
-you can have with RocksDB is limited only by the size of the local disk. Note
also that only the
-`RocksDBStateBackend` is able to do incremental snapshotting, which is a
significant benefit for
-applications with large amounts of slowly changing state.
+当使用基于堆的 state backend 保存状态时，访问和更新涉及在堆上读写对象。
+但是对于保存在 `RocksDBStateBackend` 中的对象，访问和更新涉及序列化和反序列化，所以会有更加巨大的开销。
+但 RocksDB 的状态量仅受本地磁盘大小的限制。
+还要注意，只有 `RocksDBStateBackend` 能够进行增量快照，这对于具有大量变化缓慢状态的应用程序来说是大有裨益的。
-All of these state backends are able to do asynchronous snapshotting, meaning
that they can take a
-snapshot without impeding the ongoing stream processing.
+所有这些 state backends 都能够异步执行快照，这意味着它们可以在不妨碍正在进行的流处理的情况下执行快照。
{% top %}
-## State Snapshots
+## 状态快照
-### Definitions
+### 定义
-* _Snapshot_ -- a generic term referring to a global, consistent image of the
state of a Flink job.
- A snapshot includes a pointer into each of the data sources (e.g., an offset
into a file or Kafka
- partition), as well as a copy of the state from each of the job's stateful
operators that resulted
- from having processed all of the events up to those positions in the sources.
-* _Checkpoint_ -- a snapshot taken automatically by Flink for the purpose of
being able to recover
- from faults. Checkpoints can be incremental, and are optimized for being
restored quickly.
-* _Externalized Checkpoint_ -- normally checkpoints are not intended to be
manipulated by users.
- Flink retains only the _n_-most-recent checkpoints (_n_ being configurable)
while a job is
- running, and deletes them when a job is cancelled. But you can configure
them to be retained
- instead, in which case you can manually resume from them.
-* _Savepoint_ -- a snapshot triggered manually by a user (or an API call) for
some operational
- purpose, such as a stateful
[jira] [Commented] (FLINK-18202) Introduce Protobuf format
[ https://issues.apache.org/jira/browse/FLINK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147176#comment-17147176 ] liufangliang commented on FLINK-18202: -- [~tartarus] sorry for later reply . several fixed formats are currently implemented . Some complex formats cannot be parsed with Protobuf DynamicMessage . > Introduce Protobuf format > - > > Key: FLINK-18202 > URL: https://issues.apache.org/jira/browse/FLINK-18202 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Benchao Li >Priority: Major > Attachments: image-2020-06-15-17-18-03-182.png > > > PB[1] is a very famous and wildly used (de)serialization framework. The ML[2] > also has some discussions about this. It's a useful feature. > This issue maybe needs some designs, or a FLIP. > [1] [https://developers.google.com/protocol-buffers] > [2] [http://apache-flink.147419.n8.nabble.com/Flink-SQL-UDF-td3725.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] libenchao commented on pull request #12748: [FLINK-18324][docs-zh] Translate updated data type into Chinese
libenchao commented on pull request #12748: URL: https://github.com/apache/flink/pull/12748#issuecomment-650684436 @liyubin117 Thanks for the updating. The pr LGTM now. Could you rebase your commits into one? TIP: we usually uses `git rebase` instead of `git merge` to resolve conflicts with master branch. (actually I don't know how to merge your pr if there is a merge commit in it) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18393) Translate "Canal Format" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-18393. --- Fix Version/s: 1.11.0 Resolution: Fixed - master (1.12.0): f1df8ba053e6f3aa22d4c596de8950ff1a6e5a58 - 1.11.0: a55aa1dbfcd8f71fb70de68720f99df0031ee3ac > Translate "Canal Format" page into Chinese > -- > > Key: FLINK-18393 > URL: https://issues.apache.org/jira/browse/FLINK-18393 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: Zhiye Zou >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/formats/canal.html > The markdown file is located in > flink/docs/dev/table/connectors/formats/canal.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-18439) Update sql client jar url in docs
[ https://issues.apache.org/jira/browse/FLINK-18439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-18439: --- Assignee: Leonard Xu > Update sql client jar url in docs > - > > Key: FLINK-18439 > URL: https://issues.apache.org/jira/browse/FLINK-18439 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Fix For: 1.11.0 > > > the sql client jar url should be： > [https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch6] > ... > but current is ： > [https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-elasticsearch6...] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #12771: [FLINK-18393] [docs-zh] Translate "Canal Format" page into Chinese
wuchong closed pull request #12771: URL: https://github.com/apache/flink/pull/12771 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12777: [FLINK-18262][python][e2e] Fix the unstable e2e tests of pyflink.
flinkbot edited a comment on pull request #12777: URL: https://github.com/apache/flink/pull/12777#issuecomment-650680478 ## CI report: * ef52a75fdbcddde7ac55eea60d1b6db3bbcd6056 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4066) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18439) Update sql client jar url in docs
Leonard Xu created FLINK-18439: -- Summary: Update sql client jar url in docs Key: FLINK-18439 URL: https://issues.apache.org/jira/browse/FLINK-18439 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.11.0 Reporter: Leonard Xu Fix For: 1.11.0 the sql client jar url should be： [https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch6] ... but current is ： [https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-elasticsearch6...] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18437) Error message is not correct when using UNNEST
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mzz updated FLINK-18437:
Priority: Major (was: Minor)
> Error message is not correct when using UNNEST
> --
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Test
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Major
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateUnnest(SqlValidatorImpl.java:3126)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3108)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3093)
>
[jira] [Updated] (FLINK-18437) Error message is not correct when using UNNEST
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mzz updated FLINK-18437:
Issue Type: Improvement (was: Test)
> Error message is not correct when using UNNEST
> --
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Major
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateUnnest(SqlValidatorImpl.java:3126)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3108)
> at
>
[jira] [Commented] (FLINK-18164) null <> 'str' should be true
[
https://issues.apache.org/jira/browse/FLINK-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147170#comment-17147170
]
Benchao Li commented on FLINK-18164:
[~Leonard Xu] currently in Flink, we don't have any configurations to control
this behavior. It's determined by our code generation logic. You can check
{{ScalarOperatorGens#generateEquals.}}
> null <> 'str' should be true
>
>
> Key: FLINK-18164
> URL: https://issues.apache.org/jira/browse/FLINK-18164
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Benchao Li
>Priority: Major
>
> Currently, if we compare null with other literals, the result will always be
> false.
> It's because the code gen always gives a default value (false) for the
> result. And I think it's a bug if `null <> 'str'` is false.
> It's reported from user-zh:
> http://apache-flink.147419.n8.nabble.com/flink-sql-null-false-td3640.html
> CC [~jark] [~ykt836]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18438) TaskManager start failed
[ https://issues.apache.org/jira/browse/FLINK-18438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JohnSiro updated FLINK-18438: - Environment: Java: java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) Flink: 1.10.1 (flink-1.10.1-bin-scala_2.12.tgz) OS: Windows 10 (1903) / 64-bits was: Env: Flink: 1.10.1 OS: Windows 10 1903 64-bit > TaskManager start failed > > > Key: FLINK-18438 > URL: https://issues.apache.org/jira/browse/FLINK-18438 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.10.1 > Environment: Java: java version "1.8.0_101" > Java(TM) SE Runtime Environment (build 1.8.0_101-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) > Flink: 1.10.1 (flink-1.10.1-bin-scala_2.12.tgz) > OS: Windows 10 (1903) / 64-bits >Reporter: JohnSiro >Priority: Major > > > Error: in file xxx-taskexecutor-0-xxx.out is: > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. > Improperly specified VM option 'MaxMetaspaceSize=268435456 ' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18386) Translate "Print SQL Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-18386. --- Fix Version/s: 1.11.0 Resolution: Fixed Fixed via: - master (1.12.0): 1c5f5e762000ec2465dfaaee61031a32ca0e7354 - 1.11.0: 87925de56cbc87e4950966dcad204e3eefc5b0e9 > Translate "Print SQL Connector" page into Chinese > - > > Key: FLINK-18386 > URL: https://issues.apache.org/jira/browse/FLINK-18386 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: houmaozheng >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/print.html > The markdown file is located in flink/docs/dev/table/connectors/print.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18438) TaskManager start failed
[ https://issues.apache.org/jira/browse/FLINK-18438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147168#comment-17147168 ] JohnSiro commented on FLINK-18438: -- $ java -version java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) > TaskManager start failed > > > Key: FLINK-18438 > URL: https://issues.apache.org/jira/browse/FLINK-18438 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.10.1 > Environment: Env: > Flink: 1.10.1 > OS: Windows 10 1903 64-bit >Reporter: JohnSiro >Priority: Major > > > Error: in file xxx-taskexecutor-0-xxx.out is: > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. > Improperly specified VM option 'MaxMetaspaceSize=268435456 ' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18438) TaskManager start failed
JohnSiro created FLINK-18438: Summary: TaskManager start failed Key: FLINK-18438 URL: https://issues.apache.org/jira/browse/FLINK-18438 Project: Flink Issue Type: Bug Components: Client / Job Submission Affects Versions: 1.10.1 Environment: Env: Flink: 1.10.1 OS: Windows 10 1903 64-bit Reporter: JohnSiro Error: in file xxx-taskexecutor-0-xxx.out is: Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Improperly specified VM option 'MaxMetaspaceSize=268435456 ' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12777: [FLINK-18262][python][e2e] Fix the unstable e2e tests of pyflink.
flinkbot commented on pull request #12777: URL: https://github.com/apache/flink/pull/12777#issuecomment-650680478 ## CI report: * ef52a75fdbcddde7ac55eea60d1b6db3bbcd6056 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18164) null <> 'str' should be true
[ https://issues.apache.org/jira/browse/FLINK-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147164#comment-17147164 ] Leonard Xu commented on FLINK-18164: Hi， [~libenchao] I see you closed this, Is that current behavior is right like my posted after you confirmed? > null <> 'str' should be true > > > Key: FLINK-18164 > URL: https://issues.apache.org/jira/browse/FLINK-18164 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Reporter: Benchao Li >Priority: Major > > Currently, if we compare null with other literals, the result will always be > false. > It's because the code gen always gives a default value (false) for the > result. And I think it's a bug if `null <> 'str'` is false. > It's reported from user-zh: > http://apache-flink.147419.n8.nabble.com/flink-sql-null-false-td3640.html > CC [~jark] [~ykt836] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18437) Error message is not correct when using UNNEST
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-18437:
Summary: Error message is not correct when using UNNEST (was: Table API
has no Functions like sparkSQL explode)
> Error message is not correct when using UNNEST
> --
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Test
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Minor
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateUnnest(SqlValidatorImpl.java:3126)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3108)
> at
[jira] [Commented] (FLINK-18437) Table API has no Functions like sparkSQL explode
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147163#comment-17147163
]
Jark Wu commented on FLINK-18437:
-
You are right. This might be bug in the error message.
> Table API has no Functions like sparkSQL explode
> -
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Test
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Minor
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateUnnest(SqlValidatorImpl.java:3126)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3108)
> at
>
[GitHub] [flink] wuchong closed pull request #12758: [FLINK-18386][docs-zh] Translate "Print SQL Connector" page into Chinese
wuchong closed pull request #12758: URL: https://github.com/apache/flink/pull/12758 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18437) Table API has no Functions like sparkSQL explode
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mzz updated FLINK-18437:
Issue Type: Test (was: Bug)
> Table API has no Functions like sparkSQL explode
> -
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Test
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Minor
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateUnnest(SqlValidatorImpl.java:3126)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3108)
> at
>
[jira] [Commented] (FLINK-18437) Table API has no Functions like sparkSQL explode
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147162#comment-17147162
]
mzz commented on FLINK-18437:
-
[~jark]
Thank you for your advice，Problem has solved。
This error log made me to go the wrong direction。`environment`('access',
'brand') is a json object，`advs `(["count","eventid"]) is a json array，error
log is "table has 2 columns ('access', 'brand'), whereas alias list has 1
columns"。
> Table API has no Functions like sparkSQL explode
> -
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Major
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
>
[jira] [Updated] (FLINK-18437) Table API has no Functions like sparkSQL explode
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mzz updated FLINK-18437:
Priority: Minor (was: Major)
> Table API has no Functions like sparkSQL explode
> -
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Minor
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateUnnest(SqlValidatorImpl.java:3126)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3108)
> at
>
[jira] [Closed] (FLINK-18043) Translate "Scala API Extensions" page of "DataStream API" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yubin Li closed FLINK-18043.
Resolution: Abandoned
> Translate "Scala API Extensions" page of "DataStream API" into Chinese
> --
>
> Key: FLINK-18043
> URL: https://issues.apache.org/jira/browse/FLINK-18043
> Project: Flink
> Issue Type: Task
> Components: chinese-translation, Documentation
>Reporter: Yubin Li
>Priority: Major
> Fix For: 1.12.0
>
>
> The page url is
> [https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/scala_api_extensions.html]
> The markdown file is located in {{flink/docs/dev/scala_api_extensions.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #12772: [FLINK-18394][docs-zh] Translate "Parquet Format" page into Chinese
JingsongLi commented on a change in pull request #12772: URL: https://github.com/apache/flink/pull/12772#discussion_r446592458 ## File path: docs/dev/table/connectors/formats/parquet.zh.md ## @@ -186,4 +185,4 @@ The following table lists the type mapping from Flink type to Parquet type. -Attention Composite data type: Array, Map and Row are not supported. +注意 复合数据类型（Array、Map 与 Row）不被支持。 Review comment: 暂不支持复合数据类型（Array、Map 与 Row） This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18262) PyFlink end-to-end test stalls
[
https://issues.apache.org/jira/browse/FLINK-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147150#comment-17147150
]
Wei Zhong edited comment on FLINK-18262 at 6/28/20, 2:46 AM:
-
Hi [~dian.fu], [~rmetzger] It seems the e2e test hangs when extracting the
flink tar file to prepare flink on the yarn docker container:
[https://dev.azure.com/weizhong0618/Flink/_build/results?buildId=251=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=2f5b54d0-1d28-5b01-d344-aa50ffe0cdf8]
It may be a bug of the "tar" command because the test never hangs here after
replace the "tar" command with "zip/unzip" command. See all the builds which
named "trigger test5" and "trigger test6"(about 75 builds):
[https://dev.azure.com/weizhong0618/Flink/_build?definitionId=2]
Based on past experience, if the problem persists, there will be at least one
test hang caused by the pyflink e2e test in every 40 builds.
So I think we can fix this via replace the "tar" command with the "zip/unzip"
command. What do you think?
was (Author: zhongwei):
Hi [~dian.fu], [~rmetzger] It seems the e2e test hangs when extracting the
flink tar file to prepare flink on the yarn docker container:
[https://dev.azure.com/weizhong0618/Flink/_build/results?buildId=251=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=2f5b54d0-1d28-5b01-d344-aa50ffe0cdf8]
It may be a bug of the "tar" command because the test never hangs here after
replace the "tar" command with "zip/unzip" command. See all the builds which
named "trigger test5" and "trigger test6"(about 75 builds):
[https://dev.azure.com/weizhong0618/Flink/_build?definitionId=2]
Based on past experience, if the problem persists, there will be at least one
test failure caused by the pyflink e2e test hang in every 40 tests.
So I think we can fix this via replace the "tar" command with the "zip/unizp"
command. What do you think?
> PyFlink end-to-end test stalls
> --
>
> Key: FLINK-18262
> URL: https://issues.apache.org/jira/browse/FLINK-18262
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3299=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-11T17:25:37.2919869Z We now have 2 NodeManagers up.
> 2020-06-11T18:22:11.3398295Z ##[error]The operation was canceled.
> 2020-06-11T18:22:11.3411819Z ##[section]Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12777: [FLINK-18262][python][e2e] Fix the unstable e2e tests of pyflink.
flinkbot commented on pull request #12777: URL: https://github.com/apache/flink/pull/12777#issuecomment-650676328 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ef52a75fdbcddde7ac55eea60d1b6db3bbcd6056 (Sun Jun 28 02:46:52 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18262).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18262) PyFlink end-to-end test stalls
[
https://issues.apache.org/jira/browse/FLINK-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18262:
---
Labels: pull-request-available test-stability (was: test-stability)
> PyFlink end-to-end test stalls
> --
>
> Key: FLINK-18262
> URL: https://issues.apache.org/jira/browse/FLINK-18262
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3299=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-11T17:25:37.2919869Z We now have 2 NodeManagers up.
> 2020-06-11T18:22:11.3398295Z ##[error]The operation was canceled.
> 2020-06-11T18:22:11.3411819Z ##[section]Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] WeiZhong94 opened a new pull request #12777: [FLINK-18262][python][e2e] Fix the unstable e2e tests of pyflink.
WeiZhong94 opened a new pull request #12777: URL: https://github.com/apache/flink/pull/12777 ## What is the purpose of the change *This pull request fixes the test hang problem of the pyflink e2e tests via replace the tar command with zip/unzip command in `common_yarn_docker.sh`.* ## Brief change log - *Replace the tar command with zip/unzip command in `common_yarn_docker.sh`.* ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18396) Translate "Formats Overview" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147158#comment-17147158 ] Jark Wu edited comment on FLINK-18396 at 6/28/20, 2:39 AM: --- - master (1.12.0): 5a134905fb1c00bd5ef5b78aa176beb658e5b48c - 1.11.0: 1e23c61669f48bdf600f8de7b9ced06b00f66adb was (Author: jark): - master (1.12.0): 5a134905fb1c00bd5ef5b78aa176beb658e5b48c - 1.11.1: 1e23c61669f48bdf600f8de7b9ced06b00f66adb > Translate "Formats Overview" page into Chinese > -- > > Key: FLINK-18396 > URL: https://issues.apache.org/jira/browse/FLINK-18396 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/formats/ > The markdown file is located in > flink/docs/dev/table/connectors/formats/index.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18414) Kafka Json connector in Table API support more option
[
https://issues.apache.org/jira/browse/FLINK-18414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-18414:
Fix Version/s: (was: 1.11.1)
> Kafka Json connector in Table API support more option
> -
>
> Key: FLINK-18414
> URL: https://issues.apache.org/jira/browse/FLINK-18414
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Formats (JSON, Avro, Parquet, ORC,
> SequenceFile), Table SQL / Ecosystem
>Affects Versions: 1.10.1
>Reporter: DuBin
>Priority: Major
>
> Currently, the Flink use a
> 'org.apache.flink.formats.json.JsonRowDeserializationSchema' to deserialize
> the record into Row if we define the Kafka Json Table Source.
> But the parser is hard-coded in the class :
> private final ObjectMapper objectMapper = new ObjectMapper();
> Imagine that the Json data source contains data like this:
> {"a":NaN,"b":1.2}
> or it contains some dirty data, it will throw exception in the deserialize
> function all the time, because Kafka do not have a schema validation on Json
> format.
>
> So can we add more options in the
> 'org.apache.flink.formats.json.JsonRowFormatFactory' , in the
> 'org.apache.flink.formats.json.JsonRowFormatFactory#createDeserializationSchema'?
> e.g. add more option for the objectMapper, some dirty data handler(just
> return an empty row, defined by the user)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18396) Translate "Formats Overview" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18396: Fix Version/s: (was: 1.11.1) 1.11.0 > Translate "Formats Overview" page into Chinese > -- > > Key: FLINK-18396 > URL: https://issues.apache.org/jira/browse/FLINK-18396 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/formats/ > The markdown file is located in > flink/docs/dev/table/connectors/formats/index.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18396) Translate "Formats Overview" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-18396. --- Fix Version/s: 1.11.1 Resolution: Fixed - master (1.12.0): 5a134905fb1c00bd5ef5b78aa176beb658e5b48c - 1.11.1: 1e23c61669f48bdf600f8de7b9ced06b00f66adb > Translate "Formats Overview" page into Chinese > -- > > Key: FLINK-18396 > URL: https://issues.apache.org/jira/browse/FLINK-18396 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.1 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/formats/ > The markdown file is located in > flink/docs/dev/table/connectors/formats/index.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18262) PyFlink end-to-end test stalls
[
https://issues.apache.org/jira/browse/FLINK-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147150#comment-17147150
]
Wei Zhong edited comment on FLINK-18262 at 6/28/20, 2:33 AM:
-
Hi [~dian.fu], [~rmetzger] It seems the e2e test hangs when extracting the
flink tar file to prepare flink on the yarn docker container:
[https://dev.azure.com/weizhong0618/Flink/_build/results?buildId=251=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=2f5b54d0-1d28-5b01-d344-aa50ffe0cdf8]
It may be a bug of the "tar" command because the test never hangs here after
replace the "tar" command with "zip/unzip" command. See all the builds which
named "trigger test5" and "trigger test6"(about 75 builds):
[https://dev.azure.com/weizhong0618/Flink/_build?definitionId=2]
Based on past experience, if the problem persists, there will be at least one
test failure caused by the pyflink e2e test hang in every 40 tests.
So I think we can fix this via replace the "tar" command with the "zip/unizp"
command. What do you think?
was (Author: zhongwei):
Hi [~dian.fu], [~rmetzger] It seems the e2e test hangs when extracting the
flink tar file to prepare flink on the yarn docker container:
[https://dev.azure.com/weizhong0618/Flink/_build/results?buildId=251=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=2f5b54d0-1d28-5b01-d344-aa50ffe0cdf8]
It may be a bug of the "tar" command because the test never hangs here after
replace the "tar" command with "zip/unzip" command. See all the builds which
named "trigger test5" and "trigger test6"(about 75 builds):
[https://dev.azure.com/weizhong0618/Flink/_build?definitionId=2]
So I think we can fix this via replace the "tar" command with the "zip/unizp"
command. What do you think?
> PyFlink end-to-end test stalls
> --
>
> Key: FLINK-18262
> URL: https://issues.apache.org/jira/browse/FLINK-18262
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3299=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-11T17:25:37.2919869Z We now have 2 NodeManagers up.
> 2020-06-11T18:22:11.3398295Z ##[error]The operation was canceled.
> 2020-06-11T18:22:11.3411819Z ##[section]Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #12775: [FLINK-18396][dosc]Translate "Formats Overview" page into Chinese
wuchong closed pull request #12775: URL: https://github.com/apache/flink/pull/12775 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #12775: [FLINK-18396][dosc]Translate "Formats Overview" page into Chinese
lsyldliu commented on pull request #12775: URL: https://github.com/apache/flink/pull/12775#issuecomment-650674761 > Thanks for the contribution @lsyldliu , LGTM. > > Minor tip: please also add JIRA ID and component name to first commit message, e.g. > > `[[FLINK-18396](https://issues.apache.org/jira/browse/FLINK-18396)][docs-zh] Translate "Formats Overview" page into Chinese` . got it, tks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18437) Table API has no Functions like sparkSQL explode
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147151#comment-17147151
]
Jark Wu commented on FLINK-18437:
-
The exception message has told the root cause "List of column aliases must have
same degree as table; table has 2 columns ('access', 'brand'), whereas alias
list has 1 columns".
Your "advs" has two columns "count" and "eventid", so you can try this:
{code:sql}
SELECT
ip,
ts,
environment,
count,
eventid
FROM aggs_test
CROSS JOIN UNNEST(advs) AS t (count, eventid)
{code}
> Table API has no Functions like sparkSQL explode
> -
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Major
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
[jira] [Comment Edited] (FLINK-18262) PyFlink end-to-end test stalls
[
https://issues.apache.org/jira/browse/FLINK-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147150#comment-17147150
]
Wei Zhong edited comment on FLINK-18262 at 6/28/20, 2:30 AM:
-
Hi [~dian.fu], [~rmetzger] It seems the e2e test hangs when extracting the
flink tar file to prepare flink on the yarn docker container:
[https://dev.azure.com/weizhong0618/Flink/_build/results?buildId=251=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=2f5b54d0-1d28-5b01-d344-aa50ffe0cdf8]
It may be a bug of the "tar" command because the test never hangs here after
replace the "tar" command with "zip/unzip" command. See all the builds which
named "trigger test5" and "trigger test6"(about 75 builds):
[https://dev.azure.com/weizhong0618/Flink/_build?definitionId=2]
So I think we can fix this via replace the "tar" command with the "zip/unizp"
command. What do you think?
was (Author: zhongwei):
Hi [~dian.fu], [~rmetzger] It seems the e2e test hangs when extracting the
flink tar file to prepare flink on the yarn docker container:
[https://dev.azure.com/weizhong0618/Flink/_build/results?buildId=251=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=2f5b54d0-1d28-5b01-d344-aa50ffe0cdf8]
It may be a bug of the "tar" command because the test never hangs here after
replace the "tar" command with "zip/unzip" command. See all the builds which
named "trigger test5" and "trigger test6":
[https://dev.azure.com/weizhong0618/Flink/_build?definitionId=2]
So I think we can fix this via replace the "tar" command with the "zip/unizp"
command. What do you think?
> PyFlink end-to-end test stalls
> --
>
> Key: FLINK-18262
> URL: https://issues.apache.org/jira/browse/FLINK-18262
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3299=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-11T17:25:37.2919869Z We now have 2 NodeManagers up.
> 2020-06-11T18:22:11.3398295Z ##[error]The operation was canceled.
> 2020-06-11T18:22:11.3411819Z ##[section]Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18262) PyFlink end-to-end test stalls
[
https://issues.apache.org/jira/browse/FLINK-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147150#comment-17147150
]
Wei Zhong commented on FLINK-18262:
---
Hi [~dian.fu], [~rmetzger] It seems the e2e test hangs when extracting the
flink tar file to prepare flink on the yarn docker container:
[https://dev.azure.com/weizhong0618/Flink/_build/results?buildId=251=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=2f5b54d0-1d28-5b01-d344-aa50ffe0cdf8]
It may be a bug of the "tar" command because the test never hangs here after
replace the "tar" command with "zip/unzip" command. See all the builds which
named "trigger test5" and "trigger test6":
[https://dev.azure.com/weizhong0618/Flink/_build?definitionId=2]
So I think we can fix this via replace the "tar" command with the "zip/unizp"
command. What do you think?
> PyFlink end-to-end test stalls
> --
>
> Key: FLINK-18262
> URL: https://issues.apache.org/jira/browse/FLINK-18262
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3299=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-11T17:25:37.2919869Z We now have 2 NodeManagers up.
> 2020-06-11T18:22:11.3398295Z ##[error]The operation was canceled.
> 2020-06-11T18:22:11.3411819Z ##[section]Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18437) Table API has no Functions like sparkSQL explode
[
https://issues.apache.org/jira/browse/FLINK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147148#comment-17147148
]
mzz commented on FLINK-18437:
-
execute ：select * from table，and print schema：
root
|-- ip: STRING
|-- ts: STRING
|-- environment: ROW<`access` STRING, `brand` STRING>
|-- advs: ARRAY>
I want to split the array into multiple rows，thx
> Table API has no Functions like sparkSQL explode
> -
>
> Key: FLINK-18437
> URL: https://issues.apache.org/jira/browse/FLINK-18437
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.9.3
>Reporter: mzz
>Priority: Major
>
> .withSchema(new Schema()
> .field("ip", Types.STRING())
> .field("ts", Types.STRING())
> .field("environment", Types.ROW(Array("access", "brand"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
> .field("advs", ObjectArrayTypeInfo.getInfoFor(new
> Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
> Array[TypeInformation[_]](Types.STRING(), Types.STRING
> )
> .inAppendMode()
> .registerTableSource("aggs_test")
> The code above is dataSchema，i tried this way
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
> when i execute this sql:
> val sql1 =
> """
> |SELECT
> |ip,
> |ts,
> |environment,
> |adv
> |FROM aggs_test
> |CROSS JOIN UNNEST(advs) AS t (adv)
> |""".stripMargin
> It report an error：
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> SQL validation failed. From line 8, column 31 to line 8, column 33: List of
> column aliases must have same degree as table; table has 2 columns ('access',
> 'brand'), whereas alias list has 1 columns
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
> at
> org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
> at
> QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
> column 31 to line 8, column 33: List of column aliases must have same degree
> as table; table has 2 columns ('access', 'brand'), whereas alias list has 1
> columns
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
> at
> org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
> at
> org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
> at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
> at
> org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
> at
> org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
> at
>
[jira] [Created] (FLINK-18437) Table API has no Functions like sparkSQL explode
mzz created FLINK-18437:
---
Summary: Table API has no Functions like sparkSQL explode
Key: FLINK-18437
URL: https://issues.apache.org/jira/browse/FLINK-18437
Project: Flink
Issue Type: Bug
Components: API / DataStream
Affects Versions: 1.9.3
Reporter: mzz
.withSchema(new Schema()
.field("ip", Types.STRING())
.field("ts", Types.STRING())
.field("environment", Types.ROW(Array("access", "brand"),
Array[TypeInformation[_]](Types.STRING(), Types.STRING)))
.field("advs", ObjectArrayTypeInfo.getInfoFor(new
Array[Row](0).getClass, Types.ROW(Array("count", "eventid"),
Array[TypeInformation[_]](Types.STRING(), Types.STRING
)
.inAppendMode()
.registerTableSource("aggs_test")
The code above is dataSchema，i tried this way
[https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html|http://example.com]，but
when i execute this sql:
val sql1 =
"""
|SELECT
|ip,
|ts,
|environment,
|adv
|FROM aggs_test
|CROSS JOIN UNNEST(advs) AS t (adv)
|""".stripMargin
It report an error：
Exception in thread "main" org.apache.flink.table.api.ValidationException: SQL
validation failed. From line 8, column 31 to line 8, column 33: List of column
aliases must have same degree as table; table has 2 columns ('access',
'brand'), whereas alias list has 1 columns
at
org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:128)
at
org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:83)
at
org.apache.flink.table.planner.StreamPlanner.parse(StreamPlanner.scala:115)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:298)
at
QM.COM.Flink.KafakaHelper.FlinkTableConnKafka$.main(FlinkTableConnKafka.scala:88)
at
QM.COM.Flink.KafakaHelper.FlinkTableConnKafka.main(FlinkTableConnKafka.scala)
Caused by: org.apache.calcite.runtime.CalciteContextException: From line 8,
column 31 to line 8, column 33: List of column aliases must have same degree as
table; table has 2 columns ('access', 'brand'), whereas alias list has 1 columns
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:824)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:809)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4807)
at
org.apache.calcite.sql.validate.AliasNamespace.validateImpl(AliasNamespace.java:86)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
at
org.apache.calcite.sql.validate.AbstractNamespace.getRowType(AbstractNamespace.java:115)
at
org.apache.calcite.sql.validate.AliasNamespace.getRowType(AliasNamespace.java:41)
at
org.apache.calcite.sql.validate.DelegatingScope.resolveInNamespace(DelegatingScope.java:101)
at org.apache.calcite.sql.validate.ListScope.resolve(ListScope.java:191)
at
org.apache.calcite.sql.validate.ListScope.findQualifyingTableNames(ListScope.java:156)
at
org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:238)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5699)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5684)
at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5291)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateUnnest(SqlValidatorImpl.java:3126)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3108)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3093)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateJoin(SqlValidatorImpl.java:3158)
at
org.apache.flink.table.calcite.FlinkCalciteSqlValidator.validateJoin(FlinkCalciteSqlValidator.scala:67)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3102)
at
[GitHub] [flink] JingsongLi commented on a change in pull request #12682: [FLINK-18320][hive] Fix NOTICE and license files for flink-sql-connec…
JingsongLi commented on a change in pull request #12682: URL: https://github.com/apache/flink/pull/12682#discussion_r446588819 ## File path: flink-connectors/flink-sql-connector-hive-1.2.2/src/main/resources/META-INF/NOTICE ## @@ -8,6 +8,41 @@ This project bundles the following dependencies under the Apache Software Licens - org.apache.hive:hive-exec:1.2.2 - org.apache.hive:hive-metastore:1.2.2 +- org.apache.hive:hive-common:1.2.2 +- org.apache.hive:hive-serde:1.2.2 +- org.apache.hive.shims:hive-shims-0.20S:1.2.2 +- org.apache.hive.shims:hive-shims-0.23:1.2.2 +- org.apache.hive.shims:hive-shims-common:1.2.2 +- org.apache.hive:spark-client:1.2.2 +- com.twitter:parquet-hadoop-bundle:1.6.0 +- org.apache.thrift:libthrift:0.9.2 - org.apache.thrift:libfb303:0.9.2 - org.apache.orc:orc-core:1.4.3 - io.airlift:aircompressor:0.8 +- commons-lang:commons-lang:2.6 +- org.apache.commons:commons-lang3:3.1 +- org.apache.avro:avro:1.7.5 +- org.apache.avro:avro-mapred:1.7.5 +- com.googlecode.javaewah:JavaEWAH:0.3.2 +- org.iq80.snappy:snappy:0.2 +- org.codehaus.jackson:jackson-core-asl:1.9.2 +- org.codehaus.jackson:jackson-mapper-asl:1.9.2 +- com.google.guava:guava:14.0.1 +- net.sf.opencsv:opencsv:2.3 +- joda-time:joda-time:2.5 +- org.objenesis:objenesis:1.2 + +This project bundles the following dependencies under the BSD license. +See bundled license files for details. + +- com.esotericsoftware.kryo:kryo:2.22 +- org.jodd:jodd-core:3.5.2 +- javolution:javolution:5.5.1 +- com.google.protobuf:protobuf-java:2.5.0 +- com.esotericsoftware.minlog:minlog:1.2 +- com.esotericsoftware.reflectasm:reflectasm:1.07 + +This project bundles the following dependencies under the JSON license. Review comment: Thanks @lirui-apache and @zentol for your discussion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18433) From the end-to-end performance test results, 1.11 has a regression
[
https://issues.apache.org/jira/browse/FLINK-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147142#comment-17147142
]
Congxian Qiu(klion26) commented on FLINK-18433:
---
Thanks for running these tests [~AHeise]. I guess we have to wait for [~Aihua].
From my previous experience, the result between the bounded-e2e test and the
unbounded-e2e test may be different, because of the impact of setup things.
I checked the log, found that the checkpoint expired before complete, Job fails
because `{{Exceeded checkpoint tolerable failure threshold`}}, task received
abort checkpoint message. I'm not very sure if we increase the tolerate
checkpoint failure threshold can improve the performance or not.
> From the end-to-end performance test results, 1.11 has a regression
> ---
>
> Key: FLINK-18433
> URL: https://issues.apache.org/jira/browse/FLINK-18433
> Project: Flink
> Issue Type: Bug
> Components: API / Core, API / DataStream
>Affects Versions: 1.11.0
> Environment: 3 machines
> [|https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations_1.11/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java]
>Reporter: Aihua Li
>Priority: Major
> Attachments: flink_11.log.gz
>
>
>
> I ran end-to-end performance tests between the Release-1.10 and Release-1.11.
> the results were as follows:
> |scenarioName|release-1.10|release-1.11| |
> |OneInput_Broadcast_LazyFromSource_ExactlyOnce_10_rocksdb|46.175|43.8133|-5.11%|
> |OneInput_Rescale_LazyFromSource_ExactlyOnce_100_heap|211.835|200.355|-5.42%|
> |OneInput_Rebalance_LazyFromSource_ExactlyOnce_1024_rocksdb|1721.041667|1618.32|-5.97%|
> |OneInput_KeyBy_LazyFromSource_ExactlyOnce_10_heap|46|43.615|-5.18%|
> |OneInput_Broadcast_Eager_ExactlyOnce_100_rocksdb|212.105|199.688|-5.85%|
> |OneInput_Rescale_Eager_ExactlyOnce_1024_heap|1754.64|1600.12|-8.81%|
> |OneInput_Rebalance_Eager_ExactlyOnce_10_rocksdb|45.9167|43.0983|-6.14%|
> |OneInput_KeyBy_Eager_ExactlyOnce_100_heap|212.0816667|200.727|-5.35%|
> |OneInput_Broadcast_LazyFromSource_AtLeastOnce_1024_rocksdb|1718.245|1614.381667|-6.04%|
> |OneInput_Rescale_LazyFromSource_AtLeastOnce_10_heap|46.12|43.5517|-5.57%|
> |OneInput_Rebalance_LazyFromSource_AtLeastOnce_100_rocksdb|212.038|200.388|-5.49%|
> |OneInput_KeyBy_LazyFromSource_AtLeastOnce_1024_heap|1762.048333|1606.408333|-8.83%|
> |OneInput_Broadcast_Eager_AtLeastOnce_10_rocksdb|46.0583|43.4967|-5.56%|
> |OneInput_Rescale_Eager_AtLeastOnce_100_heap|212.233|201.188|-5.20%|
> |OneInput_Rebalance_Eager_AtLeastOnce_1024_rocksdb|1720.66|1616.85|-6.03%|
> |OneInput_KeyBy_Eager_AtLeastOnce_10_heap|46.14|43.6233|-5.45%|
> |TwoInputs_Broadcast_LazyFromSource_ExactlyOnce_100_rocksdb|156.918|152.957|-2.52%|
> |TwoInputs_Rescale_LazyFromSource_ExactlyOnce_1024_heap|1415.511667|1300.1|-8.15%|
> |TwoInputs_Rebalance_LazyFromSource_ExactlyOnce_10_rocksdb|34.2967|34.1667|-0.38%|
> |TwoInputs_KeyBy_LazyFromSource_ExactlyOnce_100_heap|158.353|151.848|-4.11%|
> |TwoInputs_Broadcast_Eager_ExactlyOnce_1024_rocksdb|1373.406667|1300.056667|-5.34%|
> |TwoInputs_Rescale_Eager_ExactlyOnce_10_heap|34.5717|32.0967|-7.16%|
> |TwoInputs_Rebalance_Eager_ExactlyOnce_100_rocksdb|158.655|147.44|-7.07%|
> |TwoInputs_KeyBy_Eager_ExactlyOnce_1024_heap|1356.611667|1292.386667|-4.73%|
> |TwoInputs_Broadcast_LazyFromSource_AtLeastOnce_10_rocksdb|34.01|33.205|-2.37%|
> |TwoInputs_Rescale_LazyFromSource_AtLeastOnce_100_heap|149.588|145.997|-2.40%|
> |TwoInputs_Rebalance_LazyFromSource_AtLeastOnce_1024_rocksdb|1359.74|1299.156667|-4.46%|
> |TwoInputs_KeyBy_LazyFromSource_AtLeastOnce_10_heap|34.025|29.6833|-12.76%|
> |TwoInputs_Broadcast_Eager_AtLeastOnce_100_rocksdb|157.303|151.4616667|-3.71%|
> |TwoInputs_Rescale_Eager_AtLeastOnce_1024_heap|1368.74|1293.238333|-5.52%|
> |TwoInputs_Rebalance_Eager_AtLeastOnce_10_rocksdb|34.325|33.285|-3.03%|
> |TwoInputs_KeyBy_Eager_AtLeastOnce_100_heap|162.5116667|134.375|-17.31%|
> It can be seen that the performance of 1.11 has a regression, basically
> around 5%, and the maximum regression is 17%. This needs to be checked.
> the test code:
> flink-1.10.0:
> [https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java]
> flink-1.11.0:
> [https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations_1.11/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java]
> commit cmd like tis：
> bin/flink run -d -m 192.168.39.246:8081 -c
[GitHub] [flink] flinkbot edited a comment on pull request #12776: [FLINK-Streanming] [Streaming API] fixed punctuation
flinkbot edited a comment on pull request #12776: URL: https://github.com/apache/flink/pull/12776#issuecomment-650581477 ## CI report: * 47793c6ae6ad4a752a8d429ea8c3e662022f135c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4059) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12776: [FLINK-Streanming] [Streaming API] fixed punctuation
flinkbot edited a comment on pull request #12776: URL: https://github.com/apache/flink/pull/12776#issuecomment-650581477 ## CI report: * 47793c6ae6ad4a752a8d429ea8c3e662022f135c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4059) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12776: [FLINK-Streanming] [Streaming API] fixed punctuation
flinkbot commented on pull request #12776: URL: https://github.com/apache/flink/pull/12776#issuecomment-650581477 ## CI report: * 47793c6ae6ad4a752a8d429ea8c3e662022f135c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12776: [FLINK-Streanming] [Streaming API] fixed punctuation
flinkbot commented on pull request #12776: URL: https://github.com/apache/flink/pull/12776#issuecomment-650579161 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 47793c6ae6ad4a752a8d429ea8c3e662022f135c (Sat Jun 27 16:02:25 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dwijen opened a new pull request #12776: [FLINK-XXXX] [Streaming API] fixed punctuation
dwijen opened a new pull request #12776: URL: https://github.com/apache/flink/pull/12776 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15693) Stop receiving incoming RPC messages when RpcEndpoint is closing
[
https://issues.apache.org/jira/browse/FLINK-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146985#comment-17146985
]
Till Rohrmann commented on FLINK-15693:
---
One could for example only suppress {{handleRpcInvocation}} calls in
{{AkkaRpcActor}} if it is shutting down. But this would imply that one could
not do any external RPCs where the answer is sent as a separate RPC.
> Stop receiving incoming RPC messages when RpcEndpoint is closing
>
>
> Key: FLINK-15693
> URL: https://issues.apache.org/jira/browse/FLINK-15693
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.9.1, 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.11.0
>
>
> When calling {{RpcEndpoint#closeAsync()}}, the system triggers
> {{RpcEndpoint#onStop}} and transitions the endpoint into the
> {{TerminatingState}}. In order to allow asynchronous clean up operations, the
> main thread executor is not shut down immediately. As a side effect, the
> {{RpcEndpoint}} still accepts incoming RPC messages from other components.
> I think it would be cleaner to no longer accept incoming RPC messages once we
> are in the {{TerminatingState}}. That way we would not worry about the
> internal state of the {{RpcEndpoint}} when processing RPC messages (similar
> to
> [here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java#L952]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on pull request #8466: [FLINK-12336][monitoring] Add HTTPS support to InfluxDB reporter
tillrohrmann commented on pull request #8466: URL: https://github.com/apache/flink/pull/8466#issuecomment-650568251 @jPrest could you help updating this PR by rebasing it on the latest master? Then I'll help with finding someone for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18433) From the end-to-end performance test results, 1.11 has a regression
[ https://issues.apache.org/jira/browse/FLINK-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146983#comment-17146983 ] Till Rohrmann commented on FLINK-18433: --- Thanks for running these tests [~AHeise]. I guess we have to wait for [~Aihua] to clarify how the original tests are executed. How did you exclude the memory management concretely? Would it make sense to disable restarts or to allow checkpoint failures? As you have said if there is a restart, it would drastically thwart the overall result. > From the end-to-end performance test results, 1.11 has a regression > --- > > Key: FLINK-18433 > URL: https://issues.apache.org/jira/browse/FLINK-18433 > Project: Flink > Issue Type: Bug > Components: API / Core, API / DataStream >Affects Versions: 1.11.0 > Environment: 3 machines > [|https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations_1.11/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java] >Reporter: Aihua Li >Priority: Major > Attachments: flink_11.log.gz > > > > I ran end-to-end performance tests between the Release-1.10 and Release-1.11. > the results were as follows: > |scenarioName|release-1.10|release-1.11| | > |OneInput_Broadcast_LazyFromSource_ExactlyOnce_10_rocksdb|46.175|43.8133|-5.11%| > |OneInput_Rescale_LazyFromSource_ExactlyOnce_100_heap|211.835|200.355|-5.42%| > |OneInput_Rebalance_LazyFromSource_ExactlyOnce_1024_rocksdb|1721.041667|1618.32|-5.97%| > |OneInput_KeyBy_LazyFromSource_ExactlyOnce_10_heap|46|43.615|-5.18%| > |OneInput_Broadcast_Eager_ExactlyOnce_100_rocksdb|212.105|199.688|-5.85%| > |OneInput_Rescale_Eager_ExactlyOnce_1024_heap|1754.64|1600.12|-8.81%| > |OneInput_Rebalance_Eager_ExactlyOnce_10_rocksdb|45.9167|43.0983|-6.14%| > |OneInput_KeyBy_Eager_ExactlyOnce_100_heap|212.0816667|200.727|-5.35%| > |OneInput_Broadcast_LazyFromSource_AtLeastOnce_1024_rocksdb|1718.245|1614.381667|-6.04%| > |OneInput_Rescale_LazyFromSource_AtLeastOnce_10_heap|46.12|43.5517|-5.57%| > |OneInput_Rebalance_LazyFromSource_AtLeastOnce_100_rocksdb|212.038|200.388|-5.49%| > |OneInput_KeyBy_LazyFromSource_AtLeastOnce_1024_heap|1762.048333|1606.408333|-8.83%| > |OneInput_Broadcast_Eager_AtLeastOnce_10_rocksdb|46.0583|43.4967|-5.56%| > |OneInput_Rescale_Eager_AtLeastOnce_100_heap|212.233|201.188|-5.20%| > |OneInput_Rebalance_Eager_AtLeastOnce_1024_rocksdb|1720.66|1616.85|-6.03%| > |OneInput_KeyBy_Eager_AtLeastOnce_10_heap|46.14|43.6233|-5.45%| > |TwoInputs_Broadcast_LazyFromSource_ExactlyOnce_100_rocksdb|156.918|152.957|-2.52%| > |TwoInputs_Rescale_LazyFromSource_ExactlyOnce_1024_heap|1415.511667|1300.1|-8.15%| > |TwoInputs_Rebalance_LazyFromSource_ExactlyOnce_10_rocksdb|34.2967|34.1667|-0.38%| > |TwoInputs_KeyBy_LazyFromSource_ExactlyOnce_100_heap|158.353|151.848|-4.11%| > |TwoInputs_Broadcast_Eager_ExactlyOnce_1024_rocksdb|1373.406667|1300.056667|-5.34%| > |TwoInputs_Rescale_Eager_ExactlyOnce_10_heap|34.5717|32.0967|-7.16%| > |TwoInputs_Rebalance_Eager_ExactlyOnce_100_rocksdb|158.655|147.44|-7.07%| > |TwoInputs_KeyBy_Eager_ExactlyOnce_1024_heap|1356.611667|1292.386667|-4.73%| > |TwoInputs_Broadcast_LazyFromSource_AtLeastOnce_10_rocksdb|34.01|33.205|-2.37%| > |TwoInputs_Rescale_LazyFromSource_AtLeastOnce_100_heap|149.588|145.997|-2.40%| > |TwoInputs_Rebalance_LazyFromSource_AtLeastOnce_1024_rocksdb|1359.74|1299.156667|-4.46%| > |TwoInputs_KeyBy_LazyFromSource_AtLeastOnce_10_heap|34.025|29.6833|-12.76%| > |TwoInputs_Broadcast_Eager_AtLeastOnce_100_rocksdb|157.303|151.4616667|-3.71%| > |TwoInputs_Rescale_Eager_AtLeastOnce_1024_heap|1368.74|1293.238333|-5.52%| > |TwoInputs_Rebalance_Eager_AtLeastOnce_10_rocksdb|34.325|33.285|-3.03%| > |TwoInputs_KeyBy_Eager_AtLeastOnce_100_heap|162.5116667|134.375|-17.31%| > It can be seen that the performance of 1.11 has a regression, basically > around 5%, and the maximum regression is 17%. This needs to be checked. > the test code: > flink-1.10.0: > [https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java] > flink-1.11.0: > [https://github.com/Li-Aihua/flink/blob/test_suite_for_basic_operations_1.11/flink-end-to-end-perf-tests/flink-basic-operations/src/main/java/org/apache/flink/basic/operations/PerformanceTestJob.java] > commit cmd like tis： > bin/flink run -d -m 192.168.39.246:8081 -c > org.apache.flink.basic.operations.PerformanceTestJob > /home/admin/flink-basic-operations_2.11-1.10-SNAPSHOT.jar --topologyName > OneInput --LogicalAttributesofEdges Broadcast --ScheduleMode
[GitHub] [flink] flinkbot edited a comment on pull request #12770: [FLINK-18200][python] Replace the deprecated interfaces with the new interfaces in the tests and examples
flinkbot edited a comment on pull request #12770: URL: https://github.com/apache/flink/pull/12770#issuecomment-649764859 ## CI report: * e8821f7b8ba36c7b393039368796f484427d24d8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4058) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18433) From the end-to-end performance test results, 1.11 has a regression
[
https://issues.apache.org/jira/browse/FLINK-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146650#comment-17146650
]
Arvid Heise edited comment on FLINK-18433 at 6/27/20, 9:12 AM:
---
Hi [~liyu], thanks for the update - I feared as much.
Without further information, I did run a particular comparison that may or may
not help.
I picked TwoInputs_KeyBy_Eager_AtLeastOnce_100_heap as it had the biggest
regression (-17.31%) and ran some tests on a single m5ad.2xlarge (with SSD, but
state backend is heap). I built a flink-dist from release-1.10 and release-1.11.
Since there are no built-in evaluation metrics, I just used {{time}}. To reduce
the impact of cluster setup and to really see if it's related to heap state
backend or network stack, I simply executed on a local executor who took the
full 8 cores and I gave it 5gb RAM (job doesn't need much and I wanted to avoid
too much allocation overhead).
Full commands for reference:
{noformat}
time java -Xmx5g -Dlog.file=flink_10.log
-Dlog4j.configuration=file:///`pwd`/flink-1.10/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf/log4j.properties
-cp flink-basic-operations_2.11-1.10-SNAPSHOT.jar:"${flink_10[*]}"
org.apache.flink.basic.operations.PerformanceTestJob --topologyName TwoInputs
--LogicalAttributesofEdges KeyBy --ScheduleMode LazyFromSource --CheckpointMode
AtLeastOnce --recordSize 100 --stateBackend heap --maxCount 100
--checkpointInterval 100 --checkpointPath file:///home/ec2-user/spda/checkpoints
time java -Xmx5g -Dlog.file=`pwd`/flink_11.log
-Dlog4j.configurationFile=file:///`pwd`/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/conf/log4j.properties
-cp flink-basic-operations_2.11-1.11-SNAPSHOT.jar:"${flink_11[*]}"
org.apache.flink.basic.operations.PerformanceTestJob --topologyName TwoInputs
--LogicalAttributesofEdges KeyBy --ScheduleMode LazyFromSource --CheckpointMode
AtLeastOnce --recordSize 100 --stateBackend heap --maxCount 100
--checkpointInterval 100 --checkpointPath
file:///home/ec2-user/spda/checkpoints
--execution.checkpointing.tolerable-failed-checkpoints 1000
{noformat}
I modified the test job to compile and to create a local executor forwarding
the parameters to configuration (more on that later).
I ran these commands interleaved for a few hours and got [this
sheet|https://docs.google.com/spreadsheets/d/1NPoDQakQu1apdzWZfxD2IoRgo28MpBfD9s4K9Aq9nA4/edit?usp=sharing].
On average, we have
Flink 1.10 01m59s
Flink 1.11 01m50s
Note that less is better in this case as we measure the time needed to process
1m elements.
So TL;DR in this particular benchmark setup, it rather looks like performance
actually improved. Note that DOP=8 is higher than what [~Aihua] used. Assuming
that both benchmarks are okay I see 3 options to explain them.
# -We may have a regression on local input channels, but an improvement for
remote input channels. Since, remote input channels are usually the bottleneck,
I'd say this is rather good, but ideally we can still remove the regression
while keeping the improvement.- (just realized that we probably don't have a
single LocalInputChannel with this configuration)
# Memory management in 1.11 works differently/incorrectly. My test excludes the
memory management on TM/JM level, so that may be the root cause for the
original regression.
# I experienced restarts due to failed checkpoints in the end. My first
impression was that when the job is about to be finished may cause some
in-progress checkpoints to be canceled which is propagated to checkpoint
coordinator, which ultimately restarts the job because by default no checkpoint
is allowed to fail. In my final setup, I ignored these errors, but it is
obvious that any restart would impact the performance tremendously. In my
setup, I even ran in some kind of live lock for 1m records (100k records didn't
suffer from these issues oddly).
I'm attaching a log that shows this live lock. [~roman_khachatryan]
investigated but couldn't find anything suspicious.
The key errors are
{noformat}
2020-06-26 14:53:09,662 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable
failure threshold.
at
org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:66)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1626)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1603)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
[jira] [Comment Edited] (FLINK-18433) From the end-to-end performance test results, 1.11 has a regression
[
https://issues.apache.org/jira/browse/FLINK-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146650#comment-17146650
]
Arvid Heise edited comment on FLINK-18433 at 6/27/20, 9:12 AM:
---
Hi [~liyu], thanks for the update - I feared as much.
Without further information, I did run a particular comparison that may or may
not help.
I picked TwoInputs_KeyBy_Eager_AtLeastOnce_100_heap as it had the biggest
regression (-17.31%) and ran some tests on a single m5ad.2xlarge (with SSD, but
state backend is heap). I built a flink-dist from release-1.10 and release-1.11.
Since there are no built-in evaluation metrics, I just used {{time}}. To reduce
the impact of cluster setup and to really see if it's related to heap state
backend or network stack, I simply executed on a local executor who took the
full 8 cores and I gave it 5gb RAM (job doesn't need much and I wanted to avoid
too much allocation overhead).
Full commands for reference:
{noformat}
time java -Xmx5g -Dlog.file=flink_10.log
-Dlog4j.configuration=file:///`pwd`/flink-1.10/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf/log4j.properties
-cp flink-basic-operations_2.11-1.10-SNAPSHOT.jar:"${flink_10[*]}"
org.apache.flink.basic.operations.PerformanceTestJob --topologyName TwoInputs
--LogicalAttributesofEdges KeyBy --ScheduleMode LazyFromSource --CheckpointMode
AtLeastOnce --recordSize 100 --stateBackend heap --maxCount 100
--checkpointInterval 100 --checkpointPath file:///home/ec2-user/spda/checkpoints
time java -Xmx5g -Dlog.file=`pwd`/flink_11.log
-Dlog4j.configurationFile=file:///`pwd`/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/conf/log4j.properties
-cp flink-basic-operations_2.11-1.11-SNAPSHOT.jar:"${flink_11[*]}"
org.apache.flink.basic.operations.PerformanceTestJob --topologyName TwoInputs
--LogicalAttributesofEdges KeyBy --ScheduleMode LazyFromSource --CheckpointMode
AtLeastOnce --recordSize 100 --stateBackend heap --maxCount 100
--checkpointInterval 100 --checkpointPath
file:///home/ec2-user/spda/checkpoints
--execution.checkpointing.tolerable-failed-checkpoints 1000
{noformat}
I modified the test job to compile and to create a local executor forwarding
the parameters to configuration (more on that later).
I ran these commands interleaved for a few hours and got [this
sheet|https://docs.google.com/spreadsheets/d/1NPoDQakQu1apdzWZfxD2IoRgo28MpBfD9s4K9Aq9nA4/edit?usp=sharing].
On average, we have
Flink 1.10 01m59s
Flink 1.11 01m50s
Note that less is better in this case as we measure the time needed to process
1m elements.
So TL;DR in this particular benchmark setup, it rather looks like performance
actually improved. Note that DOP=8 is higher than what [~Aihua] used. Assuming
that both benchmarks are okay I see 3 options to explain them.
-# We may have a regression on local input channels, but an improvement for
remote input channels. Since, remote input channels are usually the bottleneck,
I'd say this is rather good, but ideally we can still remove the regression
while keeping the improvement.- (just realized that we probably don't have a
single LocalInputChannel with this configuration)
# Memory management in 1.11 works differently/incorrectly. My test excludes the
memory management on TM/JM level, so that may be the root cause for the
original regression.
# I experienced restarts due to failed checkpoints in the end. My first
impression was that when the job is about to be finished may cause some
in-progress checkpoints to be canceled which is propagated to checkpoint
coordinator, which ultimately restarts the job because by default no checkpoint
is allowed to fail. In my final setup, I ignored these errors, but it is
obvious that any restart would impact the performance tremendously. In my
setup, I even ran in some kind of live lock for 1m records (100k records didn't
suffer from these issues oddly).
I'm attaching a log that shows this live lock. [~roman_khachatryan]
investigated but couldn't find anything suspicious.
The key errors are
{noformat}
2020-06-26 14:53:09,662 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable
failure threshold.
at
org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:66)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1626)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1603)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
[GitHub] [flink] flinkbot edited a comment on pull request #12770: [FLINK-18200][python] Replace the deprecated interfaces with the new interfaces in the tests and examples
flinkbot edited a comment on pull request #12770: URL: https://github.com/apache/flink/pull/12770#issuecomment-649764859 ## CI report: * 7916851295ea609627070348b3eabf2f3de94e47 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4043) * e8821f7b8ba36c7b393039368796f484427d24d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4058) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12770: [FLINK-18200][python] Replace the deprecated interfaces with the new interfaces in the tests and examples
flinkbot edited a comment on pull request #12770: URL: https://github.com/apache/flink/pull/12770#issuecomment-649764859 ## CI report: * 7916851295ea609627070348b3eabf2f3de94e47 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4043) * e8821f7b8ba36c7b393039368796f484427d24d8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18427) Job failed under java 11
[
https://issues.apache.org/jira/browse/FLINK-18427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146825#comment-17146825
]
Zhang Hao commented on FLINK-18427:
---
Thanks for [~sewen]'s reply, because Chinese Dragon Boat Festival, I will try
to adjust
'[{{taskmanager.memory.task.off-heap.size}}|https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#taskmanager-memory-task-off-heap-size]'
on 29th June , then I will tell you the result.
> Job failed under java 11
>
>
> Key: FLINK-18427
> URL: https://issues.apache.org/jira/browse/FLINK-18427
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration, Runtime / Network
>Affects Versions: 1.10.0
>Reporter: Zhang Hao
>Priority: Critical
>
> flink version:1.10.0
> deployment mode:cluster
> os:linux redhat7.5
> Job parallelism:greater than 1
> My job run normally under java 8, but failed under java 11.Excpetion info
> like below,netty send message failed.In addition, I found job would failed
> when task was distributed on multi node, if I set job's parallelism = 1, job
> run normally under java 11 too.
>
> 2020-06-24 09:52:162020-06-24
> 09:52:16org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
> Sending the partition request to '/170.0.50.19:33320' failed. at
> org.apache.flink.runtime.io.network.netty.NettyPartitionRequestClient$1.operationComplete(NettyPartitionRequestClient.java:124)
> at
> org.apache.flink.runtime.io.network.netty.NettyPartitionRequestClient$1.operationComplete(NettyPartitionRequestClient.java:115)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:474)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:818)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:718)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:708)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.access$1700(AbstractChannelHandlerContext.java:56)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1102)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1149)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1073)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.base/java.lang.Thread.run(Thread.java:834)Caused by:
> java.io.IOException: Error while serializing message:
> PartitionRequest(8059a0b47f7ba0ff814ea52427c584e7@6750c1170c861176ad3ceefe9b02f36e:0:2)
> at
> org.apache.flink.runtime.io.network.netty.NettyMessage$NettyMessageEncoder.write(NettyMessage.java:177)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716)
> ... 11 moreCaused by: java.io.IOException: java.lang.OutOfMemoryError:
> Direct buffer memory at
> org.apache.flink.runtime.io.network.netty.NettyMessage$PartitionRequest.write(NettyMessage.java:497)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #12775: [FLINK-18396][dosc]Translate "Formats Overview" page into Chinese
flinkbot edited a comment on pull request #12775: URL: https://github.com/apache/flink/pull/12775#issuecomment-650482221 ## CI report: * 2ceef8370fcb1901caeaecd35a8d62419e7c5acf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4057) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
