[GitHub] [flink] flinkbot commented on issue #9401: [hotfix] Fix logger arguments in CatalogManager
flinkbot commented on issue #9401: [hotfix] Fix logger arguments in CatalogManager URL: https://github.com/apache/flink/pull/9401#issuecomment-519803058 ## CI report: * 29b3421165df912e3cc114566ba4e12cac901ede : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122546104) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9401: [hotfix] Fix logger arguments in CatalogManager
flinkbot commented on issue #9401: [hotfix] Fix logger arguments in CatalogManager URL: https://github.com/apache/flink/pull/9401#issuecomment-519801400 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 29b3421165df912e3cc114566ba4e12cac901ede (Fri Aug 09 06:49:21 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows
godfreyhe commented on a change in pull request #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows URL: https://github.com/apache/flink/pull/9349#discussion_r312348542 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/BatchLogicalWindowAggregateRule.scala ## @@ -73,6 +76,12 @@ class BatchLogicalWindowAggregateRule ref.getIndex) } } + + def getOperandAsLong(call: RexCall, idx: Int): Long = +call.getOperands.get(idx) match { + case v: RexLiteral => v.getValue.asInstanceOf[JBigDecimal].longValue() + case _ => throw new TableException("Only constant window descriptors are supported") Review comment: this aligns with Flink planner, see [DataSetLogicalWindowAggregateRule](https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/plan/rules/dataSet/DataSetLogicalWindowAggregateRule.scala) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info
wuchong commented on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info URL: https://github.com/apache/flink/pull/9347#issuecomment-519800299 Could you rebase master to trigger the travis again ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zjffdu opened a new pull request #9401: [hotfix] Fix logger arguments in CatalogManager
zjffdu opened a new pull request #9401: [hotfix] Fix logger arguments in CatalogManager URL: https://github.com/apache/flink/pull/9401 ## What is the purpose of the change Trivial fix for the logger arguments in CatalogManager ## Brief change log Reorder the arguments ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9)
flinkbot edited a comment on issue #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9) URL: https://github.com/apache/flink/pull/9400#issuecomment-519772402 ## CI report: * 2fef590f19082d27f35d60a1dc87e802f673d777 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122537159) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-13587) Fix some transformation names are not set in blink planner
[ https://issues.apache.org/jira/browse/FLINK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-13587. - Resolution: Fixed [table-planner-blink] Fix some operator names are not set in blink planner - master: bf37130d34d762d4ebbfc594ad1a52e63feae71a - 1.9: 4517949700b0401923eaa0bd6ec6704f05b7bdf3 [table-planner-blink] Introduces a framework to reuse code of "explainTerms" to generate operator names - master: c649c8bef5a457f49508382e77b923b2e0da643c - 1.9: 0c5d0a112e8c0e5a6c5bef00ca90f263cbabe902 > Fix some transformation names are not set in blink planner > -- > > Key: FLINK-13587 > URL: https://issues.apache.org/jira/browse/FLINK-13587 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Jark Wu >Assignee: Jark Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, there are some transformation names are not set in blink planner. > For example, LookupJoin transformation uses "LookupJoin" directly which loses > a lot of informatoion. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] wuchong merged pull request #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner
wuchong merged pull request #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner URL: https://github.com/apache/flink/pull/9363 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] lirui-apache commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
lirui-apache commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#issuecomment-519797730 @bowenli86 PR updated according to your suggestions. Please take another look. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner
wuchong commented on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner URL: https://github.com/apache/flink/pull/9363#issuecomment-519797060 Travis passed. Merging... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9392: [FLINK-13611] [table-planner-blink] Introduce statistic generator utility to generate table & column statistics
flinkbot edited a comment on issue #9392: [FLINK-13611] [table-planner-blink] Introduce statistic generator utility to generate table & column statistics URL: https://github.com/apache/flink/pull/9392#issuecomment-519408914 ## CI report: * 46a8d8ca2b5b5f0af043c35770083053507938c5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122393172) * 07d04ce5e6e76ef01dd6efc9790999f5107c3155 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122437582) * 96d8b4cbc1fc1823a39dc41a938b42c215ead0f1 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122543281) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13405) Translate "Basic API Concepts" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-13405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903588#comment-16903588 ] WangHengWei commented on FLINK-13405: - [~jark], thank you. I think it's qq email's problem. I have changed to 163 mailbox and received email. Thanks very much. > Translate "Basic API Concepts" page into Chinese > > > Key: FLINK-13405 > URL: https://issues.apache.org/jira/browse/FLINK-13405 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: WangHengWei >Assignee: WangHengWei >Priority: Major > Labels: documentation, pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The page url is > [https://github.com/apache/flink/blob/master/docs/dev/api_concepts.zh.md] > The markdown file is located in flink/docs/dev/api_concepts.zh.md -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (FLINK-13523) Verify and correct arithmetic function's semantic for Blink planner
[ https://issues.apache.org/jira/browse/FLINK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-13523. - Resolution: Fixed > Verify and correct arithmetic function's semantic for Blink planner > --- > > Key: FLINK-13523 > URL: https://issues.apache.org/jira/browse/FLINK-13523 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] wuchong merged pull request #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9)
wuchong merged pull request #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9) URL: https://github.com/apache/flink/pull/9400 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-13523) Verify and correct arithmetic function's semantic for Blink planner
[ https://issues.apache.org/jira/browse/FLINK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903530#comment-16903530 ] Jark Wu edited comment on FLINK-13523 at 8/9/19 6:08 AM: - [table-planner-blink] Refactor AVG aggregate function to keep it compatible with old planner - master: 421f0a559a3038d2f9f56ba2cbab8e6d1832812a - 1.9: 2fef590f19082d27f35d60a1dc87e802f673d777 [table-planner-blink] Remove non-standard bitwise scalar function and DIV(), DIV_INT() function from blink planner - master: fbd3225e9d63a10eb6e55fca136d2acfcc777250 - 1.9: dc2f8a2e277d31634a0da90026737ed4d53171da [table-planner-blink] Refactor DIVIDE function to keep it compatible with old planner - master: 4d560df50d08f22b68e8b8c9a0e2086f45b5f4b4 - 1.9: c27ea21ec8b029c8f0fcbf61678c7dc505589d6d was (Author: jark): [table-planner-blink] Refactor AVG aggregate function to keep it compatible with old planner - master: 421f0a559a3038d2f9f56ba2cbab8e6d1832812a - 1.9: [table-planner-blink] Remove non-standard bitwise scalar function and DIV(), DIV_INT() function from blink planner - master: fbd3225e9d63a10eb6e55fca136d2acfcc777250 - 1.9: [table-planner-blink] Refactor DIVIDE function to keep it compatible with old planner - master: 4d560df50d08f22b68e8b8c9a0e2086f45b5f4b4 - 1.9: > Verify and correct arithmetic function's semantic for Blink planner > --- > > Key: FLINK-13523 > URL: https://issues.apache.org/jira/browse/FLINK-13523 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
flinkbot edited a comment on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#issuecomment-519768509 ## CI report: * 3fbe05bd67589e1f46d05df3e6db7cc79281b450 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/122536079) * f3695092d7d395e08ab0006b713e2cb09fb76449 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122542026) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala
flinkbot edited a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala URL: https://github.com/apache/flink/pull/8906#issuecomment-519474213 ## CI report: * bea9f0b86c27d30c763e8578edc5ef46a7d76e9e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122421031) * a31d440b058e9fbe2d0a908c4a4fb63750685340 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/122532295) * d9fdee4191a92ea21dd1eb5d400ed54a627750b3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122535437) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows
flinkbot edited a comment on issue #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows URL: https://github.com/apache/flink/pull/9349#issuecomment-517912973 ## CI report: * 6b1ab58d9d153f44f5cbdaee4804bcd5e27544db : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/121837224) * 6bc0849d18697d9a6e4e2899d05ab6e2e160df46 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122534498) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] lirui-apache commented on a change in pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
lirui-apache commented on a change in pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#discussion_r312335044 ## File path: flink-table/flink-sql-client/src/test/java/org/apache/flink/table/client/gateway/local/LocalExecutorITCase.java ## @@ -506,6 +506,30 @@ public void testUseCatalogAndUseDatabase() throws Exception { } } + @Test + public void testUseNonExistingDatabase() throws Exception { + final Executor executor = createDefaultExecutor(clusterClient); + final SessionContext session = new SessionContext("test-session", new Environment()); + + try { + executor.useDatabase(session, "nonexistingdb"); + } catch (SqlExecutionException e) { Review comment: It's following the pattern in `testValidateSession`. But I guess it's also fine to use `ExpectedException`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9342: [FLINK-13438][hive] Fix DataTypes.DATE/TIME/TIMESTAMP support for hive connectors
flinkbot edited a comment on issue #9342: [FLINK-13438][hive] Fix DataTypes.DATE/TIME/TIMESTAMP support for hive connectors URL: https://github.com/apache/flink/pull/9342#issuecomment-517770642 ## CI report: * 76704f271662b57cbe36679d3d249bcdd7fdf66a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/121784366) * 7b4a9226cfffc1ea505c8d20b5b5f9ce8c5d2113 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122239651) * ec81369c4e332d9290a2b42e386f9be724d8e2ad : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122307651) * b2d4875b20874041f90db3473010cf454a2cba66 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122365586) * 83860f4cb617d777093dce251e4145c5e8f79e7f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122423996) * 8e2e5fee6859c7cfa1e3cfcc6f5d3bfe0dd8edbc : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122540373) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-6962) Add a create table SQL DDL
[ https://issues.apache.org/jira/browse/FLINK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903577#comment-16903577 ] Jark Wu commented on FLINK-6962: Yes [~twalthr], I just created an issue to track this effort: FLINK-13661 > Add a create table SQL DDL > -- > > Key: FLINK-6962 > URL: https://issues.apache.org/jira/browse/FLINK-6962 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Shaoxuan Wang >Assignee: Danny Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This Jira adds support to allow user define the DDL for source and sink > tables, including the waterMark(on source table) and emit SLA (on result > table). The detailed design doc will be attached soon. > This issue covered adding batch DDL support. Streaming-specific DDL support > will be added later. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13661) Add a stream specific CREATE TABLE SQL DDL
Jark Wu created FLINK-13661: --- Summary: Add a stream specific CREATE TABLE SQL DDL Key: FLINK-13661 URL: https://issues.apache.org/jira/browse/FLINK-13661 Project: Flink Issue Type: Sub-task Components: Table SQL / API Reporter: Jark Wu Fix For: 1.10.0 FLINK-6962 has introduced a basic SQL DDL to create a table. However, it doesn't support stream specific features, for example, watermark definition, changeflag definition, computed columns, primary keys and so on. We started a FLIP design doc[1] to discuss the concepts of source and sink to help us have a well-defined DDL. Once the FLIP is accepted, we can start the work. [1]: https://docs.google.com/document/d/1yrKXEIRATfxHJJ0K3t6wUgXAtZq8D-XgvEnvl2uUcr0/edit#heading=h.c05t427gfgxa -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] TsReaper commented on a change in pull request #9342: [FLINK-13438][hive] Fix DataTypes.DATE/TIME/TIMESTAMP support for hive connectors
TsReaper commented on a change in pull request #9342: [FLINK-13438][hive] Fix DataTypes.DATE/TIME/TIMESTAMP support for hive connectors URL: https://github.com/apache/flink/pull/9342#discussion_r312330806 ## File path: flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/util/HiveTypeUtil.java ## @@ -256,9 +258,9 @@ private static DataType toFlinkPrimitiveType(PrimitiveTypeInfo hiveType) { case DOUBLE: return DataTypes.DOUBLE(); case DATE: - return DataTypes.DATE(); + return DataTypes.DATE().bridgedTo(Date.class); case TIMESTAMP: - return DataTypes.TIMESTAMP(); + return DataTypes.TIMESTAMP(3).bridgedTo(Timestamp.class); Review comment: Thanks for the discussion on the precision issue. As the default precision in hive is 9, the connector will be literally not functional if we directly throws an exception. So I think it will be better to log a warning message and add in the connector's document that timestamp precision may be lost. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner
flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner URL: https://github.com/apache/flink/pull/9363#issuecomment-518265997 ## CI report: * 1fe6c332279c34546ec3db24a574dfd53500d20b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/121971735) * fa3e7406f9664a59efcb448748511b656474e74c : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/122089384) * 28175449cb1d5eb8f318359090ea87e5b2af42d1 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122102858) * 81593a4dcb3573843c1c02cba0cb17abe1693065 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122290876) * 829830f8df1eb814ac44716f230c6aedcfaa5128 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122369808) * 0dcba7f669a5702e5730aac0273eb4a58fce2be9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122460702) * a5052f45f08b0760cdd4a163a87c0be7d6383c42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122479843) * e8a680f4d832f794eb80d6e569d323babc4da518 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122530453) * bf37130d34d762d4ebbfc594ad1a52e63feae71a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122534827) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
bowenli86 commented on a change in pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#discussion_r312327916 ## File path: flink-table/flink-sql-client/src/test/java/org/apache/flink/table/client/gateway/local/LocalExecutorITCase.java ## @@ -506,6 +506,30 @@ public void testUseCatalogAndUseDatabase() throws Exception { } } + @Test + public void testUseNonExistingDatabase() throws Exception { + final Executor executor = createDefaultExecutor(clusterClient); + final SessionContext session = new SessionContext("test-session", new Environment()); + + try { + executor.useDatabase(session, "nonexistingdb"); + } catch (SqlExecutionException e) { + // expected + } + } + + @Test + public void testUseNonExistingCatalog() throws Exception { + final Executor executor = createDefaultExecutor(clusterClient); + final SessionContext session = new SessionContext("test-session", new Environment()); + + try { + executor.useCatalog(session, "nonexistingcatalog"); + } catch (SqlExecutionException e) { Review comment: shall we just assert the exception is expected instead of try-catch-ignore? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
bowenli86 commented on a change in pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#discussion_r312327896 ## File path: flink-table/flink-sql-client/src/test/java/org/apache/flink/table/client/gateway/local/LocalExecutorITCase.java ## @@ -506,6 +506,30 @@ public void testUseCatalogAndUseDatabase() throws Exception { } } + @Test + public void testUseNonExistingDatabase() throws Exception { + final Executor executor = createDefaultExecutor(clusterClient); + final SessionContext session = new SessionContext("test-session", new Environment()); + + try { + executor.useDatabase(session, "nonexistingdb"); + } catch (SqlExecutionException e) { Review comment: shall we just assert the exception is expected instead of try-catch-ignore? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13405) Translate "Basic API Concepts" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-13405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903554#comment-16903554 ] Jark Wu commented on FLINK-13405: - Did you send email to dev-subsr...@flink.apache.org to subscribe this ML? You can check whether they are in the SPAM? > Translate "Basic API Concepts" page into Chinese > > > Key: FLINK-13405 > URL: https://issues.apache.org/jira/browse/FLINK-13405 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: WangHengWei >Assignee: WangHengWei >Priority: Major > Labels: documentation, pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The page url is > [https://github.com/apache/flink/blob/master/docs/dev/api_concepts.zh.md] > The markdown file is located in flink/docs/dev/api_concepts.zh.md -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot commented on issue #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9)
flinkbot commented on issue #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9) URL: https://github.com/apache/flink/pull/9400#issuecomment-519772402 ## CI report: * 2fef590f19082d27f35d60a1dc87e802f673d777 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122537159) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9)
flinkbot commented on issue #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9) URL: https://github.com/apache/flink/pull/9400#issuecomment-519771498 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 2fef590f19082d27f35d60a1dc87e802f673d777 (Fri Aug 09 04:21:05 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong opened a new pull request #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9)
wuchong opened a new pull request #9400: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner (release-1.9) URL: https://github.com/apache/flink/pull/9400 This is a cherry pick from master to run travis. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner
flinkbot edited a comment on issue #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner URL: https://github.com/apache/flink/pull/9394#issuecomment-519467299 ## CI report: * 861d120d83964d267c95d024993b82186b1e6e7a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122417819) * a423b9b46d10f61091bd70aa0ea1bd2367fd25a0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122531452) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13473) Add GroupWindowed FlatAggregate support to stream Table API(blink planner), i.e, align with flink planner
[ https://issues.apache.org/jira/browse/FLINK-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13473: Component/s: (was: Table SQL / API) Table SQL / Planner > Add GroupWindowed FlatAggregate support to stream Table API(blink planner), > i.e, align with flink planner > - > > Key: FLINK-13473 > URL: https://issues.apache.org/jira/browse/FLINK-13473 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add GroupWindowed FlatAggregate support to stream Table API(blink planner), > i.e, align with flink planner. > The API looks like: > {code} > TableAggregateFunction tableAggFunc = new MyTableAggregateFunction(); > tableEnv.registerFunction("tableAggFunc", tableAggFunc); > windowGroupedTable > .flatAggregate("tableAggFunc(a, b) as (x, y, z)") > .select("key, window.start, x, y, z") > {code} > The detail can be found in > [Flip-29|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] wuchong commented on issue #9396: [FLINK-13473][table] Add stream Windowed FlatAggregate support for blink planner
wuchong commented on issue #9396: [FLINK-13473][table] Add stream Windowed FlatAggregate support for blink planner URL: https://github.com/apache/flink/pull/9396#issuecomment-519769654 I would suggest to modify the component name in commit message a bit. How about: [FLINK-13473][table-planner-blink] Support windowed TableAggregate in some MetadataHandle [FLINK-13473][table-runtime-blink] Add tests for window operator [FLINK-13473][table-blink] Add runtime support for windowed flatAggregat on blink planner [FLINK-13473][table-planner-blink] Add plan support for windowed flatAggregate on blink planner And for the pull request title, I would suggest to use "[table-blink]" because it doesn't contain API changes. What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-12481) Make processing time timer trigger run via the mailbox
[ https://issues.apache.org/jira/browse/FLINK-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903548#comment-16903548 ] Biao Liu commented on FLINK-12481: -- Hi [~1u0], thanks for feedback. :) > Make processing time timer trigger run via the mailbox > -- > > Key: FLINK-12481 > URL: https://issues.apache.org/jira/browse/FLINK-12481 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: Stefan Richter >Assignee: Alex >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This sub-task integrates the mailbox with processing time timer triggering. > Those triggers should now be enqueued as mailbox events and picked up by the > stream task's main thread for processing. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
flinkbot commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#issuecomment-519768509 ## CI report: * 3fbe05bd67589e1f46d05df3e6db7cc79281b450 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122536079) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows
wuchong commented on a change in pull request #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows URL: https://github.com/apache/flink/pull/9349#discussion_r312321622 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/BatchLogicalWindowAggregateRule.scala ## @@ -73,6 +76,12 @@ class BatchLogicalWindowAggregateRule ref.getIndex) } } + + def getOperandAsLong(call: RexCall, idx: Int): Long = +call.getOperands.get(idx) match { + case v: RexLiteral => v.getValue.asInstanceOf[JBigDecimal].longValue() + case _ => throw new TableException("Only constant window descriptors are supported") Review comment: Should we also update the exception message to align with Stream Window Aggregate? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] lirui-apache commented on a change in pull request #9390: [FLINK-13534][hive] Unable to query Hive table with decimal column
lirui-apache commented on a change in pull request #9390: [FLINK-13534][hive] Unable to query Hive table with decimal column URL: https://github.com/apache/flink/pull/9390#discussion_r312321615 ## File path: flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/TableEnvHiveConnectorTest.java ## @@ -138,6 +138,34 @@ private void readWriteFormat(String format) throws Exception { hiveShell.execute("drop database db1 cascade"); } + @Test + public void testDecimal() throws Exception { + hiveShell.execute("create database db1"); + try { + // Hive's default decimal is decimal(10, 0) + hiveShell.execute("create table db1.src1 (x decimal)"); + hiveShell.execute("create table db1.src2 (x decimal)"); + hiveShell.execute("create table db1.dest (x decimal)"); + // populate src1 from Hive + hiveShell.execute("insert into db1.src1 values (1),(2.0),(5.4),(5.5),(123456789123)"); + + TableEnvironment tableEnv = getTableEnvWithHiveCatalog(); + // populate src2 with same data from Flink + tableEnv.sqlUpdate("insert into db1.src2 values (cast(1 as decimal(10,0))), (cast(2.0 as decimal(10,0))), " + Review comment: Do you mean insert some non-zero-scale decimals into zero-scale decimal column? It's not allowed by the planner. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info
flinkbot edited a comment on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info URL: https://github.com/apache/flink/pull/9347#issuecomment-517895998 ## CI report: * 212cd6c24fdb696aa13bed1cfff875f9bfc01d09 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/121831822) * fa92c963b1f41d75b1399d41708d88cd47a632e9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122530462) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner
wuchong commented on a change in pull request #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner URL: https://github.com/apache/flink/pull/9394#discussion_r312319794 ## File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/functions/sql/FlinkSqlOperatorTable.java ## @@ -502,69 +482,56 @@ public void lookupOperatorOverloads( SqlKind.OTHER_FUNCTION, VARCHAR_2000_NULLABLE, null, - OperandTypes.or( - OperandTypes.family(SqlTypeFamily.STRING), - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.STRING)), + OperandTypes.family(SqlTypeFamily.STRING), SqlFunctionCategory.STRING); public static final SqlFunction SHA1 = new SqlFunction( "SHA1", SqlKind.OTHER_FUNCTION, VARCHAR_2000_NULLABLE, null, - OperandTypes.or( - OperandTypes.family(SqlTypeFamily.STRING), - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.STRING)), + OperandTypes.family(SqlTypeFamily.STRING), SqlFunctionCategory.STRING); public static final SqlFunction SHA224 = new SqlFunction( "SHA224", SqlKind.OTHER_FUNCTION, VARCHAR_2000_NULLABLE, null, - OperandTypes.or( - OperandTypes.family(SqlTypeFamily.STRING), - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.STRING)), + OperandTypes.family(SqlTypeFamily.STRING), SqlFunctionCategory.STRING); public static final SqlFunction SHA256 = new SqlFunction( "SHA256", SqlKind.OTHER_FUNCTION, VARCHAR_2000_NULLABLE, null, - OperandTypes.or( - OperandTypes.family(SqlTypeFamily.STRING), - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.STRING)), + OperandTypes.family(SqlTypeFamily.STRING), SqlFunctionCategory.STRING); public static final SqlFunction SHA384 = new SqlFunction( "SHA384", SqlKind.OTHER_FUNCTION, VARCHAR_2000_NULLABLE, null, - OperandTypes.or( - OperandTypes.family(SqlTypeFamily.STRING), - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.STRING)), + OperandTypes.family(SqlTypeFamily.STRING), SqlFunctionCategory.STRING); public static final SqlFunction SHA512 = new SqlFunction( "SHA512", SqlKind.OTHER_FUNCTION, VARCHAR_2000_NULLABLE, null, - OperandTypes.or( - OperandTypes.family(SqlTypeFamily.STRING), - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.STRING)), + OperandTypes.family(SqlTypeFamily.STRING), SqlFunctionCategory.STRING); public static final SqlFunction SHA2 = new SqlFunction( "SHA2", SqlKind.OTHER_FUNCTION, VARCHAR_2000_NULLABLE, null, - OperandTypes.or( - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.INTEGER), - OperandTypes.family(SqlTypeFamily.STRING, SqlTypeFamily.STRING, SqlTypeFamily.INTEGER)), + OperandTypes.sequence("'(DATA, HASH_LENGTH)'", Review comment: ```suggestion OperandTypes.sequence("'SHA2(, )'", ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner
wuchong commented on a change in pull request #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner URL: https://github.com/apache/flink/pull/9394#discussion_r312318835 ## File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/dataformat/BinaryStringUtil.java ## @@ -749,21 +749,21 @@ public static BinaryString concat(Iterable inputs) { } /** -* Concatenates input strings together into a single string using the separator. -* A null input is skipped. For example, concat(",", "a", null, "c") would yield "a,c". +* Concatenates input strings together into a single string using the separator. +* Returns NULL If the separator is NULL. +* +* Note: CONCAT_WS() does not skip any empty strings, however it does skip any NULL values after +* the separator. For example, concat(",", "a", null, "c") would yield "a,c". Review comment: ```suggestion * the separator. For example, concat_ws(",", "a", null, "c") would yield "a,c". ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] lirui-apache commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
lirui-apache commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#issuecomment-519767241 @xuefuz @bowenli86 @zjuwangg please have a look. I suppose it's trivial change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
flinkbot commented on issue #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399#issuecomment-519767122 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3fbe05bd67589e1f46d05df3e6db7cc79281b450 (Fri Aug 09 03:52:17 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13526).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-13660) Cannot submit job on Flink session cluster on kubernetes with multiple JM pods (zk HA) through web frontend
MalcolmSanders created FLINK-13660: -- Summary: Cannot submit job on Flink session cluster on kubernetes with multiple JM pods (zk HA) through web frontend Key: FLINK-13660 URL: https://issues.apache.org/jira/browse/FLINK-13660 Project: Flink Issue Type: Bug Components: Runtime / Coordination, Runtime / Web Frontend Affects Versions: 1.9.0 Reporter: MalcolmSanders Hi, all, Previously I'm testing HighAvailabilityService of Flink 1.9 on k8s. When testing Flink session cluster with 3 JM pods deployed on k8s, I find the jar I previously uploaded to the web frontend will continuously dispear in "Uploaded Jars" web page. As a result, it's hard to submit the job. After investigation, I find that it has something to do with (1) the implementation of method "handleRequest" of "JarListHandler" and "JarUploadHandler" RestHandlers along with (2) the routing mechanism of k8s service. (1) It seem to me that "handleRequest" method should dispatch the message through "DispatcherGateway gateway" to the leader JM. While the two RestHanders don't use the gateway and just do things locally. That is to say if a "upload jar" request or "list loaded jars" request is sent to any of the 3 JMs, the web frontend will only storage or fetch jars from local directory. (2) I use k8s service to open a flink web page, the URL pattern is (PS: start "kubectl proxy" locally): http://127.0.0.1:8001/api/v1/namespaces/${my_ns}/services/${my_session_cluster_service}:ui/proxy/#/submit Since there a 3 endpoints (3 JMs) of this k8s service, the k8s routing mechanism will randomly choose which endpoint (JM) a REST message sends to. As a result of the two factors, Flink session cluster previously cannot be deployed with multiple JMs using HighAvailablityService on k8s. Proposals: (1) redirect jar related REST messages to the leader JM (2) (along with proposal(1)) synchronize jar files with the standby JMs incase of standby JM taking the leadership (3) support upload jars to global filesystem (etc. dfs) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] lirui-apache opened a new pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data…
lirui-apache opened a new pull request #9399: [FLINK-13526][sql-client] Switching to a non existing catalog or data… URL: https://github.com/apache/flink/pull/9399 …base crashes sql-client ## What is the purpose of the change Avoid crashing sql-client when switching to non-existing catalog or database. ## Brief change log - Catch `CatalogException` thrown by TableEnvironment and throw a `SqlExecutionException` instead. - Add test cases. ## Verifying this change New test cases. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? NA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client
[ https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13526: --- Labels: pull-request-available (was: ) > Switching to a non existing catalog or database crashes sql-client > -- > > Key: FLINK-13526 > URL: https://issues.apache.org/jira/browse/FLINK-13526 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Reporter: Rui Li >Priority: Major > Labels: pull-request-available > > sql-client crashes if user tries to switch to a non-existing DB: > {noformat} > Exception in thread "main" org.apache.flink.table.client.SqlClientException: > Unexpected exception. This is a bug. Please consider filing an issue. > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206) > Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A > database with name [foo] does not exist in the catalog: [myhive]. > at > org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258) > at > org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256) > at > org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282) > at java.util.Optional.ifPresent(Optional.java:159) > at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala
flinkbot edited a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala URL: https://github.com/apache/flink/pull/8906#issuecomment-519474213 ## CI report: * bea9f0b86c27d30c763e8578edc5ef46a7d76e9e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122421031) * a31d440b058e9fbe2d0a908c4a4fb63750685340 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/122532295) * d9fdee4191a92ea21dd1eb5d400ed54a627750b3 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122535437) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner
flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner URL: https://github.com/apache/flink/pull/9363#issuecomment-518265997 ## CI report: * 1fe6c332279c34546ec3db24a574dfd53500d20b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/121971735) * fa3e7406f9664a59efcb448748511b656474e74c : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/122089384) * 28175449cb1d5eb8f318359090ea87e5b2af42d1 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122102858) * 81593a4dcb3573843c1c02cba0cb17abe1693065 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122290876) * 829830f8df1eb814ac44716f230c6aedcfaa5128 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122369808) * 0dcba7f669a5702e5730aac0273eb4a58fce2be9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122460702) * a5052f45f08b0760cdd4a163a87c0be7d6383c42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122479843) * e8a680f4d832f794eb80d6e569d323babc4da518 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122530453) * bf37130d34d762d4ebbfc594ad1a52e63feae71a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122534827) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client
[ https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-13526: --- Summary: Switching to a non existing catalog or database crashes sql-client (was: Switching to a non existing database crashes sql-client) > Switching to a non existing catalog or database crashes sql-client > -- > > Key: FLINK-13526 > URL: https://issues.apache.org/jira/browse/FLINK-13526 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Reporter: Rui Li >Priority: Major > > sql-client crashes if user tries to switch to a non-existing DB: > {noformat} > Exception in thread "main" org.apache.flink.table.client.SqlClientException: > Unexpected exception. This is a bug. Please consider filing an issue. > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206) > Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A > database with name [foo] does not exist in the catalog: [myhive]. > at > org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258) > at > org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256) > at > org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282) > at java.util.Optional.ifPresent(Optional.java:159) > at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows
flinkbot edited a comment on issue #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows URL: https://github.com/apache/flink/pull/9349#issuecomment-517912973 ## CI report: * 6b1ab58d9d153f44f5cbdaee4804bcd5e27544db : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/121837224) * 6bc0849d18697d9a6e4e2899d05ab6e2e160df46 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122534498) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-13659) Add method listDatabases(catalog) and listTables(catalog, database) in TableEnvironment
Jeff Zhang created FLINK-13659: -- Summary: Add method listDatabases(catalog) and listTables(catalog, database) in TableEnvironment Key: FLINK-13659 URL: https://issues.apache.org/jira/browse/FLINK-13659 Project: Flink Issue Type: Improvement Components: Table SQL / API Affects Versions: 1.9.0 Reporter: Jeff Zhang It would be nice to add method listDatabases(catalog) and listTables(catalog, database) in TableEnvironment. So that I can listDatabases with specified catalog and listTables with specified catalog and databases. Otherwise I would always need to call useCatalog and useDatabase beforehand. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13523) Verify and correct arithmetic function's semantic for Blink planner
[ https://issues.apache.org/jira/browse/FLINK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903530#comment-16903530 ] Jark Wu commented on FLINK-13523: - [table-planner-blink] Refactor AVG aggregate function to keep it compatible with old planner - master: 421f0a559a3038d2f9f56ba2cbab8e6d1832812a - 1.9: [table-planner-blink] Remove non-standard bitwise scalar function and DIV(), DIV_INT() function from blink planner - master: fbd3225e9d63a10eb6e55fca136d2acfcc777250 - 1.9: [table-planner-blink] Refactor DIVIDE function to keep it compatible with old planner - master: 4d560df50d08f22b68e8b8c9a0e2086f45b5f4b4 - 1.9: > Verify and correct arithmetic function's semantic for Blink planner > --- > > Key: FLINK-13523 > URL: https://issues.apache.org/jira/browse/FLINK-13523 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner
flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner URL: https://github.com/apache/flink/pull/9363#issuecomment-518265997 ## CI report: * 1fe6c332279c34546ec3db24a574dfd53500d20b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/121971735) * fa3e7406f9664a59efcb448748511b656474e74c : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/122089384) * 28175449cb1d5eb8f318359090ea87e5b2af42d1 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122102858) * 81593a4dcb3573843c1c02cba0cb17abe1693065 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122290876) * 829830f8df1eb814ac44716f230c6aedcfaa5128 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122369808) * 0dcba7f669a5702e5730aac0273eb4a58fce2be9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122460702) * a5052f45f08b0760cdd4a163a87c0be7d6383c42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122479843) * e8a680f4d832f794eb80d6e569d323babc4da518 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122530453) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #9331: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner
wuchong commented on issue #9331: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner URL: https://github.com/apache/flink/pull/9331#issuecomment-519762758 Merged with 4d560df50d08f22b68e8b8c9a0e2086f45b5f4b4 fbd3225e9d63a10eb6e55fca136d2acfcc777250 421f0a559a3038d2f9f56ba2cbab8e6d1832812a This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong closed pull request #9331: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner
wuchong closed pull request #9331: [FLINK-13523][table-planner-blink] Verify and correct arithmetic function's semantic for Blink planner URL: https://github.com/apache/flink/pull/9331 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] godfreyhe commented on issue #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows
godfreyhe commented on issue #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows URL: https://github.com/apache/flink/pull/9349#issuecomment-519762649 thanks @wuchong, i have updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13653) ResultStore should avoid using RowTypeInfo when creating a result
[ https://issues.apache.org/jira/browse/FLINK-13653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903527#comment-16903527 ] Rui Li commented on FLINK-13653: I thought we should just use the new type system for {{CollectBatchTableSink}} and {{CollectStreamTableSink}}. However, according to the JavaDoc of {{TableSink.getOutputType()}} and {{TableSource.getReturnType()}}, user should "_use either the old or the new type system consistently to avoid unintended behavior_". And if the table sinks created in SQL client need to support table sources that may use new or old type systems, I'm not sure whether we have to create different sinks for new and old type systems respectively? > ResultStore should avoid using RowTypeInfo when creating a result > - > > Key: FLINK-13653 > URL: https://issues.apache.org/jira/browse/FLINK-13653 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Reporter: Rui Li >Priority: Major > > Creating a RowTypeInfo from a TableSchema can lose type parameters. As a > result, querying a Hive table with decimal column from SQL CLI will hit the > following exception: > {noformat} > Caused by: org.apache.flink.table.api.ValidationException: Field types of > query result and registered TableSink [default_catalog, default_database, > default: select * from foo] do not match. > Query result schema: [x: BigDecimal] > TableSink schema:[x: BigDecimal] > at > org.apache.flink.table.planner.sinks.TableSinkUtils$.validateSink(TableSinkUtils.scala:69) > at > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:179) > at > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(PlannerBase.scala:178) > at > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:146) > at > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:146) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:146) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:439) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.insertInto(TableEnvironmentImpl.java:327) > at > org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:428) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeQueryInternal$10(LocalExecutor.java:477) > at > org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryInternal(LocalExecutor.java:475) > ... 8 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] godfreyhe commented on a change in pull request #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows
godfreyhe commented on a change in pull request #9349: [FLINK-13564] [table-planner-blink] throw exception if constant with YEAR TO MONTH resolution was used for group windows URL: https://github.com/apache/flink/pull/9349#discussion_r312316102 ## File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/agg/WindowAggregateTest.scala ## @@ -85,6 +85,18 @@ class WindowAggregateTest extends TableTestBase { util.verifyPlanNotExpected(sql, "TUMBLE(rowtime") } + @Test + def testWindowWrongWindowParameter(): Unit = { +expectedException.expect(classOf[TableException]) +expectedException.expectMessage( + "Only constant window intervals with millisecond resolution are supported") + +val sqlQuery = + "SELECT COUNT(*) FROM MyTable GROUP BY TUMBLE(proctime, INTERVAL '2-10' YEAR TO MONTH)" Review comment: > Add one more test `INTERVAL '35' DAYS` which should work? yes, it's valid This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13405) Translate "Basic API Concepts" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-13405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903515#comment-16903515 ] WangHengWei commented on FLINK-13405: - [~jark] [~xccui], thank you very much. I think "数据汇" is a good translation but since it's not a general term I choose not to translate it. BTW, my qq email can not receive any message from Jira or d...@flink.apache.org, do you know what's the problem and how to fix it? > Translate "Basic API Concepts" page into Chinese > > > Key: FLINK-13405 > URL: https://issues.apache.org/jira/browse/FLINK-13405 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: WangHengWei >Assignee: WangHengWei >Priority: Major > Labels: documentation, pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The page url is > [https://github.com/apache/flink/blob/master/docs/dev/api_concepts.zh.md] > The markdown file is located in flink/docs/dev/api_concepts.zh.md -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (FLINK-13655) Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception
[ https://issues.apache.org/jira/browse/FLINK-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiJun updated FLINK-13655: -- Labels: KryoSerializer (was: ) > Caused by: java.io.IOException: Thread 'SortMerger spilling thread' > terminated due to an exception > -- > > Key: FLINK-13655 > URL: https://issues.apache.org/jira/browse/FLINK-13655 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.6.3 > Environment: {color:#e8bf6a} > {color}{color:#e8bf6a} > {color}UTF-8{color:#e8bf6a} > {color}{color:#e8bf6a} > {color}1.5.6{color:#e8bf6a} > {color}{color:#e8bf6a} > {color}1.7.7{color:#e8bf6a} > {color}{color:#e8bf6a} > {color}1.2.17{color:#e8bf6a} > {color}{color:#e8bf6a} > {color}2.11{color:#e8bf6a} > {color}{color:#e8bf6a} > {color}2.11.12{color:#e8bf6a} > {color}{color:#e8bf6a}{color} > {color:#e8bf6a}_parameters.setBoolean("recursive.file_{color:#33}.enumeration",true){color}{color} >Reporter: LiJun >Priority: Minor > Labels: KryoSerializer > > Symptom: > flink program can sucessfully read and process single ORC file from > HDFS,whatever given reading path is file's parent folder or specific file > path. However,I put them together in the same folder and program read that > folder, the following error always occurs. > {color:#cc7832}val {color}configHadoop = {color:#cc7832}new > {color}org.apache.hadoop.conf.Configuration() > configHadoop.set({color:#6a8759}"HADOOP_USER_NAME"{color}{color:#cc7832}, > {color}{color:#6a8759}"user"{color}) > configHadoop.set({color:#6a8759}"fs.defaultFS"{color}{color:#cc7832}, > {color}{color:#6a8759}"xx.xxx.xx.xx"{color}) > {color:#cc7832}val {color}env = ExecutionEnvironment.getExecutionEnvironment > {color:#cc7832}val {color}bTableEnv = > TableEnvironment.getTableEnvironment(env) > {color:#cc7832}val {color}orcTableSource = OrcTableSource.builder() > {color:#808080}// path to ORC file(s). NOTE: By default, directories are > recursively scanned.{color} .path(inPath) > {color:#808080}// schema of ORC files{color} > .forOrcSchema({color:#6a8759}"struct"{color}) > {color:#808080}// Hadoop configuration{color} > .withConfiguration(configHadoop) > {color:#808080}// build OrcTableSource{color} .build() > > The following is stack info > Root exception > Timestamp: 2019-08-08, 20:15:05 > java.lang.Exception: The data preparation for task 'CHAIN GroupReduce > (GroupReduce at > com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) > -> Map (Key Extractor)' , caused an error: Error obtaining the sorted input: > Thread 'SortMerger spilling thread' terminated due to an exception: Index: > 97, Size: 17 > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479) > at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: Error obtaining the sorted input: > Thread 'SortMerger spilling thread' terminated due to an exception: Index: > 97, Size: 17 > at > org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650) > at > org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108) > at > org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99) > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473) > ... 3 more > Caused by: java.io.IOException: Thread 'SortMerger spilling thread' > terminated due to an exception: Index: 97, Size: 17 > at > org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831) > Caused by: java.lang.IndexOutOfBoundsException: Index: 97, Size: 17 > at java.util.ArrayList.rangeCheck(ArrayList.java:653) > at java.util.ArrayList.get(ArrayList.java:429) > at > com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42) > at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:759) > at > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:315) > at > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:335) > at > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:350) > at > org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) > at > org.apache.flink.api.scala.
[jira] [Updated] (FLINK-13655) Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception
[ https://issues.apache.org/jira/browse/FLINK-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiJun updated FLINK-13655: -- Description: Symptom: flink program can sucessfully read and process single ORC file from HDFS,whatever given reading path is file's parent folder or specific file path. However,I put them together in the same folder and program reads that folder, the following error always occurs. {color:#cc7832}val {color}configHadoop = {color:#cc7832}new {color}org.apache.hadoop.conf.Configuration() configHadoop.set({color:#6a8759}"HADOOP_USER_NAME"{color}{color:#cc7832}, {color}{color:#6a8759}"user"{color}) configHadoop.set({color:#6a8759}"fs.defaultFS"{color}{color:#cc7832}, {color}{color:#6a8759}"xx.xxx.xx.xx"{color}) {color:#cc7832}val {color}env = ExecutionEnvironment.getExecutionEnvironment {color:#cc7832}val {color}bTableEnv = TableEnvironment.getTableEnvironment(env) {color:#cc7832}val {color}orcTableSource = OrcTableSource.builder() {color:#808080}// path to ORC file(s). NOTE: By default, directories are recursively scanned.{color} .path(inPath) {color:#808080}// schema of ORC files{color} .forOrcSchema({color:#6a8759}"struct"{color}) {color:#808080}// Hadoop configuration{color} .withConfiguration(configHadoop) {color:#808080}// build OrcTableSource{color} .build() The following is stack info Root exception Timestamp: 2019-08-08, 20:15:05 java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650) at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108) at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473) ... 3 more Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831) Caused by: java.lang.IndexOutOfBoundsException: Index: 97, Size: 17 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42) at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:759) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:315) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:335) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:350) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.api.scala.typeutils.TraversableSerializer.copy(TraversableSerializer.scala:84) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:519) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1375) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:827) CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor) (305/480) Timestamp: 2019-08-08, 20:15:05 Location: LF-BCC-POD0-172-21-60-234.hadoop.jd.local:15837 java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spil
[jira] [Updated] (FLINK-13655) Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception
[ https://issues.apache.org/jira/browse/FLINK-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiJun updated FLINK-13655: -- Description: Symptom: flink program can sucessfully read and process single ORC file from HDFS,whatever given reading path is file's parent folder or specific file path. However,I put them together in the same folder and program read that folder, the following error always occurs. {color:#cc7832}val {color}configHadoop = {color:#cc7832}new {color}org.apache.hadoop.conf.Configuration() configHadoop.set({color:#6a8759}"HADOOP_USER_NAME"{color}{color:#cc7832}, {color}{color:#6a8759}"user"{color}) configHadoop.set({color:#6a8759}"fs.defaultFS"{color}{color:#cc7832}, {color}{color:#6a8759}"xx.xxx.xx.xx"{color}) {color:#cc7832}val {color}env = ExecutionEnvironment.getExecutionEnvironment {color:#cc7832}val {color}bTableEnv = TableEnvironment.getTableEnvironment(env) {color:#cc7832}val {color}orcTableSource = OrcTableSource.builder() {color:#808080}// path to ORC file(s). NOTE: By default, directories are recursively scanned.{color} .path(inPath) {color:#808080}// schema of ORC files{color} .forOrcSchema({color:#6a8759}"struct"{color}) {color:#808080}// Hadoop configuration{color} .withConfiguration(configHadoop) {color:#808080}// build OrcTableSource{color} .build() The following is stack info Root exception Timestamp: 2019-08-08, 20:15:05 java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650) at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108) at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473) ... 3 more Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831) Caused by: java.lang.IndexOutOfBoundsException: Index: 97, Size: 17 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42) at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:759) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:315) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:335) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:350) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.api.scala.typeutils.TraversableSerializer.copy(TraversableSerializer.scala:84) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:519) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1375) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:827) CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor) (305/480) Timestamp: 2019-08-08, 20:15:05 Location: LF-BCC-POD0-172-21-60-234.hadoop.jd.local:15837 java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spill
[GitHub] [flink] flinkbot edited a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala
flinkbot edited a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala URL: https://github.com/apache/flink/pull/8906#issuecomment-519474213 ## CI report: * bea9f0b86c27d30c763e8578edc5ef46a7d76e9e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122421031) * a31d440b058e9fbe2d0a908c4a4fb63750685340 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122532295) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangpeibin713 removed a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala
wangpeibin713 removed a comment on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala URL: https://github.com/apache/flink/pull/8906#issuecomment-519753292 @flinkbot restart travis-ci This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangpeibin713 commented on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala
wangpeibin713 commented on issue #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala URL: https://github.com/apache/flink/pull/8906#issuecomment-519753292 @flinkbot restart travis-ci This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13510) Show fail attemp for subtask in timelime
[ https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903507#comment-16903507 ] lining commented on FLINK-13510: [~till.rohrmann], Now failover will cancle all tasks. After failover, can not cancled attempts of task schedule timeline. If failover spent long time, it's difficult to find the reason. Show all attempts of tasks timeline, we can analyze which attempts spend long time to cancle. > Show fail attemp for subtask in timelime > > > Key: FLINK-13510 > URL: https://issues.apache.org/jira/browse/FLINK-13510 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST, Runtime / Web Frontend >Reporter: lining >Priority: Major > > Now, user just can see subtask current attempt in timeline. If job failover, > can not see some has cancled task timeline. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner
flinkbot edited a comment on issue #9394: [FLINK-13547][table-planner-blink] Verify and correct string function's semantic for Blink planner URL: https://github.com/apache/flink/pull/9394#issuecomment-519467299 ## CI report: * 861d120d83964d267c95d024993b82186b1e6e7a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122417819) * a423b9b46d10f61091bd70aa0ea1bd2367fd25a0 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122531452) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangpeibin713 opened a new pull request #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala
wangpeibin713 opened a new pull request #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala URL: https://github.com/apache/flink/pull/8906 ## What is the purpose of the change - The goal is to fix the findbugs warning in AggregationsFunctio.scala https://issues.apache.org/jira/browse/FLINK-13008 ## Brief change log - *throw the unsupportedOperationException rather than return it* ## Verifying this change * This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangpeibin713 closed pull request #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala
wangpeibin713 closed pull request #8906: [FLINK-13008]fix the findbugs warning in AggregationsFunctio.scala URL: https://github.com/apache/flink/pull/8906 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info
flinkbot edited a comment on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info URL: https://github.com/apache/flink/pull/9347#issuecomment-517895998 ## CI report: * 212cd6c24fdb696aa13bed1cfff875f9bfc01d09 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/121831822) * fa92c963b1f41d75b1399d41708d88cd47a632e9 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122530462) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9364: [FLINK-13593][checkpointing] Prevent failing the wrong execution attempt in CheckpointFailureManager
flinkbot edited a comment on issue #9364: [FLINK-13593][checkpointing] Prevent failing the wrong execution attempt in CheckpointFailureManager URL: https://github.com/apache/flink/pull/9364#issuecomment-518519629 ## CI report: * db9e30b162a1f8f7958a3e1a5ff61214f205de1c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122057050) * df05d6d8946ec764655082180d893918bfc2a290 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122180572) * d35d5392868ee76b9869a2f62fb23ddc443498fa : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122472526) * efbf1541fd5e495af8316cd19af24d7e93be78ac : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122525773) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13340) Add more Kafka topic option of flink-connector-kafka
[ https://issues.apache.org/jira/browse/FLINK-13340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903496#comment-16903496 ] DuBin commented on FLINK-13340: --- hi [~twalthr], can you please help review this PR? > Add more Kafka topic option of flink-connector-kafka > > > Key: FLINK-13340 > URL: https://issues.apache.org/jira/browse/FLINK-13340 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka, Table SQL / API >Affects Versions: 1.8.1 >Reporter: DuBin >Assignee: DuBin >Priority: Major > Labels: features, pull-request-available > Original Estimate: 48h > Time Spent: 10m > Remaining Estimate: 47h 50m > > Currently, only 'topic' option implemented in the Kafka Connector Descriptor, > we can only use it like : > {code:java} > val env = StreamExecutionEnvironment.getExecutionEnvironment > val tableEnv = StreamTableEnvironment.create(env) > tableEnv > .connect( > new Kafka() > .version("0.11") > .topic("test-flink-1") > .startFromEarliest() > .property("zookeeper.connect", "localhost:2181") > .property("bootstrap.servers", "localhost:9092")) > .withFormat( > new Json() > .deriveSchema() > ) > .withSchema( > new Schema() > .field("name", Types.STRING) > .field("age", Types.STRING) > ){code} > but we cannot consume multiple topics or a topic regex pattern. > Here is my thoughts: > {code:java} > .topic("test-flink-1") > //.topics("test-flink-1,test-flink-2") or topics(List > topics) > //.subscriptionPattern("test-flink-.*") or > subscriptionPattern(Pattern pattern) > {code} > I already implement the code on my local env with help of the > FlinkKafkaConsumer, and it works. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner
flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner URL: https://github.com/apache/flink/pull/9363#issuecomment-518265997 ## CI report: * 1fe6c332279c34546ec3db24a574dfd53500d20b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/121971735) * fa3e7406f9664a59efcb448748511b656474e74c : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/122089384) * 28175449cb1d5eb8f318359090ea87e5b2af42d1 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122102858) * 81593a4dcb3573843c1c02cba0cb17abe1693065 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122290876) * 829830f8df1eb814ac44716f230c6aedcfaa5128 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122369808) * 0dcba7f669a5702e5730aac0273eb4a58fce2be9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122460702) * a5052f45f08b0760cdd4a163a87c0be7d6383c42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122479843) * e8a680f4d832f794eb80d6e569d323babc4da518 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122530453) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #9346: [FLINK-13562] [table-planner-blink] fix incorrect input type for local stream group aggregate in FlinkRelMdColumnInterval
godfreyhe commented on a change in pull request #9346: [FLINK-13562] [table-planner-blink] fix incorrect input type for local stream group aggregate in FlinkRelMdColumnInterval URL: https://github.com/apache/flink/pull/9346#discussion_r312306197 ## File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/agg/DistinctAggregateTest.scala ## @@ -117,6 +117,11 @@ class DistinctAggregateTest( util.verifyPlan(sqlQuery) } + @Test + def testTwoDistinctAggregateWithNonDistinctAgg(): Unit = { +util.verifyPlan("SELECT c, SUM(DISTINCT a), SUM(a), COUNT(DISTINCT b) FROM MyTable GROUP BY c") Review comment: yes, to verify the bug This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9346: [FLINK-13562] [table-planner-blink] fix incorrect input type for local stream group aggregate in FlinkRelMdColumnInterval
flinkbot edited a comment on issue #9346: [FLINK-13562] [table-planner-blink] fix incorrect input type for local stream group aggregate in FlinkRelMdColumnInterval URL: https://github.com/apache/flink/pull/9346#issuecomment-517895087 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 255f2f78ad478b7e3cfb13a17af43872d6ad658f (Fri Aug 09 02:01:15 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13562).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] godfreyhe commented on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info
godfreyhe commented on issue #9347: [FLINK-13563] [table-planner-blink] TumblingGroupWindow should implement toString method to explain more info URL: https://github.com/apache/flink/pull/9347#issuecomment-519748929 thanks for reminding @wuchong. rebased This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner
flinkbot edited a comment on issue #9363: [FLINK-13587][table-planner-blink] Fix some operator names are not set in blink planner URL: https://github.com/apache/flink/pull/9363#issuecomment-518263446 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e8a680f4d832f794eb80d6e569d323babc4da518 (Fri Aug 09 01:57:13 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell
flinkbot edited a comment on issue #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell URL: https://github.com/apache/flink/pull/9389#issuecomment-519351757 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1e89a949f197d2feb5c56f737edfe67c250b346b (Fri Aug 09 01:54:08 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13645).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zjffdu commented on issue #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell
zjffdu commented on issue #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell URL: https://github.com/apache/flink/pull/9389#issuecomment-519747709 @wuchong Could you help take a look at it ? Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell
flinkbot edited a comment on issue #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell URL: https://github.com/apache/flink/pull/9389#issuecomment-519351757 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1e89a949f197d2feb5c56f737edfe67c250b346b (Fri Aug 09 01:51:05 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13645).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell
danny0405 commented on a change in pull request #9389: [FLINK-13645][table-planner] Error in code-gen when using blink planner in scala shell URL: https://github.com/apache/flink/pull/9389#discussion_r312304638 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/CodeGeneratorContext.scala ## @@ -603,7 +603,7 @@ class CodeGeneratorContext(val tableConfig: TableConfig) { val byteArray = InstantiationUtil.serializeObject(obj) val objCopy: AnyRef = InstantiationUtil.deserializeObject( byteArray, - obj.getClass.getClassLoader) + Thread.currentThread().getContextClassLoader) references += objCopy Review comment: I'm fine with this, there seems some compile error. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13655) Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception
[ https://issues.apache.org/jira/browse/FLINK-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiJun updated FLINK-13655: -- Description: Symptom: There are two files (about 600MB) with ORC format in HDFS.My flink program can read and process single file sucessfully,whatever reading path is given with that file's parent folder OR file's full path. However,I put them together in the same folder and program read that folder, the following error always occurs. {color:#cc7832}val {color}configHadoop = {color:#cc7832}new {color}org.apache.hadoop.conf.Configuration() configHadoop.set({color:#6a8759}"HADOOP_USER_NAME"{color}{color:#cc7832}, {color}{color:#6a8759}"user"{color}) configHadoop.set({color:#6a8759}"fs.defaultFS"{color}{color:#cc7832}, {color}{color:#6a8759}"xx.xxx.xx.xx"{color}) {color:#cc7832}val {color}env = ExecutionEnvironment.getExecutionEnvironment {color:#cc7832}val {color}bTableEnv = TableEnvironment.getTableEnvironment(env) {color:#cc7832}val {color}orcTableSource = OrcTableSource.builder() {color:#808080}// path to ORC file(s). NOTE: By default, directories are recursively scanned.{color} .path(inPath) {color:#808080}// schema of ORC files{color} .forOrcSchema({color:#6a8759}"struct"{color}) {color:#808080}// Hadoop configuration{color} .withConfiguration(configHadoop) {color:#808080}// build OrcTableSource{color} .build() The following is stack info Root exception Timestamp: 2019-08-08, 20:15:05 java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650) at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108) at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473) ... 3 more Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Index: 97, Size: 17 at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831) Caused by: java.lang.IndexOutOfBoundsException: Index: 97, Size: 17 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42) at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:759) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:315) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:335) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:350) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.api.scala.typeutils.TraversableSerializer.copy(TraversableSerializer.scala:84) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:99) at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:519) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1375) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:827) CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor) (305/480) Timestamp: 2019-08-08, 20:15:05 Location: LF-BCC-POD0-172-21-60-234.hadoop.jd.local:15837 java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at com.jd.risk.flink.analysis.framework.core.EventOfflineServiceFrameWork.startService(EventOfflineServiceFrameWork.scala:41)) -> Map (Key Extractor)' , caused an error: Erro
[jira] [Commented] (FLINK-13603) Flink Table ApI not working with nested Json schema starting From 1.6.x
[ https://issues.apache.org/jira/browse/FLINK-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903441#comment-16903441 ] Rong Rong commented on FLINK-13603: --- to answer your questions [~jacky.du0...@gmail.com], any critical bug fixes should be back ported to older release branches ( at least 2 if I am not mistaken) > Flink Table ApI not working with nested Json schema starting From 1.6.x > --- > > Key: FLINK-13603 > URL: https://issues.apache.org/jira/browse/FLINK-13603 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.6.4, 1.7.2, 1.8.1 >Reporter: Yu Du >Priority: Major > Labels: bug > Attachments: FlinkTableBugCode, jsonSchema.json, jsonSchema2.json, > schema_mapping_error_screenshot .png > > > starting from Flink 1.6.2 , some schema not working when have nested object . > issue like : Caused by: > org.apache.calcite.sql.validate.SqlValidatorException: Column > 'data.interaction.action_type' not found in table > Even we can see that column from Table Schema . > And the same schema and query working on 1.5.2 , but not working for 1.6.x , > 1.7.x and 1.8.x > > I tried to dive into the bug, and found the root cause is calcite library > doesn't mapping the column name with the correct Row type . > I checked Flink 1.6 using the same version of Calcite as Flink 1.5 . Not > sure if Calcite is the root cause of this issue . > Attached with the code sample and two issue json schemas . both examples give > column not found exception . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13603) Flink Table ApI not working with nested Json schema starting From 1.6.x
[ https://issues.apache.org/jira/browse/FLINK-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903440#comment-16903440 ] Rong Rong commented on FLINK-13603: --- Based on what I saw. FLINK-12848 is labeled as improvement so I am not sure whether it can make it to any branch older than 1.9. My suggestion is to fix this and let FLINK-12848 continue as an improvement. Does anyone have any suggestions on this, CC [~twalthr], the original owner of FLINK-9444 ? > Flink Table ApI not working with nested Json schema starting From 1.6.x > --- > > Key: FLINK-13603 > URL: https://issues.apache.org/jira/browse/FLINK-13603 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.6.4, 1.7.2, 1.8.1 >Reporter: Yu Du >Priority: Major > Labels: bug > Attachments: FlinkTableBugCode, jsonSchema.json, jsonSchema2.json, > schema_mapping_error_screenshot .png > > > starting from Flink 1.6.2 , some schema not working when have nested object . > issue like : Caused by: > org.apache.calcite.sql.validate.SqlValidatorException: Column > 'data.interaction.action_type' not found in table > Even we can see that column from Table Schema . > And the same schema and query working on 1.5.2 , but not working for 1.6.x , > 1.7.x and 1.8.x > > I tried to dive into the bug, and found the root cause is calcite library > doesn't mapping the column name with the correct Row type . > I checked Flink 1.6 using the same version of Calcite as Flink 1.5 . Not > sure if Calcite is the root cause of this issue . > Attached with the code sample and two issue json schemas . both examples give > column not found exception . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #9364: [FLINK-13593][checkpointing] Prevent failing the wrong execution attempt in CheckpointFailureManager
flinkbot edited a comment on issue #9364: [FLINK-13593][checkpointing] Prevent failing the wrong execution attempt in CheckpointFailureManager URL: https://github.com/apache/flink/pull/9364#issuecomment-518519629 ## CI report: * db9e30b162a1f8f7958a3e1a5ff61214f205de1c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122057050) * df05d6d8946ec764655082180d893918bfc2a290 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122180572) * d35d5392868ee76b9869a2f62fb23ddc443498fa : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/122472526) * efbf1541fd5e495af8316cd19af24d7e93be78ac : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122525773) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9364: [FLINK-13593][checkpointing] Prevent failing the wrong execution attempt in CheckpointFailureManager
flinkbot edited a comment on issue #9364: [FLINK-13593][checkpointing] Prevent failing the wrong execution attempt in CheckpointFailureManager URL: https://github.com/apache/flink/pull/9364#issuecomment-518518670 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit efbf1541fd5e495af8316cd19af24d7e93be78ac (Fri Aug 09 00:25:36 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13603) Flink Table ApI not working with nested Json schema starting From 1.6.x
[ https://issues.apache.org/jira/browse/FLINK-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903432#comment-16903432 ] Yu Du commented on FLINK-13603: --- hi, [~walterddr] Thanks for reply . yes, I did a local flink-core build and change hashCode() do fix my issue . I think those two issues are pretty similar as FlinkTypeFactory caching the TypeInfo and compare the hashcode() to check the if the TypeInfo in the cache. if this change doesn't have side affect to other Flink components will be happy to see next release fix . Will this change also be applied to Flink 1.6x and 1.7x ? > Flink Table ApI not working with nested Json schema starting From 1.6.x > --- > > Key: FLINK-13603 > URL: https://issues.apache.org/jira/browse/FLINK-13603 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.6.4, 1.7.2, 1.8.1 >Reporter: Yu Du >Priority: Major > Labels: bug > Attachments: FlinkTableBugCode, jsonSchema.json, jsonSchema2.json, > schema_mapping_error_screenshot .png > > > starting from Flink 1.6.2 , some schema not working when have nested object . > issue like : Caused by: > org.apache.calcite.sql.validate.SqlValidatorException: Column > 'data.interaction.action_type' not found in table > Even we can see that column from Table Schema . > And the same schema and query working on 1.5.2 , but not working for 1.6.x , > 1.7.x and 1.8.x > > I tried to dive into the bug, and found the root cause is calcite library > doesn't mapping the column name with the correct Row type . > I checked Flink 1.6 using the same version of Calcite as Flink 1.5 . Not > sure if Calcite is the root cause of this issue . > Attached with the code sample and two issue json schemas . both examples give > column not found exception . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot commented on issue #9398: [FLINK-13479] [flink-cassandra-connector] Fix for Deterministic ordering for prepared statement
flinkbot commented on issue #9398: [FLINK-13479] [flink-cassandra-connector] Fix for Deterministic ordering for prepared statement URL: https://github.com/apache/flink/pull/9398#issuecomment-519727123 ## CI report: * 08e406e0565f2f945437c4baeebbf0a6c4489ff3 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/122523237) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13479) Cassandra POJO Sink - Prepared Statement query does not have deterministic ordering of columns - causing prepared statement cache overflow
[ https://issues.apache.org/jira/browse/FLINK-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903417#comment-16903417 ] Ronak Thakrar commented on FLINK-13479: --- [~aljoscha] - I have created the pull request - can you please review the request and plan for merging them? [https://github.com/apache/flink/pull/9398] > Cassandra POJO Sink - Prepared Statement query does not have deterministic > ordering of columns - causing prepared statement cache overflow > -- > > Key: FLINK-13479 > URL: https://issues.apache.org/jira/browse/FLINK-13479 > Project: Flink > Issue Type: Improvement > Components: Connectors / Cassandra >Affects Versions: 1.7.2 >Reporter: Ronak Thakrar >Assignee: Ronak Thakrar >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > While using Cassandra POJO Sink as part of Flink Jobs - prepared statements > query string which is automatically generated while inserting the data(using > Mapper.saveQuery method), Cassandra entity does not have deterministic > ordering enforced-so every time column position is changed a new prepared > statement is generated and used. As an effect of that prepared statement > query cache is overflown because every time when insert statement query > string is generated by - columns are in random order. > Following is the detailed explanation for what happens inside the Datastax > java driver([https://datastax-oss.atlassian.net/browse/JAVA-1587]): > The current Mapper uses random ordering of columns when it creates prepared > queries. This is fine when only 1 java client is accessing a cluster (and > assuming the application developer does the correct thing by re-using a > Mapper), since each Mapper will reused prepared statement. However when you > have many java clients accessing a cluster, they will each create their own > permutations of column ordering, and can thrash the prepared statement cache > on the cluster. > I propose that the Mapper uses a TreeMap instead of a HashMap when it builds > its set of AliasedMappedProperty - sorted by the column name > (col.mappedProperty.getMappedName()). This would create a deterministic > ordering of columns, and all java processes accessing the same cluster would > end up with the same prepared queries for the same entities. > This issue is already fixed in the Datastax java driver update version(3.3.1) > which is not used by Flink Cassandra connector (using 3.0.0). > I upgraded the driver version to 3.3.1 locally in Flink Cassandra connector > and tested, it stopped creating new prepared statements with different > ordering of column for the same entity. I have the fix for this issue and > would like to contribute the change and will raise the PR request for the > same. > Flink Cassandra Connector Version: flink-connector-cassandra_2.11 > Flink Version: 1.7.1 > I am creating PR request for the same and which can be merged accordingly and > re released in new minor release or patch release as required. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot commented on issue #9398: [FLINK-13479] [flink-cassandra-connector] Fix for Deterministic ordering for prepared statement
flinkbot commented on issue #9398: [FLINK-13479] [flink-cassandra-connector] Fix for Deterministic ordering for prepared statement URL: https://github.com/apache/flink/pull/9398#issuecomment-519726006 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 08e406e0565f2f945437c4baeebbf0a6c4489ff3 (Thu Aug 08 23:44:18 UTC 2019) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13479) Cassandra POJO Sink - Prepared Statement query does not have deterministic ordering of columns - causing prepared statement cache overflow
[ https://issues.apache.org/jira/browse/FLINK-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13479: --- Labels: pull-request-available (was: ) > Cassandra POJO Sink - Prepared Statement query does not have deterministic > ordering of columns - causing prepared statement cache overflow > -- > > Key: FLINK-13479 > URL: https://issues.apache.org/jira/browse/FLINK-13479 > Project: Flink > Issue Type: Improvement > Components: Connectors / Cassandra >Affects Versions: 1.7.2 >Reporter: Ronak Thakrar >Assignee: Ronak Thakrar >Priority: Major > Labels: pull-request-available > > While using Cassandra POJO Sink as part of Flink Jobs - prepared statements > query string which is automatically generated while inserting the data(using > Mapper.saveQuery method), Cassandra entity does not have deterministic > ordering enforced-so every time column position is changed a new prepared > statement is generated and used. As an effect of that prepared statement > query cache is overflown because every time when insert statement query > string is generated by - columns are in random order. > Following is the detailed explanation for what happens inside the Datastax > java driver([https://datastax-oss.atlassian.net/browse/JAVA-1587]): > The current Mapper uses random ordering of columns when it creates prepared > queries. This is fine when only 1 java client is accessing a cluster (and > assuming the application developer does the correct thing by re-using a > Mapper), since each Mapper will reused prepared statement. However when you > have many java clients accessing a cluster, they will each create their own > permutations of column ordering, and can thrash the prepared statement cache > on the cluster. > I propose that the Mapper uses a TreeMap instead of a HashMap when it builds > its set of AliasedMappedProperty - sorted by the column name > (col.mappedProperty.getMappedName()). This would create a deterministic > ordering of columns, and all java processes accessing the same cluster would > end up with the same prepared queries for the same entities. > This issue is already fixed in the Datastax java driver update version(3.3.1) > which is not used by Flink Cassandra connector (using 3.0.0). > I upgraded the driver version to 3.3.1 locally in Flink Cassandra connector > and tested, it stopped creating new prepared statements with different > ordering of column for the same entity. I have the fix for this issue and > would like to contribute the change and will raise the PR request for the > same. > Flink Cassandra Connector Version: flink-connector-cassandra_2.11 > Flink Version: 1.7.1 > I am creating PR request for the same and which can be merged accordingly and > re released in new minor release or patch release as required. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] ronakthakrar opened a new pull request #9398: [FLINK-13479] [flink-cassandra-connector] Fix for Deterministic ordering for prepared statement
ronakthakrar opened a new pull request #9398: [FLINK-13479] [flink-cassandra-connector] Fix for Deterministic ordering for prepared statement URL: https://github.com/apache/flink/pull/9398 ## What is the purpose of the change This pull request fixes the issue with Cassandra Connector issue FLINK-13479 where the deterministic order of columns in prepared statements is not working. ## Brief change log - Updated the cassandra driver version from 3.0.0 to 3.3.1 where the issue is fixed (https://datastax-oss.atlassian.net/browse/JAVA-1587) ## Verifying this change This is trivial configuration change - only changes the depenndent cassandra driver version. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes - cassandra driver upgraded from 3.0.0 to 3.3.1 - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: No - The S3 file system connector: No ## Documentation - Does this pull request introduce a new feature? No - If yes, how is the feature documented? Not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13603) Flink Table ApI not working with nested Json schema starting From 1.6.x
[ https://issues.apache.org/jira/browse/FLINK-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903404#comment-16903404 ] Rong Rong commented on FLINK-13603: --- Hi [~jacky.du0...@gmail.com]. I wouldn't think they are necessarily the same. Does changing the {{hashCode()}} function resolve your issue? I just did a local change and run through the test on {{flink-core}} and it didn't affect any test committed with FLINK-9444. so I am assuming it is not necessary to change the haseCode function in that PR. This would be a much quicker fix (and I think easier to get this in 1.9 release) > Flink Table ApI not working with nested Json schema starting From 1.6.x > --- > > Key: FLINK-13603 > URL: https://issues.apache.org/jira/browse/FLINK-13603 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.6.4, 1.7.2, 1.8.1 >Reporter: Yu Du >Priority: Major > Labels: bug > Attachments: FlinkTableBugCode, jsonSchema.json, jsonSchema2.json, > schema_mapping_error_screenshot .png > > > starting from Flink 1.6.2 , some schema not working when have nested object . > issue like : Caused by: > org.apache.calcite.sql.validate.SqlValidatorException: Column > 'data.interaction.action_type' not found in table > Even we can see that column from Table Schema . > And the same schema and query working on 1.5.2 , but not working for 1.6.x , > 1.7.x and 1.8.x > > I tried to dive into the bug, and found the root cause is calcite library > doesn't mapping the column name with the correct Row type . > I checked Flink 1.6 using the same version of Calcite as Flink 1.5 . Not > sure if Calcite is the root cause of this issue . > Attached with the code sample and two issue json schemas . both examples give > column not found exception . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] flinkbot edited a comment on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one
flinkbot edited a comment on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one URL: https://github.com/apache/flink/pull/9397#issuecomment-519695998 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8c5feb16a2c3814e378dcc67245c554dd0b67eaa (Thu Aug 08 22:35:44 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13658).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] winitzki commented on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one
winitzki commented on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one URL: https://github.com/apache/flink/pull/9397#issuecomment-519711728 The CI error is `21:38:13.853 [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project flink-parent: Too many files with unapproved license: 1 See RAT report in: /home/travis/build/flink-ci/flink/target/rat.txt -> [Help 1] 21:38:13.854 [ERROR] ` and I have no idea what that means. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one
flinkbot commented on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one URL: https://github.com/apache/flink/pull/9397#issuecomment-519697931 ## CI report: * 8c5feb16a2c3814e378dcc67245c554dd0b67eaa : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122512074) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one
flinkbot commented on issue #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one URL: https://github.com/apache/flink/pull/9397#issuecomment-519695998 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8c5feb16a2c3814e378dcc67245c554dd0b67eaa (Thu Aug 08 21:34:05 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13658).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13658) Combine two triggers into one (for streaming windows)
[ https://issues.apache.org/jira/browse/FLINK-13658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13658: --- Labels: pull-request-available (was: ) > Combine two triggers into one (for streaming windows) > - > > Key: FLINK-13658 > URL: https://issues.apache.org/jira/browse/FLINK-13658 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Sergei Winitzki >Priority: Major > Labels: pull-request-available > > Combine two `Trigger`s into one. This allows users to write in one line a > windowed stream whose windows are defined by max element count together with > a max time delay between windows. > > Presently, Flink documentation and Stack Overflow discussions tell users to > implement such triggers manually as custom triggers. However, the > `TriggerResult` enumeration type can be defined as a monoid, and so two > results can be naturally combined into one. This allows users to combine two > or more triggers automatically. > > This implementation is a Scala-only prototype. I am new to Flink and may not > be able to contribute a fully compliant PR. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [flink] winitzki opened a new pull request #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one
winitzki opened a new pull request #9397: [FLINK-13658] [API / DataStream] Combine two triggers into one URL: https://github.com/apache/flink/pull/9397 This is a new feature: combine two triggers into one. A new case class `TriggerOf[T, W]` provides this functionality. Sample (working) code: ```scala val combinedTrigger = TriggerOf(PurgingTrigger.of(CountTrigger.of[TimeWindow](windowSize)), PurgingTrigger.of(ProcessingTimeTrigger.create())) env.addSource(...) . keyBy(element => ...) .timeWindow(Time.milliseconds(windowTimeoutMs)) .trigger(combinedTrigger) .process(...) .addSink(...) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-13658) Combine two triggers into one (for streaming windows)
Sergei Winitzki created FLINK-13658: --- Summary: Combine two triggers into one (for streaming windows) Key: FLINK-13658 URL: https://issues.apache.org/jira/browse/FLINK-13658 Project: Flink Issue Type: New Feature Components: API / DataStream Reporter: Sergei Winitzki Combine two `Trigger`s into one. This allows users to write in one line a windowed stream whose windows are defined by max element count together with a max time delay between windows. Presently, Flink documentation and Stack Overflow discussions tell users to implement such triggers manually as custom triggers. However, the `TriggerResult` enumeration type can be defined as a monoid, and so two results can be naturally combined into one. This allows users to combine two or more triggers automatically. This implementation is a Scala-only prototype. I am new to Flink and may not be able to contribute a fully compliant PR. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13658) Combine two triggers into one (for streaming windows)
[ https://issues.apache.org/jira/browse/FLINK-13658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903350#comment-16903350 ] Sergei Winitzki commented on FLINK-13658: - {code:java} package org.apache.flink.streaming.api.windowing.triggers import org.apache.flink.streaming.api.windowing.triggers.{Trigger, TriggerResult} import org.apache.flink.streaming.api.windowing.windows.Window object TriggerOf { import org.apache.flink.streaming.api.windowing.triggers.TriggerResult._ /** Combine two [[TriggerResult]] values. This is a monoidal operation. * * @param r1 The first [[TriggerResult]] value. * @param r2 The second [[TriggerResult]] value. * @return A new [[TriggerResult]] value that combines the two values. */ def or(r1: TriggerResult, r2: TriggerResult): TriggerResult = (r1, r2) match { case (CONTINUE, x) ⇒ x case (x, CONTINUE) ⇒ x case (_, FIRE_AND_PURGE) | (FIRE_AND_PURGE, _) ⇒ FIRE_AND_PURGE case (PURGE, FIRE) | (FIRE, PURGE) ⇒ FIRE_AND_PURGE // This could also be defined as `FIRE` or as `PURGE` without violating the monoid associativity law. case (PURGE, PURGE) ⇒ PURGE case (FIRE, FIRE) ⇒ FIRE } // Syntax extension for the combining operation for [[TriggerResult]] values. implicit class OrOps(val r1: TriggerResult) extends AnyVal { def \/(r2: TriggerResult): TriggerResult = or(r1, r2) } } /** Combine two triggers into one. The new trigger fires whenever one of the two previously defined triggers fire. * Both triggers must have the same type of window and the same type of stream elements. * * The firing events (`CONTINUE`, `FIRE`, `PURGE`, `FIRE_AND_PURGE`) are combined according to the monoid operation `\/` defined above. * * @param t1 A first trigger. * @param t2 A second trigger. * @tparam T The type of stream elements. * @tparam W The type of window. */ final case class TriggerOf[T, W <: Window](t1: Trigger[T, W], t2: Trigger[T, W]) extends Trigger[T, W] { import TriggerOf._ override def onElement(element: T, timestamp: Long, window: W, ctx: Trigger.TriggerContext): TriggerResult = t1.onElement(element, timestamp, window, ctx) \/ t2.onElement(element, timestamp, window, ctx) override def onProcessingTime(time: Long, window: W, ctx: Trigger.TriggerContext): TriggerResult = t1.onProcessingTime(time, window, ctx) \/ t2.onProcessingTime(time, window, ctx) override def onEventTime(time: Long, window: W, ctx: Trigger.TriggerContext): TriggerResult = t1.onEventTime(time, window, ctx) \/ t2.onEventTime(time, window, ctx) override def canMerge: Boolean = t1.canMerge && t2.canMerge override def onMerge(window: W, ctx: Trigger.OnMergeContext): Unit = { t1.onMerge(window, ctx) t2.onMerge(window, ctx) } override def clear(window: W, ctx: Trigger.TriggerContext): Unit = { t1.clear(window, ctx) t2.clear(window, ctx) } }{code} > Combine two triggers into one (for streaming windows) > - > > Key: FLINK-13658 > URL: https://issues.apache.org/jira/browse/FLINK-13658 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Sergei Winitzki >Priority: Major > > Combine two `Trigger`s into one. This allows users to write in one line a > windowed stream whose windows are defined by max element count together with > a max time delay between windows. > > Presently, Flink documentation and Stack Overflow discussions tell users to > implement such triggers manually as custom triggers. However, the > `TriggerResult` enumeration type can be defined as a monoid, and so two > results can be naturally combined into one. This allows users to combine two > or more triggers automatically. > > This implementation is a Scala-only prototype. I am new to Flink and may not > be able to contribute a fully compliant PR. -- This message was sent by Atlassian JIRA (v7.6.14#76016)