[jira] [Commented] (FLINK-20386) ClassCastException when lookup join a JDBC table on INT UNSIGNED column
[
https://issues.apache.org/jira/browse/FLINK-20386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247069#comment-17247069
]
jiawen xiao commented on FLINK-20386:
-
ok，i will do a test for other unsigned types.Looking forward to the latest
progress
> ClassCastException when lookup join a JDBC table on INT UNSIGNED column
> ---
>
> Key: FLINK-20386
> URL: https://issues.apache.org/jira/browse/FLINK-20386
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC, Table SQL / Ecosystem
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Jark Wu
>Assignee: jiawen xiao
>Priority: Major
>
> The primary key of the MySQL is an INT UNSIGNED column, but declared INT in
> Flink.
> I know the
> [docs|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/jdbc.html#data-type-mapping]
> say it should be decalred BIGINT in Flink, however, would be better not fail
> the job.
> At least, the exception is hard to understand for users. We can also check
> the schema before start the job.
> {code}
> java.lang.ClassCastException: java.lang.Long cannot be cast to
> java.lang.Integer
> at
> org.apache.flink.table.data.GenericRowData.getInt(GenericRowData.java:149)
> ~[flink-table-blink_2.11-1.11-vvr-2.1.1-SNAPSHOT.jar:1.11-vvr-2.1.1-SNAPSHOT]
> at JoinTableFuncCollector$6460.collect(Unknown Source) ~[?:?]
> at
> org.apache.flink.table.functions.TableFunction.collect(TableFunction.java:203)
> ~[flink-table-blink_2.11-1.11-vvr-2.1.1-SNAPSHOT.jar:1.11-vvr-2.1.1-SNAPSHOT]
> at
> org.apache.flink.connector.jdbc.table.JdbcRowDataLookupFunction.eval(JdbcRowDataLookupFunction.java:162)
> ~[?:?]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] twalthr commented on a change in pull request #14329: [FLINK-18890][table] Update LookupTableSource to the new type system
twalthr commented on a change in pull request #14329:
URL: https://github.com/apache/flink/pull/14329#discussion_r539949533
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/extraction/DataTypeExtractor.java
##
@@ -377,6 +397,11 @@ private void checkForCommonErrors(Type type) {
"Usually, this indicates that class
information is missing or got lost. " +
"Please specify a more concrete class
or treat it as a RAW type.",
Object.class.getName());
+ } else if (INTERNAL_DATA_STRUCTURES.contains(clazz)) {
+ throw extractionError(
+ "Cannot extract a data type from a pure
internal '%s' class. " +
Review comment:
I improved the message to:
```
Cannot extract a data type from an internal '%s' class without further
information. " +
"Please use annotations to define the
full logical type.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13907: [FLINK-19942][Connectors / JDBC]Support sink parallelism configuration to JDBC connector
flinkbot edited a comment on pull request #13907: URL: https://github.com/apache/flink/pull/13907#issuecomment-721230727 ## CI report: * f4f1cf14d4d413c4c87881516215c7d5be64 UNKNOWN * 5ba8c4410ef2616561f46462d156949e8c6b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10745) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10705) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20531) Flink docs are not building anymore due to builder change
[
https://issues.apache.org/jira/browse/FLINK-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247061#comment-17247061
]
Robert Metzger commented on FLINK-20531:
Problem is still not resolved.
Medium and long term approaches to fix the problem:
1. Spark/Kafka approach: The docs are copied from the source repo into the
website repository
https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Websiteupdateprocess
/ http://spark.apache.org/release-process.html
We could push the docs to an s3 bucket using azure (where we also validate the
docs), and use ASFs buildbot only to push the docs to the website?!
Side note for the flink website: Apache Airflow is using GitHub Actions to
automatically publish their website:
https://github.com/apache/airflow-site/blob/master/.github/workflows/build.yml#L65
We could also use GH Actions to build the full website (including the
documentation).
> Flink docs are not building anymore due to builder change
> -
>
> Key: FLINK-20531
> URL: https://issues.apache.org/jira/browse/FLINK-20531
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Blocker
>
> The Flink docs are not building anymore, due to
> {code}
> r1068824 | dfoulks | 2020-12-07 18:53:38 +0100 (Mon, 07 Dec 2020) | 1 line
> Moved bb-slave1 jobs to bb-slave7 and bb-slave2 jobs to bb-slave8
> {code}
> bb-slave2 has "rvm" installed, "bb-slave8" doesn't:
> https://ci.apache.org/builders/flink-docs-release-1.11/builds/161
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13907: [FLINK-19942][Connectors / JDBC]Support sink parallelism configuration to JDBC connector
flinkbot edited a comment on pull request #13907: URL: https://github.com/apache/flink/pull/13907#issuecomment-721230727 ## CI report: * f4f1cf14d4d413c4c87881516215c7d5be64 UNKNOWN * 5ba8c4410ef2616561f46462d156949e8c6b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10705) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10745) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on pull request #13907: [FLINK-19942][Connectors / JDBC]Support sink parallelism configuration to JDBC connector
fsk119 commented on pull request #13907: URL: https://github.com/apache/flink/pull/13907#issuecomment-742297048 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20509) Refactor verifyPlan method in TableTestBase
[
https://issues.apache.org/jira/browse/FLINK-20509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247050#comment-17247050
]
Jark Wu commented on FLINK-20509:
-
What do you think about renaming {{planBefore}}, {{planAfter}}, {{planMiddle}}
to specific plan name, such as {{AST}}, {{Physical Plan}}, {{Exec Plan}}. This
can keep align with the explain result and avoid different meaning of
{{planAfter}}.
Maybe we can also add {{verifyPhysicalPlan}} instead of {{verifyRelPlan}} which
keeps align with the plan names.
> Refactor verifyPlan method in TableTestBase
> ---
>
> Key: FLINK-20509
> URL: https://issues.apache.org/jira/browse/FLINK-20509
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Priority: Major
>
> Currently, we use {{verifyPlan}} method to verify the plan result for both
> {{RelNode}} plan and {{ExecNode}} plan, because their instances are the same.
> But once the implementation of {{RelNode}} and {{ExecNode}} are separated, we
> can't get {{ESTIMATED_COST}} and {{CHANGELOG_MODE}} on {{ExecNode}} plan. So
> in order to make those methods more clear, the {{verifyPlan}} method will be
> separated into two methods, {{verifyRelPlan}} for verifying the {{RelNode}}
> plan, and {{verifyExecPlan}} for verifying the {{ExecNode}} plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20509) Refactor verifyPlan method in TableTestBase
[
https://issues.apache.org/jira/browse/FLINK-20509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247046#comment-17247046
]
godfrey he commented on FLINK-20509:
after an offline discussion with [~jark], we reach the following agreement:
1. keep {{verifyPlan}} method, which will print {{planBefore}}, {{planMiddle}}
and {{planAfter}}. {{planBefore}} is original (un-optimized) logical plan,
corresponding to {{Abstract Syntax Tree}} item in the explain result;
{{planMiddle}} is the optimized physical plan, corresponding to {{Optimized
Physical Plan}} item in the explain result. and {{ESTIMATED_COST}} and
{{CHANGELOG_MODE}} could be applied on physical plan; {{planAfter}} is the
optimized execution plan, corresponding to {{Optimized Execution Plan}} item
in the explain result. see https://issues.apache.org/jira/browse/FLINK-20478
for more details about explain refactor
2. add {{verifyRelPlan}} method, which will print {{planBefore}} and
{{planAfter}}. {{planBefore}} is original (un-optimized) logical plan,
{{planAfter}} is the optimized physical plan. We could use this method to check
the rule tests and the tests that only care about the calcite optimized result,
such as {{ESTIMATED_COST}} and {{CHANGELOG_MODE}}
3. add {{verifyExecPlan}} method, which will print {{planBefore}} and
{{planAfter}}. {{planBefore}} is original (un-optimized) logical plan,
{{planAfter}} is the optimized execution plan. currently, most test case will
use this method to verify the plan, such as {{JoinTest}}, {{CalcTest}}, etc.
> Refactor verifyPlan method in TableTestBase
> ---
>
> Key: FLINK-20509
> URL: https://issues.apache.org/jira/browse/FLINK-20509
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Priority: Major
>
> Currently, we use {{verifyPlan}} method to verify the plan result for both
> {{RelNode}} plan and {{ExecNode}} plan, because their instances are the same.
> But once the implementation of {{RelNode}} and {{ExecNode}} are separated, we
> can't get {{ESTIMATED_COST}} and {{CHANGELOG_MODE}} on {{ExecNode}} plan. So
> in order to make those methods more clear, the {{verifyPlan}} method will be
> separated into two methods, {{verifyRelPlan}} for verifying the {{RelNode}}
> plan, and {{verifyExecPlan}} for verifying the {{ExecNode}} plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20540) Failed connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC
[
https://issues.apache.org/jira/browse/FLINK-20540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-20540:
---
Assignee: zhangzhao
> Failed connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via
> JDBC
> --
>
> Key: FLINK-20540
> URL: https://issues.apache.org/jira/browse/FLINK-20540
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.12.0, 1.11.1
>Reporter: zhangzhao
>Assignee: zhangzhao
>Priority: Major
> Original Estimate: 1h
> Remaining Estimate: 1h
>
>
> {code:java}
> //代码占位符
> import org.apache.flink.connector.jdbc.catalog.JdbcCatalog
> new JdbcCatalog(name, defaultDatabase, username, password, baseUrl){code}
>
> The baseUrl must be endswith / when instantiate JdbcCatalog.
> But according to [Flink
> document|https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/table/connectors/jdbc.html#usage-of-postgrescatalog]
> and code comments, baseUrl should be support format
> {{"jdbc:postgresql://:"}}
>
> When i use baseUrl "{{jdbc:postgresql://:}}", the error stack is:
> {code:java}
> //代码占位符
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103)
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)\\nCaused by:
> java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not execute application.
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)\\n\\t...
> 7 more\\nCaused by: org.apache.flink.util.FlinkRuntimeException: Could not
> execute application.
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81)
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67)
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)\\n\\t...
> 7 more\\nCaused by:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Failed connecting to
> jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC.
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78)\\n\\t...
> 10 more\\nCaused by: org.apache.flink.table.api.ValidationException: Failed
> connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC.
> org.apache.flink.connector.jdbc.catalog.AbstractJdbcCatalog.open(AbstractJdbcCatalog.java:100)
> org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:191)
> org.apache.flink.table.api.internal.TableEnvImpl.registerCatalog(TableEnvImpl.scala:267)
> com.upai.jobs.TableBodySentFields.registerCatalog(TableBodySentFields.scala:25)
> com.upai.jobs.FusionGifShow$.run(FusionGifShow.scala:28)
> com.upai.jobs.FlinkTask$.delayedEndpoint$com$upai$jobs$FlinkTask$1(FlinkTask.scala:41)
> com.upai.jobs.FlinkTask$delayedInit$body.apply(FlinkTask.scala:11)
> scala.Function0$class.apply$mcV$sp(Function0.scala:34)
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> scala.App$$anonfun$main$1.apply(App.scala:76)
> scala.App$$anonfun$main$1.apply(App.scala:76)
> scala.collection.immutable.List.foreach(List.scala:392)
>
[jira] [Commented] (FLINK-20540) Failed connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC
[
https://issues.apache.org/jira/browse/FLINK-20540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247045#comment-17247045
]
Leonard Xu commented on FLINK-20540:
Happy to hear you're willing to contribute, I think we can fix the doc and
tests in one PR, please ping me if you need a reviewer.
> Failed connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via
> JDBC
> --
>
> Key: FLINK-20540
> URL: https://issues.apache.org/jira/browse/FLINK-20540
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.12.0, 1.11.1
>Reporter: zhangzhao
>Priority: Major
> Original Estimate: 1h
> Remaining Estimate: 1h
>
>
> {code:java}
> //代码占位符
> import org.apache.flink.connector.jdbc.catalog.JdbcCatalog
> new JdbcCatalog(name, defaultDatabase, username, password, baseUrl){code}
>
> The baseUrl must be endswith / when instantiate JdbcCatalog.
> But according to [Flink
> document|https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/table/connectors/jdbc.html#usage-of-postgrescatalog]
> and code comments, baseUrl should be support format
> {{"jdbc:postgresql://:"}}
>
> When i use baseUrl "{{jdbc:postgresql://:}}", the error stack is:
> {code:java}
> //代码占位符
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103)
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)\\nCaused by:
> java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not execute application.
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)\\n\\t...
> 7 more\\nCaused by: org.apache.flink.util.FlinkRuntimeException: Could not
> execute application.
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81)
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67)
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)\\n\\t...
> 7 more\\nCaused by:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Failed connecting to
> jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC.
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78)\\n\\t...
> 10 more\\nCaused by: org.apache.flink.table.api.ValidationException: Failed
> connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC.
> org.apache.flink.connector.jdbc.catalog.AbstractJdbcCatalog.open(AbstractJdbcCatalog.java:100)
> org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:191)
> org.apache.flink.table.api.internal.TableEnvImpl.registerCatalog(TableEnvImpl.scala:267)
> com.upai.jobs.TableBodySentFields.registerCatalog(TableBodySentFields.scala:25)
> com.upai.jobs.FusionGifShow$.run(FusionGifShow.scala:28)
> com.upai.jobs.FlinkTask$.delayedEndpoint$com$upai$jobs$FlinkTask$1(FlinkTask.scala:41)
> com.upai.jobs.FlinkTask$delayedInit$body.apply(FlinkTask.scala:11)
> scala.Function0$class.apply$mcV$sp(Function0.scala:34)
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> scala.App$$anonfun$main$1.apply(App.scala:76)
>
[GitHub] [flink] flinkbot edited a comment on pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
flinkbot edited a comment on pull request #14315: URL: https://github.com/apache/flink/pull/14315#issuecomment-739151875 ## CI report: * a1e0179a7a99f8f4783954d142b8135f810dfd71 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10735) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20556) org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.
Zezheng Qin created FLINK-20556: --- Summary: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. Key: FLINK-20556 URL: https://issues.apache.org/jira/browse/FLINK-20556 Project: Flink Issue Type: Bug Components: API / Scala, Client / Job Submission, Table SQL / Runtime Affects Versions: 1.11.2 Environment: flink-1.11.2 IntelliJ IDEA 2019.2 x64 window10 scala Reporter: Zezheng Qin Attachments: bug-log.log, flink-datahub-bug.zip if i defined a pojo class in a package object, the job excute failly. but the job can excute successfully what if i defined a pojo class in the package -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20540) Failed connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC
[
https://issues.apache.org/jira/browse/FLINK-20540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247042#comment-17247042
]
zhangzhao commented on FLINK-20540:
---
Ok, i will submit 2 pull requests to fix document and the tests this week.
> Failed connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via
> JDBC
> --
>
> Key: FLINK-20540
> URL: https://issues.apache.org/jira/browse/FLINK-20540
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.12.0, 1.11.1
>Reporter: zhangzhao
>Priority: Major
> Original Estimate: 1h
> Remaining Estimate: 1h
>
>
> {code:java}
> //代码占位符
> import org.apache.flink.connector.jdbc.catalog.JdbcCatalog
> new JdbcCatalog(name, defaultDatabase, username, password, baseUrl){code}
>
> The baseUrl must be endswith / when instantiate JdbcCatalog.
> But according to [Flink
> document|https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/table/connectors/jdbc.html#usage-of-postgrescatalog]
> and code comments, baseUrl should be support format
> {{"jdbc:postgresql://:"}}
>
> When i use baseUrl "{{jdbc:postgresql://:}}", the error stack is:
> {code:java}
> //代码占位符
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103)
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)\\nCaused by:
> java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not execute application.
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)\\n\\t...
> 7 more\\nCaused by: org.apache.flink.util.FlinkRuntimeException: Could not
> execute application.
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81)
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67)
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100)
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)\\n\\t...
> 7 more\\nCaused by:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Failed connecting to
> jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC.
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78)\\n\\t...
> 10 more\\nCaused by: org.apache.flink.table.api.ValidationException: Failed
> connecting to jdbc:postgresql://flink-postgres.cdn-flink:5432flink via JDBC.
> org.apache.flink.connector.jdbc.catalog.AbstractJdbcCatalog.open(AbstractJdbcCatalog.java:100)
> org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:191)
> org.apache.flink.table.api.internal.TableEnvImpl.registerCatalog(TableEnvImpl.scala:267)
> com.upai.jobs.TableBodySentFields.registerCatalog(TableBodySentFields.scala:25)
> com.upai.jobs.FusionGifShow$.run(FusionGifShow.scala:28)
> com.upai.jobs.FlinkTask$.delayedEndpoint$com$upai$jobs$FlinkTask$1(FlinkTask.scala:41)
> com.upai.jobs.FlinkTask$delayedInit$body.apply(FlinkTask.scala:11)
> scala.Function0$class.apply$mcV$sp(Function0.scala:34)
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> scala.App$$anonfun$main$1.apply(App.scala:76)
> scala.App$$anonfun$main$1.apply(App.scala:76)
>
[jira] [Updated] (FLINK-20555) 404 url dependency error in Flink Website
[ https://issues.apache.org/jira/browse/FLINK-20555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DaiXinyu updated FLINK-20555: - Issue Type: Bug (was: New Feature) > 404 url dependency error in Flink Website > - > > Key: FLINK-20555 > URL: https://issues.apache.org/jira/browse/FLINK-20555 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: DaiXinyu >Priority: Minor > Attachments: image-2020-12-10-14-21-42-372.png, > image-2020-12-10-14-22-12-678.png > > > in [Download page|https://flink.apache.org/downloads.html#apache-flink-1112] > !image-2020-12-10-14-21-42-372.png!!image-2020-12-10-14-22-12-678.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20555) 404 url dependency error in Flink Website
[ https://issues.apache.org/jira/browse/FLINK-20555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DaiXinyu updated FLINK-20555: - Description: in [Download page|https://flink.apache.org/downloads.html#apache-flink-1112] !image-2020-12-10-14-21-42-372.png!!image-2020-12-10-14-22-12-678.png! (was: in [Download page |#apache-flink-1112]!image-2020-12-10-14-21-42-372.png!!image-2020-12-10-14-22-12-678.png!) > 404 url dependency error in Flink Website > - > > Key: FLINK-20555 > URL: https://issues.apache.org/jira/browse/FLINK-20555 > Project: Flink > Issue Type: New Feature > Components: Project Website >Reporter: DaiXinyu >Priority: Minor > Attachments: image-2020-12-10-14-21-42-372.png, > image-2020-12-10-14-22-12-678.png > > > in [Download page|https://flink.apache.org/downloads.html#apache-flink-1112] > !image-2020-12-10-14-21-42-372.png!!image-2020-12-10-14-22-12-678.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20555) 404 url dependency error in Flink Website
DaiXinyu created FLINK-20555: Summary: 404 url dependency error in Flink Website Key: FLINK-20555 URL: https://issues.apache.org/jira/browse/FLINK-20555 Project: Flink Issue Type: New Feature Components: Project Website Reporter: DaiXinyu Attachments: image-2020-12-10-14-21-42-372.png, image-2020-12-10-14-22-12-678.png in [Download page |#apache-flink-1112]!image-2020-12-10-14-21-42-372.png!!image-2020-12-10-14-22-12-678.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20552) JdbcDynamicTableSink doesn't sink buffered data on checkpoint
[ https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-20552: --- Assignee: mei jie > JdbcDynamicTableSink doesn't sink buffered data on checkpoint > - > > Key: FLINK-20552 > URL: https://issues.apache.org/jira/browse/FLINK-20552 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: mei jie >Assignee: mei jie >Priority: Major > Labels: starter > > JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when > createSinkTransformation at CommonPhysicalSink class. but > OutputFormatSinkFunction don't implement CheckpointedFunction interface, so > the flush method of JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20552) JdbcDynamicTableSink doesn't sink buffered data on checkpoint
[
https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247034#comment-17247034
]
Jark Wu commented on FLINK-20552:
-
+1 to [~Leonard Xu]'s proposal.
We should also add a proper testing for this. An easier way to test this is
using {{OneInputStreamOperatorTestHarness}}.
You can have a look at the
{{GroupAggregateHarnessTest#testAggregateWithRetraction}} about how to create
an operator harness by using SQL. And have a look at
{{RowTimeDeduplicateFunctionTest#testRowTimeDeduplicateKeepLastRow(boolean,
boolean, java.util.List)}} about how to snapshot and restore.
> JdbcDynamicTableSink doesn't sink buffered data on checkpoint
> -
>
> Key: FLINK-20552
> URL: https://issues.apache.org/jira/browse/FLINK-20552
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC, Table SQL / Ecosystem
>Reporter: mei jie
>Priority: Major
> Labels: starter
>
> JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when
> createSinkTransformation at CommonPhysicalSink class. but
> OutputFormatSinkFunction don't implement CheckpointedFunction interface, so
> the flush method of JdbcBatchingOutputFormat can't be called when checkpoint
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20506) Support FlatMap Operation in Python Table API
[
https://issues.apache.org/jira/browse/FLINK-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-20506.
---
Resolution: Fixed
Merged to master via 0a6e457e6b2bff9acc25e45c3083fc12a95fd717
> Support FlatMap Operation in Python Table API
> -
>
> Key: FLINK-20506
> URL: https://issues.apache.org/jira/browse/FLINK-20506
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Support Python UDTF for FlatMap Operation in Python Table API
> The usage:
> {code:java}
> t = ... # type: Table, table schema: [a: String, b: Int, c: Int]
> # flat_map Python UDTF
> flat_map_func = udtf(lambda x: for i in range(x): yield Row(i + 1, i * i),
>result_types=[DataTypes.INT(), DataTypes.INT()]
> t.flat_map(flat_map_func(t.b)).alias("a", "b"){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14328: [FLINK-19659][table-planner] Array type supports equals and not_equal…
flinkbot edited a comment on pull request #14328: URL: https://github.com/apache/flink/pull/14328#issuecomment-740445577 ## CI report: * d05d4f7df4446697093e8feea44c75b8a3a60722 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10722) * 3d82f6f5f542bb0edeaf4faaefcf59e910d43a2c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10742) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu closed pull request #14352: [FLINK-20506][python] Support FlatMap Operation in Python Table API
dianfu closed pull request #14352: URL: https://github.com/apache/flink/pull/14352 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20552) JdbcDynamicTableSink doesn't sink buffered data on checkpoint
[ https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247031#comment-17247031 ] mei jie commented on FLINK-20552: - Hi, [~Leonard Xu] I will fix this by the method you mentioned > JdbcDynamicTableSink doesn't sink buffered data on checkpoint > - > > Key: FLINK-20552 > URL: https://issues.apache.org/jira/browse/FLINK-20552 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: mei jie >Priority: Major > Labels: starter > > JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when > createSinkTransformation at CommonPhysicalSink class. but > OutputFormatSinkFunction don't implement CheckpointedFunction interface, so > the flush method of JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu commented on a change in pull request #14352: [FLINK-20506][python] Support FlatMap Operation in Python Table API
dianfu commented on a change in pull request #14352: URL: https://github.com/apache/flink/pull/14352#discussion_r539861089 ## File path: flink-python/pyflink/table/table.py ## @@ -780,6 +780,29 @@ def map(self, func: Union[str, Expression]) -> 'Table': else: return Table(self._j_table.map(func._j_expr), self._t_env) +def flat_map(self, func: Union[str, Expression]) -> 'Table': +""" +Performs a flatMap operation with an user-defined table function. Review comment: ```suggestion Performs a flatMap operation with a user-defined table function. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20552) JdbcDynamicTableSink doesn't sink buffered data on checkpoint
[ https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-20552: --- Summary: JdbcDynamicTableSink doesn't sink buffered data on checkpoint (was: Jdbc connector don't sink buffer data when checkpoint) > JdbcDynamicTableSink doesn't sink buffered data on checkpoint > - > > Key: FLINK-20552 > URL: https://issues.apache.org/jira/browse/FLINK-20552 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: mei jie >Priority: Major > Labels: starter > > JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when > createSinkTransformation at CommonPhysicalSink class. but > OutputFormatSinkFunction don't implement CheckpointedFunction interface, so > the flush method of JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20547) Batch job fails due to the exception in network stack
[
https://issues.apache.org/jira/browse/FLINK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247029#comment-17247029
]
Yingjie Cao commented on FLINK-20547:
-
I also encountered this issue these days when testing the new optimization for
sort-merge shuffle but I was suspecting that this exception is caused by my new
added code. From this case, I suspect that it is a bug from master branch.
cc [~AHeise].
> Batch job fails due to the exception in network stack
> -
>
> Key: FLINK-20547
> URL: https://issues.apache.org/jira/browse/FLINK-20547
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.13.0
>Reporter: Zhilong Hong
>Priority: Major
> Attachments: inconsistent.tar.gz
>
>
> I run a simple batch job with only two job vertices: a source and a sink.
> The parallelisms of them are both 8000. They are connected via all-to-all
> blocking edges.
> During the running of sink tasks, an exception raises:
> {code:java}
> 2020-12-09 18:43:48,981 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: Sink
> 1 (1595/8000) (08bd4214d6e0dc144e9654f1faaa3b28) switched from RUNNING to
> FAILED on [masked container name] @ [masked address] (dataPort=47872).
> java.io.IOException: java.lang.IllegalStateException: Inconsistent
> availability: expected true
> at
> org.apache.flink.runtime.io.network.partition.consumer.InputChannel.checkError(InputChannel.java:232)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.partition.consumer.RecoveredInputChannel.getNextBuffer(RecoveredInputChannel.java:165)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.waitAndGetNextData(SingleInputGate.java:626)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:603)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.pollNext(SingleInputGate.java:591)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.pollNext(InputGateWithMetrics.java:109)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:142)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:157)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:575)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:539)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102]
> Caused by: java.lang.IllegalStateException: Inconsistent availability:
> expected true
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.checkConsistentAvailability(LocalBufferPool.java:434)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.setNumBuffers(LocalBufferPool.java:564)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.redistributeBuffers(NetworkBufferPool.java:509)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.tryRedistributeBuffers(NetworkBufferPool.java:438)
>
[jira] [Closed] (FLINK-20514) Introduce StreamExecPythonGroupTableAggregateRule and StreamExecPythonGroupTableAggregate
[ https://issues.apache.org/jira/browse/FLINK-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-20514. --- Assignee: Huang Xingbo Resolution: Fixed Merged to master via 47ee32b2a880938086bd362beb82b5fe055a2032 > Introduce StreamExecPythonGroupTableAggregateRule and > StreamExecPythonGroupTableAggregate > - > > Key: FLINK-20514 > URL: https://issues.apache.org/jira/browse/FLINK-20514 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Introduce StreamExecPythonGroupTableAggregateRule and > StreamExecPythonGroupTableAggregate for supporting Stream Python > TableAggregate Function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #14331: [FLINK-20514][python] Introduce StreamExecPythonGroupTableAggregateRule and StreamExecPythonGroupTableAggregate
dianfu closed pull request #14331: URL: https://github.com/apache/flink/pull/14331 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14328: [FLINK-19659][table-planner] Array type supports equals and not_equal…
flinkbot edited a comment on pull request #14328: URL: https://github.com/apache/flink/pull/14328#issuecomment-740445577 ## CI report: * d05d4f7df4446697093e8feea44c75b8a3a60722 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10722) * 3d82f6f5f542bb0edeaf4faaefcf59e910d43a2c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20483) Support Pandas UDF for Map Operation in Python Table API
[
https://issues.apache.org/jira/browse/FLINK-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-20483.
---
Resolution: Fixed
Merged to master via a67a20d51b73ecfa917a581a8b3c096d8908964a
> Support Pandas UDF for Map Operation in Python Table API
>
>
> Key: FLINK-20483
> URL: https://issues.apache.org/jira/browse/FLINK-20483
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Support Pandas UDF for Map Operation in Python Table API
> The usage:
> {code:java}
> t = ... # type: Table, table schema: [a: String, b: Int, c: Int]
> # map Pandas UDF
> import pandas
> pandas_map_func = udf(lambda x, y: pd.concat([x, y], axis=1),
>
> result_type=DataTypes.ROW([DataTypes.FIELD("a",DataTypes.INT()),
> DataTypes.FIELD("b",
> DataTypes.INT())]))
> t.map(pandas_map_func(b, c))
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #14347: [FLINK-20483][python] Support Pandas UDF for Map Operation in Python Table API
dianfu closed pull request #14347: URL: https://github.com/apache/flink/pull/14347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14352: [FLINK-20506][python] Support FlatMap Operation in Python Table API
flinkbot edited a comment on pull request #14352: URL: https://github.com/apache/flink/pull/14352#issuecomment-742231209 ## CI report: * 840b814bfb02267c41f25cda226b4c0e1442d2d9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10740) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14307: [FLINK-20209][web] Add tolerable failed checkpoints config to web ui
flinkbot edited a comment on pull request #14307: URL: https://github.com/apache/flink/pull/14307#issuecomment-738548986 ## CI report: * 3b92f7190fec3320934756b997f8e97deb7b8e0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10738) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10513) * c305f2af483b203cf280d0da3732b4b39433a124 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10739) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14352: [FLINK-20506][python] Support FlatMap Operation in Python Table API
flinkbot commented on pull request #14352: URL: https://github.com/apache/flink/pull/14352#issuecomment-742231209 ## CI report: * 840b814bfb02267c41f25cda226b4c0e1442d2d9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14307: [FLINK-20209][web] Add tolerable failed checkpoints config to web ui
flinkbot edited a comment on pull request #14307: URL: https://github.com/apache/flink/pull/14307#issuecomment-738548986 ## CI report: * 3b92f7190fec3320934756b997f8e97deb7b8e0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10738) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10513) * c305f2af483b203cf280d0da3732b4b39433a124 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
godfreyhe commented on a change in pull request #14315: URL: https://github.com/apache/flink/pull/14315#discussion_r539834621 ## File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/ValuesTest.xml ## @@ -34,9 +34,9 @@ LogicalProject(a=[$0], b=[$1]), rowType=[RecordType(INTEGER a, DECIMAL(20, 1) b)
[jira] [Commented] (FLINK-20509) Refactor verifyPlan method in TableTestBase
[
https://issues.apache.org/jira/browse/FLINK-20509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247004#comment-17247004
]
godfrey he commented on FLINK-20509:
what if we use `verifyPlan` with JSON_EXEC_PLAN to check the result of a rule
test, which result is logical rel plan, or we use `verifyPlan` with
ESTIMATED_COST to check the exec node plan, which has not concept of cost.
most plan test cases in blink planner are exec plan test, so if we just add a
new method verifyExecPlan, there is also a lot of change. and the concept of
{{verifyPlan}} is bigger than {{verifyExecPlan}}, developers may choose wrong
method and get the unexpected result.
So I trend to separate verifyPlan into verifyRelPlan and verifyExecPlan, the
new methods are clearer than before
> Refactor verifyPlan method in TableTestBase
> ---
>
> Key: FLINK-20509
> URL: https://issues.apache.org/jira/browse/FLINK-20509
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Priority: Major
>
> Currently, we use {{verifyPlan}} method to verify the plan result for both
> {{RelNode}} plan and {{ExecNode}} plan, because their instances are the same.
> But once the implementation of {{RelNode}} and {{ExecNode}} are separated, we
> can't get {{ESTIMATED_COST}} and {{CHANGELOG_MODE}} on {{ExecNode}} plan. So
> in order to make those methods more clear, the {{verifyPlan}} method will be
> separated into two methods, {{verifyRelPlan}} for verifying the {{RelNode}}
> plan, and {{verifyExecPlan}} for verifying the {{ExecNode}} plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20552) Jdbc connector don't sink buffer data when checkpoint
[
https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247002#comment-17247002
]
Leonard Xu commented on FLINK-20552:
Hi, [~meijies]
Thanks for the report， A simple way to fix this is wrapping the
`JdbcBatchingOutputFormat` with `GenericJdbcSinkFunction`
[here|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicTableSink.java#L87%C2%A0]
{code:java}
return OutputFormatProvider.of(builder.build());
=>
return SinkFunctionProvider.of(new GenericJdbcSinkFunction<>(builder.build()));
{code}
And we can add a test for this change, [~meijies] Do you like to help fix this ?
> Jdbc connector don't sink buffer data when checkpoint
> -
>
> Key: FLINK-20552
> URL: https://issues.apache.org/jira/browse/FLINK-20552
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC, Table SQL / Ecosystem
>Reporter: mei jie
>Priority: Major
> Labels: starter
>
> JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when
> createSinkTransformation at CommonPhysicalSink class. but
> OutputFormatSinkFunction don't implement CheckpointedFunction interface, so
> the flush method of JdbcBatchingOutputFormat can't be called when checkpoint
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20554) The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI
[
https://issues.apache.org/jira/browse/FLINK-20554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247000#comment-17247000
]
ming li commented on FLINK-20554:
-
Hi, [~yunta], What do you think?
> The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly
> displayed on the Overview page of the UI
> -
>
> Key: FLINK-20554
> URL: https://issues.apache.org/jira/browse/FLINK-20554
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Web Frontend
>Affects Versions: 1.11.0
>Reporter: ming li
>Priority: Major
> Attachments: image-2020-12-10-11-57-56-888.png
>
>
> The {{Checkpointed Data Size}} of the {{Latest Completed Checkpoint}} always
> shows '-' in the {{Overview}} of the UI.
> !image-2020-12-10-11-57-56-888.png|width=862,height=104!
> I think it should be {{state_size}} instead of {{checkpointed_data_size}} in
> the
> code([https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html#L52]),
> which should fix this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14347: [FLINK-20483][python] Support Pandas UDF for Map Operation in Python Table API
flinkbot edited a comment on pull request #14347: URL: https://github.com/apache/flink/pull/14347#issuecomment-741764329 ## CI report: * 1e95f0353e4eeabfa89314e35ddf5a54d62050f8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10713) * 64e6af6703db598de7035b20c655d28b63fb561e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10737) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20554) The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI
[
https://issues.apache.org/jira/browse/FLINK-20554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ming li updated FLINK-20554:
Description:
The {{Checkpointed Data Size}} of the {{Latest Completed Checkpoint}} always
shows '-' in the {{Overview}} of the UI.
!image-2020-12-10-11-57-56-888.png|width=862,height=104!
I think it should be {{state_size}} instead of {{checkpointed_data_size}} in
the
code([https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html#L52]),
which should fix this problem.
was:
The {{Checkpointed Data Size}} of the {{Latest Completed Checkpoint}} always
shows '-' in the {{Overview}} of the UI.
!image-2020-12-10-11-57-56-888.png|width=862,height=104!
I think it should be ??state_size?? instead of ??checkpointed_data_size?? in
the
code(https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html#L52),
which should fix this problem.
> The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly
> displayed on the Overview page of the UI
> -
>
> Key: FLINK-20554
> URL: https://issues.apache.org/jira/browse/FLINK-20554
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Web Frontend
>Affects Versions: 1.11.0
>Reporter: ming li
>Priority: Major
> Attachments: image-2020-12-10-11-57-56-888.png
>
>
> The {{Checkpointed Data Size}} of the {{Latest Completed Checkpoint}} always
> shows '-' in the {{Overview}} of the UI.
> !image-2020-12-10-11-57-56-888.png|width=862,height=104!
> I think it should be {{state_size}} instead of {{checkpointed_data_size}} in
> the
> code([https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html#L52]),
> which should fix this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20552) Jdbc connector don't sink buffer data when checkpoint
[ https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246999#comment-17246999 ] zhuxiaoshang commented on FLINK-20552: -- Maybe OutputFormatSinkFunction should implements CheckpointedFunction > Jdbc connector don't sink buffer data when checkpoint > - > > Key: FLINK-20552 > URL: https://issues.apache.org/jira/browse/FLINK-20552 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: mei jie >Priority: Major > Labels: starter > > JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when > createSinkTransformation at CommonPhysicalSink class. but > OutputFormatSinkFunction don't implement CheckpointedFunction interface, so > the flush method of JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14307: [FLINK-20209][web] Add tolerable failed checkpoints config to web ui
flinkbot edited a comment on pull request #14307: URL: https://github.com/apache/flink/pull/14307#issuecomment-738548986 ## CI report: * Unknown: [CANCELED](TBD) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20554) The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI
ming li created FLINK-20554:
---
Summary: The Checkpointed Data Size of the Latest Completed
Checkpoint is incorrectly displayed on the Overview page of the UI
Key: FLINK-20554
URL: https://issues.apache.org/jira/browse/FLINK-20554
Project: Flink
Issue Type: Bug
Components: Runtime / Web Frontend
Affects Versions: 1.11.0
Reporter: ming li
Attachments: image-2020-12-10-11-57-56-888.png
The {{Checkpointed Data Size}} of the {{Latest Completed Checkpoint}} always
shows '-' in the {{Overview}} of the UI.
!image-2020-12-10-11-57-56-888.png|width=862,height=104!
I think it should be ??state_size?? instead of ??checkpointed_data_size?? in
the
code(https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html#L52),
which should fix this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14352: [FLINK-20506][python] Support FlatMap Operation in Python Table API
flinkbot commented on pull request #14352: URL: https://github.com/apache/flink/pull/14352#issuecomment-742224292 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 840b814bfb02267c41f25cda226b4c0e1442d2d9 (Thu Dec 10 04:02:56 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20506) Support FlatMap Operation in Python Table API
[
https://issues.apache.org/jira/browse/FLINK-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20506:
---
Labels: pull-request-available (was: )
> Support FlatMap Operation in Python Table API
> -
>
> Key: FLINK-20506
> URL: https://issues.apache.org/jira/browse/FLINK-20506
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Support Python UDTF for FlatMap Operation in Python Table API
> The usage:
> {code:java}
> t = ... # type: Table, table schema: [a: String, b: Int, c: Int]
> # flat_map Python UDTF
> flat_map_func = udtf(lambda x: for i in range(x): yield Row(i + 1, i * i),
>result_types=[DataTypes.INT(), DataTypes.INT()]
> t.flat_map(flat_map_func(t.b)).alias("a", "b"){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20552) Jdbc connector don't sink buffer data when checkpoint
[ https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mei jie updated FLINK-20552: Description: JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when createSinkTransformation at CommonPhysicalSink class. but OutputFormatSinkFunction don't implement CheckpointedFunction interface, so the flush method of JdbcBatchingOutputFormat can't be called when checkpoint (was: JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when createSinkTransformation at CommonPhysicalSink class. but CommonPhysicalSink don't implement CheckpointedFunction interface, so the flush method of JdbcBatchingOutputFormat can't be called when checkpoint) > Jdbc connector don't sink buffer data when checkpoint > - > > Key: FLINK-20552 > URL: https://issues.apache.org/jira/browse/FLINK-20552 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: mei jie >Priority: Major > Labels: starter > > JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when > createSinkTransformation at CommonPhysicalSink class. but > OutputFormatSinkFunction don't implement CheckpointedFunction interface, so > the flush method of JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] HuangXingBo opened a new pull request #14352: [FLINK-20506][python] Support FlatMap Operation in Python Table API
HuangXingBo opened a new pull request #14352: URL: https://github.com/apache/flink/pull/14352 ## What is the purpose of the change *This pull request will support FlatMap Operation in Python Table API* ## Brief change log - *Add `flat_map` api in Python `Table`* ## Verifying this change This change added tests and can be verified as follows: - *IT `test_flat_map` in `test_row_based_operation.py`* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] coolderli commented on pull request #14307: [FLINK-20209][web] Add tolerable failed checkpoints config to web ui
coolderli commented on pull request #14307: URL: https://github.com/apache/flink/pull/14307#issuecomment-742223504 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20552) Jdbc connector don't sink buffer data when checkpoint
[ https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20552: Component/s: Table SQL / Ecosystem > Jdbc connector don't sink buffer data when checkpoint > - > > Key: FLINK-20552 > URL: https://issues.apache.org/jira/browse/FLINK-20552 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: mei jie >Priority: Major > > JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when > createSinkTransformation at CommonPhysicalSink class. but CommonPhysicalSink > don't implement CheckpointedFunction interface, so the flush method of > JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20552) Jdbc connector don't sink buffer data when checkpoint
[ https://issues.apache.org/jira/browse/FLINK-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20552: Labels: starter (was: ) > Jdbc connector don't sink buffer data when checkpoint > - > > Key: FLINK-20552 > URL: https://issues.apache.org/jira/browse/FLINK-20552 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: mei jie >Priority: Major > Labels: starter > > JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when > createSinkTransformation at CommonPhysicalSink class. but CommonPhysicalSink > don't implement CheckpointedFunction interface, so the flush method of > JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] coolderli commented on pull request #14307: [FLINK-20209][web] Add tolerable failed checkpoints config to web ui
coolderli commented on pull request #14307: URL: https://github.com/apache/flink/pull/14307#issuecomment-742223340 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20546) Misuse of the method in KafkaDynamicTableFactoryTest
[ https://issues.apache.org/jira/browse/FLINK-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-20546: --- Assignee: zoucao > Misuse of the method in KafkaDynamicTableFactoryTest > > > Key: FLINK-20546 > URL: https://issues.apache.org/jira/browse/FLINK-20546 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.0 >Reporter: zoucao >Assignee: zoucao >Priority: Major > > hi all, I am reading something about DynamicTable Source/Sink Factory > recently, and one thing let me confused. In `KafkaDynamicTableFactoryTest`, > `getBasicSourceOptions()` and `getBasicSinkOptions()` are created for source > table and sink table, respectively. However, in the situation of validating > sink table, `getBasicSourceOptions()` is called, for example in method > `testSinkWithTopicListOrTopicPattern`、`testInvalidSinkSemantic` etc. > Although no exception is thrown, it also should be fixed. WDYT? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20546) Misuse of the method in KafkaDynamicTableFactoryTest
[ https://issues.apache.org/jira/browse/FLINK-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20546: Component/s: Table SQL / Ecosystem > Misuse of the method in KafkaDynamicTableFactoryTest > > > Key: FLINK-20546 > URL: https://issues.apache.org/jira/browse/FLINK-20546 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka, Table SQL / Ecosystem >Affects Versions: 1.12.0 >Reporter: zoucao >Assignee: zoucao >Priority: Major > > hi all, I am reading something about DynamicTable Source/Sink Factory > recently, and one thing let me confused. In `KafkaDynamicTableFactoryTest`, > `getBasicSourceOptions()` and `getBasicSinkOptions()` are created for source > table and sink table, respectively. However, in the situation of validating > sink table, `getBasicSourceOptions()` is called, for example in method > `testSinkWithTopicListOrTopicPattern`、`testInvalidSinkSemantic` etc. > Although no exception is thrown, it also should be fixed. WDYT? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20553) Add end-to-end test case for new Kafka source
Qingsheng Ren created FLINK-20553:
-
Summary: Add end-to-end test case for new Kafka source
Key: FLINK-20553
URL: https://issues.apache.org/jira/browse/FLINK-20553
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.12.0
Reporter: Qingsheng Ren
Fix For: 1.12.1
The new Kafka source needs an E2E test case to be run periodically on CI.
Currently we have one for the old Kafka source under
{{flink-end-to-end-tests}}, and we can modify this and add a configuration for
running the case with new Kafka source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20552) Jdbc connector don't sink buffer data when checkpoint
mei jie created FLINK-20552: --- Summary: Jdbc connector don't sink buffer data when checkpoint Key: FLINK-20552 URL: https://issues.apache.org/jira/browse/FLINK-20552 Project: Flink Issue Type: Bug Components: Connectors / JDBC Reporter: mei jie JdbcBatchingOutputFormat is wrapped to OutputFormatSinkFunction``` when createSinkTransformation at CommonPhysicalSink class. but CommonPhysicalSink don't implement CheckpointedFunction interface, so the flush method of JdbcBatchingOutputFormat can't be called when checkpoint -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14347: [FLINK-20483][python] Support Pandas UDF for Map Operation in Python Table API
flinkbot edited a comment on pull request #14347: URL: https://github.com/apache/flink/pull/14347#issuecomment-741764329 ## CI report: * 1e95f0353e4eeabfa89314e35ddf5a54d62050f8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10713) * 64e6af6703db598de7035b20c655d28b63fb561e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20546) Misuse of the method in KafkaDynamicTableFactoryTest
[ https://issues.apache.org/jira/browse/FLINK-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246996#comment-17246996 ] zoucao commented on FLINK-20546: hi [~jark]，thanks for your reply, Firstly. Of course I am willing to take it. > Misuse of the method in KafkaDynamicTableFactoryTest > > > Key: FLINK-20546 > URL: https://issues.apache.org/jira/browse/FLINK-20546 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.0 >Reporter: zoucao >Priority: Major > > hi all, I am reading something about DynamicTable Source/Sink Factory > recently, and one thing let me confused. In `KafkaDynamicTableFactoryTest`, > `getBasicSourceOptions()` and `getBasicSinkOptions()` are created for source > table and sink table, respectively. However, in the situation of validating > sink table, `getBasicSourceOptions()` is called, for example in method > `testSinkWithTopicListOrTopicPattern`、`testInvalidSinkSemantic` etc. > Although no exception is thrown, it also should be fixed. WDYT? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] godfreyhe commented on a change in pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
godfreyhe commented on a change in pull request #14315:
URL: https://github.com/apache/flink/pull/14315#discussion_r53982
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/utils/ExecNodePlanDumper.java
##
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils;
+
+import org.apache.flink.table.planner.plan.nodes.calcite.LegacySink;
+import org.apache.flink.table.planner.plan.nodes.calcite.Sink;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNode;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeVisitor;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeVisitorImpl;
+import org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An utility class for converting an exec node plan to a string as a tree
style.
+ */
+public class ExecNodePlanDumper {
+
+ /**
+* Converts an {@link ExecNode} tree to a string as a tree style.
+*
+* @param node the ExecNode to convert
+* @return explain plan of ExecNode
+*
+* The following tree of {@link ExecNode}
Review comment:
if there is only one sink (root) node, it's a rooted tree, see
https://en.wikipedia.org/wiki/Tree_(graph_theory)#Rooted_tree
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on pull request #14347: [FLINK-20483][python] Support Pandas UDF for Map Operation in Python Table API
HuangXingBo commented on pull request #14347: URL: https://github.com/apache/flink/pull/14347#issuecomment-742213876 @dianfu Thanks a lot for the review. I have addressed the comments at the latest commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
godfreyhe commented on a change in pull request #14315: URL: https://github.com/apache/flink/pull/14315#discussion_r539817633 ## File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/ValuesTest.xml ## @@ -34,9 +34,9 @@ LogicalProject(a=[$0], b=[$1]), rowType=[RecordType(INTEGER a, DECIMAL(20, 1) b)
[jira] [Updated] (FLINK-20551) Make SQL documentation Blink only
[ https://issues.apache.org/jira/browse/FLINK-20551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20551: Component/s: Table SQL / Ecosystem Documentation > Make SQL documentation Blink only > -- > > Key: FLINK-20551 > URL: https://issues.apache.org/jira/browse/FLINK-20551 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Ecosystem >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > As Blink has been the default planner for some time, and 1.12 now offers > bounded data stream support, we should make the documentation "blink only". > We will update the documentation to assume users are always using the > Blink planner. As the legacy planner still exists we would create a > dedicated legacy planner page for users that have not migrated for whatever > reason - likely dataset interop. On this page, we will clearly list the > features that are not supported by the legacy planner and any semantics > that differ from the Blink planner. > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Make-SQL-docs-Blink-only-td47048.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20546) Misuse of the method in KafkaDynamicTableFactoryTest
[ https://issues.apache.org/jira/browse/FLINK-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246993#comment-17246993 ] Jark Wu commented on FLINK-20546: - I think you are right. We can fix this. Do you want to take this issue? [~zoucao] > Misuse of the method in KafkaDynamicTableFactoryTest > > > Key: FLINK-20546 > URL: https://issues.apache.org/jira/browse/FLINK-20546 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.0 >Reporter: zoucao >Priority: Major > > hi all, I am reading something about DynamicTable Source/Sink Factory > recently, and one thing let me confused. In `KafkaDynamicTableFactoryTest`, > `getBasicSourceOptions()` and `getBasicSinkOptions()` are created for source > table and sink table, respectively. However, in the situation of validating > sink table, `getBasicSourceOptions()` is called, for example in method > `testSinkWithTopicListOrTopicPattern`、`testInvalidSinkSemantic` etc. > Although no exception is thrown, it also should be fixed. WDYT? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Jiayi-Liao edited a comment on pull request #11480: [FLINK-16597] handle exception thrown from close method in OutputFormat
Jiayi-Liao edited a comment on pull request #11480: URL: https://github.com/apache/flink/pull/11480#issuecomment-742210155 @rkhachatryan I notice that the community is doing the unification of streaming and batch. But this #DataSinkTask can only be used in DataSet job, I'm not sure that this work should be continued. Is there any conflicts between this work and the unification? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18202) Introduce Protobuf format
[ https://issues.apache.org/jira/browse/FLINK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246991#comment-17246991 ] Jark Wu commented on FLINK-18202: - #2 sounds good to me. I think maintainability and stability is more important than performance. > Introduce Protobuf format > - > > Key: FLINK-18202 > URL: https://issues.apache.org/jira/browse/FLINK-18202 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Benchao Li >Priority: Major > Attachments: image-2020-06-15-17-18-03-182.png > > > PB[1] is a very famous and wildly used (de)serialization framework. The ML[2] > also has some discussions about this. It's a useful feature. > This issue maybe needs some designs, or a FLIP. > [1] [https://developers.google.com/protocol-buffers] > [2] [http://apache-flink.147419.n8.nabble.com/Flink-SQL-UDF-td3725.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] godfreyhe commented on a change in pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
godfreyhe commented on a change in pull request #14315:
URL: https://github.com/apache/flink/pull/14315#discussion_r539814663
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/FlinkRelNode.scala
##
@@ -89,19 +89,31 @@ trait FlinkRelNode extends RelNode {
val ops = c.getOperands.map(
getExpressionString(_, inFields, localExprsTable, expressionFormat))
c.getOperator match {
- case _ : SqlAsOperator => ops.head
+ case _: SqlAsOperator => ops.head
case _ =>
-expressionFormat match {
- case ExpressionFormat.Infix if ops.size() == 1 =>
-val operand = ops.head
-c.getKind match {
- case IS_FALSE | IS_NOT_FALSE | IS_TRUE | IS_NOT_TRUE |
IS_UNKNOWN | IS_NULL |
- IS_NOT_NULL => s"$operand $op"
- case _ => s"$op($operand)"
-}
- case ExpressionFormat.Infix => s"(${ops.mkString(s" $op ")})"
- case ExpressionFormat.PostFix => s"(${ops.mkString(", ")})$op"
- case ExpressionFormat.Prefix => s"$op(${ops.mkString(", ")})"
+if (ops.size() == 1) {
+ val operand = ops.head
+ c.getKind match {
+case IS_FALSE | IS_NOT_FALSE | IS_TRUE | IS_NOT_TRUE |
IS_UNKNOWN
+ | IS_NULL | IS_NOT_NULL => s"$operand $op"
+case _ => s"$op($operand)"
+ }
+} else {
+ c.getKind match {
+case TIMES | DIVIDE | PLUS | MINUS
Review comment:
The udfs should be prefix format. `my_add(a, b)` for prefix format is
easier readable than `a my_add b` for infix format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Jiayi-Liao commented on pull request #11480: [FLINK-16597] handle exception thrown from close method in OutputFormat
Jiayi-Liao commented on pull request #11480: URL: https://github.com/apache/flink/pull/11480#issuecomment-742210155 @rkhachatryan I notice that the community is doing the unification of streaming and batch. But this #DataSinkTask can only be used in DataSet job, I'm not sure that this work should be continued. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
godfreyhe commented on a change in pull request #14315:
URL: https://github.com/apache/flink/pull/14315#discussion_r539813438
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/utils/ExecNodePlanDumper.java
##
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils;
+
+import org.apache.flink.table.planner.plan.nodes.calcite.LegacySink;
+import org.apache.flink.table.planner.plan.nodes.calcite.Sink;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNode;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeVisitor;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeVisitorImpl;
+import org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An utility class for converting an exec node plan to a string as a tree
style.
+ */
+public class ExecNodePlanDumper {
+
+ /**
+* Converts an {@link ExecNode} tree to a string as a tree style.
+*
+* @param node the ExecNode to convert
+* @return explain plan of ExecNode
+*
+* The following tree of {@link ExecNode}
+* {@code
+*Sink
+* |
+*Join
+* / \
+* Filter1 Filter2
+* \ /
+*Project1
+* |
+* Scan
+* }
+*
+* would be converted to the tree style as following:
+* {@code
+* Sink
+* +- Join
+*:- Filter1
+*: +- Project1(reuse_id=[1])
+*: +- Scan
+*+- Filter2
+* +- Reused(reference_id=[1])
+* }
+* }
+*/
+ public static String treeToString(ExecNode node) {
+ return treeToString(node, new ArrayList<>(), false);
+ }
+
+ /**
+* Converts an {@link ExecNode} tree to a string as a tree style.
+*
+* @param node the ExecNode to convert
+* @param borders node sets that stop visit when meet them
+* @param includingBorders Whether print the border nodes
+* @return the plan of ExecNode
+*/
+ public static String treeToString(ExecNode node, List>
borders, boolean includingBorders) {
+ checkNotNull(node, "node should not be null.");
+ // convert to mutable list
+ List> borderList = new
ArrayList<>(checkNotNull(borders, "borders should not be null."));
+ TreeReuseInfo reuseInfo = new TreeReuseInfo(node, borderList);
+ return doConvertTreeToString(node, reuseInfo, true, borderList,
includingBorders);
+ }
+
+ /**
+* Converts an {@link ExecNode} DAG to a string as a tree style.
+*
+* @param nodes the ExecNodes to convert
+* @return the plan of ExecNode
+*
+* The following DAG of {@link ExecNode}
+* {@code
+* Sink1Sink2
+* ||
+* Filter3 Filter4
+* \ /
+*Join
+* / \
+* Filter1 Filter2
+* \ /
+*Project1
+* |
+* Scan
+* }
+*
+* would be converted to the tree style as following:
+* {@code
Review comment:
how to display the reused (common) plan ? If the result of the example
just contains two trees, each tree will display all nodes from sink to source.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #14337: [FLINK-20512][table-planner-blink] Introduce getDesc(), getOutputType(), replaceInputEdge(int, ExecEdge) methods for ExecNode
godfreyhe commented on a change in pull request #14337:
URL: https://github.com/apache/flink/pull/14337#discussion_r539812153
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNode.java
##
@@ -31,6 +32,18 @@
*/
public interface ExecNode {
+ /**
+* Returns a string which describes this node.
+* TODO rename to `getDescription` once all ExecNodes do not extend
from FlinkPhysicalRel,
+* because RelNode already has `getDescription` method.
+*/
+ String getDesc();
Review comment:
no, if this pr is merged before
https://github.com/apache/flink/pull/14315, I will update ExecPlanDumper based
on `getDesc`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #14337: [FLINK-20512][table-planner-blink] Introduce getDesc(), getOutputType(), replaceInputEdge(int, ExecEdge) methods for ExecNode
godfreyhe commented on a change in pull request #14337:
URL: https://github.com/apache/flink/pull/14337#discussion_r539812153
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNode.java
##
@@ -31,6 +32,18 @@
*/
public interface ExecNode {
+ /**
+* Returns a string which describes this node.
+* TODO rename to `getDescription` once all ExecNodes do not extend
from FlinkPhysicalRel,
+* because RelNode already has `getDescription` method.
+*/
+ String getDesc();
Review comment:
no, if this pr is merged before
https://github.com/apache/flink/pull/14315, I will update
https://github.com/apache/flink/pull/14315. and if
https://github.com/apache/flink/pull/14315 is merged before this pr, I will
update ExecPlanDumper
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17948) Use new type system for SQL Client collect sink
[
https://issues.apache.org/jira/browse/FLINK-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246985#comment-17246985
]
Jark Wu commented on FLINK-17948:
-
A user also reported TIMESTAMP WITH LOCAL TIME ZONE type can't be used in SQL
Client query.
http://apache-flink.147419.n8.nabble.com/flink-sql-postgres-timestamp-td9235.html
> Use new type system for SQL Client collect sink
> ---
>
> Key: FLINK-17948
> URL: https://issues.apache.org/jira/browse/FLINK-17948
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.11.0
> Environment: mysql:
> image: mysql:8.0
> volumes:
> - ./mysql/mktable.sql:/docker-entrypoint-initdb.d/mktable.sql
> environment:
> MYSQL_ROOT_PASSWORD: 123456
> ports:
> - "3306:3306"
>Reporter: Shengkai Fang
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: image-2020-05-26-22-56-43-835.png,
> image-2020-05-26-22-58-02-326.png
>
>
> My job is following:
>
> {code:java}
> CREATE TABLE currency (
> currency_id BIGINT,
> currency_name STRING,
> rate DOUBLE,
> currency_timestamp TIMESTAMP,
> country STRING,
> precise_timestamp TIMESTAMP(6),
> precise_time TIME(6),
> gdp DECIMAL(10, 6)
> ) WITH (
>'connector' = 'jdbc',
>'url' = 'jdbc:mysql://localhost:3306/flink',
>'username' = 'root',
>'password' = '123456',
>'table-name' = 'currency',
>'driver' = 'com.mysql.jdbc.Driver',
>'lookup.cache.max-rows' = '500',
>'lookup.cache.ttl' = '10s',
>'lookup.max-retries' = '3')
> {code}
> When select * from currency, the precision of results are not as same as
> expected. The precision of field precise_timestamp is 3 not 6, and the
> field gdp has more digit as expected.
>
> !image-2020-05-26-22-56-43-835.png!
> The data in mysql is following:
> !image-2020-05-26-22-58-02-326.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17948) Use new type system for SQL Client collect sink
[
https://issues.apache.org/jira/browse/FLINK-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-17948:
Summary: Use new type system for SQL Client collect sink (was: Sql client
lost precision for Timestamp and Decimal Data type)
> Use new type system for SQL Client collect sink
> ---
>
> Key: FLINK-17948
> URL: https://issues.apache.org/jira/browse/FLINK-17948
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.11.0
> Environment: mysql:
> image: mysql:8.0
> volumes:
> - ./mysql/mktable.sql:/docker-entrypoint-initdb.d/mktable.sql
> environment:
> MYSQL_ROOT_PASSWORD: 123456
> ports:
> - "3306:3306"
>Reporter: Shengkai Fang
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: image-2020-05-26-22-56-43-835.png,
> image-2020-05-26-22-58-02-326.png
>
>
> My job is following:
>
> {code:java}
> CREATE TABLE currency (
> currency_id BIGINT,
> currency_name STRING,
> rate DOUBLE,
> currency_timestamp TIMESTAMP,
> country STRING,
> precise_timestamp TIMESTAMP(6),
> precise_time TIME(6),
> gdp DECIMAL(10, 6)
> ) WITH (
>'connector' = 'jdbc',
>'url' = 'jdbc:mysql://localhost:3306/flink',
>'username' = 'root',
>'password' = '123456',
>'table-name' = 'currency',
>'driver' = 'com.mysql.jdbc.Driver',
>'lookup.cache.max-rows' = '500',
>'lookup.cache.ttl' = '10s',
>'lookup.max-retries' = '3')
> {code}
> When select * from currency, the precision of results are not as same as
> expected. The precision of field precise_timestamp is 3 not 6, and the
> field gdp has more digit as expected.
>
> !image-2020-05-26-22-56-43-835.png!
> The data in mysql is following:
> !image-2020-05-26-22-58-02-326.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14331: [FLINK-20514][python] Introduce StreamExecPythonGroupTableAggregateRule and StreamExecPythonGroupTableAggregate
flinkbot edited a comment on pull request #14331: URL: https://github.com/apache/flink/pull/14331#issuecomment-740479953 ## CI report: * b365d4d901d59b59e01b057ce331c64fbd1a5c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10707) * 827e99c471c5ab49c35e5a71c38357426a4f21a5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10736) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
flinkbot edited a comment on pull request #14315: URL: https://github.com/apache/flink/pull/14315#issuecomment-739151875 ## CI report: * c89b7c0522e126bb172acaef9d182d602a4a7d34 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10712) * a1e0179a7a99f8f4783954d142b8135f810dfd71 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10735) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17948) Sql client lost precision for Timestamp and Decimal Data type
[
https://issues.apache.org/jira/browse/FLINK-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-17948:
Fix Version/s: 1.13.0
> Sql client lost precision for Timestamp and Decimal Data type
> -
>
> Key: FLINK-17948
> URL: https://issues.apache.org/jira/browse/FLINK-17948
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.11.0
> Environment: mysql:
> image: mysql:8.0
> volumes:
> - ./mysql/mktable.sql:/docker-entrypoint-initdb.d/mktable.sql
> environment:
> MYSQL_ROOT_PASSWORD: 123456
> ports:
> - "3306:3306"
>Reporter: Shengkai Fang
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: image-2020-05-26-22-56-43-835.png,
> image-2020-05-26-22-58-02-326.png
>
>
> My job is following:
>
> {code:java}
> CREATE TABLE currency (
> currency_id BIGINT,
> currency_name STRING,
> rate DOUBLE,
> currency_timestamp TIMESTAMP,
> country STRING,
> precise_timestamp TIMESTAMP(6),
> precise_time TIME(6),
> gdp DECIMAL(10, 6)
> ) WITH (
>'connector' = 'jdbc',
>'url' = 'jdbc:mysql://localhost:3306/flink',
>'username' = 'root',
>'password' = '123456',
>'table-name' = 'currency',
>'driver' = 'com.mysql.jdbc.Driver',
>'lookup.cache.max-rows' = '500',
>'lookup.cache.ttl' = '10s',
>'lookup.max-retries' = '3')
> {code}
> When select * from currency, the precision of results are not as same as
> expected. The precision of field precise_timestamp is 3 not 6, and the
> field gdp has more digit as expected.
>
> !image-2020-05-26-22-56-43-835.png!
> The data in mysql is following:
> !image-2020-05-26-22-58-02-326.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wenlong88 commented on a change in pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
wenlong88 commented on a change in pull request #14315:
URL: https://github.com/apache/flink/pull/14315#discussion_r539792067
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/FlinkRelNode.scala
##
@@ -89,19 +89,31 @@ trait FlinkRelNode extends RelNode {
val ops = c.getOperands.map(
getExpressionString(_, inFields, localExprsTable, expressionFormat))
c.getOperator match {
- case _ : SqlAsOperator => ops.head
+ case _: SqlAsOperator => ops.head
case _ =>
-expressionFormat match {
- case ExpressionFormat.Infix if ops.size() == 1 =>
-val operand = ops.head
-c.getKind match {
- case IS_FALSE | IS_NOT_FALSE | IS_TRUE | IS_NOT_TRUE |
IS_UNKNOWN | IS_NULL |
- IS_NOT_NULL => s"$operand $op"
- case _ => s"$op($operand)"
-}
- case ExpressionFormat.Infix => s"(${ops.mkString(s" $op ")})"
- case ExpressionFormat.PostFix => s"(${ops.mkString(", ")})$op"
- case ExpressionFormat.Prefix => s"$op(${ops.mkString(", ")})"
+if (ops.size() == 1) {
+ val operand = ops.head
+ c.getKind match {
+case IS_FALSE | IS_NOT_FALSE | IS_TRUE | IS_NOT_TRUE |
IS_UNKNOWN
+ | IS_NULL | IS_NOT_NULL => s"$operand $op"
+case _ => s"$op($operand)"
+ }
+} else {
+ c.getKind match {
+case TIMES | DIVIDE | PLUS | MINUS
Review comment:
what is the purpose to change this? is there any bad case to use Infix
in all expresssion?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wenlong88 commented on a change in pull request #14337: [FLINK-20512][table-planner-blink] Introduce getDesc(), getOutputType(), replaceInputEdge(int, ExecEdge) methods for ExecNode
wenlong88 commented on a change in pull request #14337:
URL: https://github.com/apache/flink/pull/14337#discussion_r539797832
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNode.java
##
@@ -31,6 +32,18 @@
*/
public interface ExecNode {
+ /**
+* Returns a string which describes this node.
+* TODO rename to `getDescription` once all ExecNodes do not extend
from FlinkPhysicalRel,
+* because RelNode already has `getDescription` method.
+*/
+ String getDesc();
Review comment:
this pr should wait https://github.com/apache/flink/pull/14315, and use
this method in ExecPlanDumper right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wenlong88 commented on a change in pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
wenlong88 commented on a change in pull request #14315:
URL: https://github.com/apache/flink/pull/14315#discussion_r539785239
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/utils/ExecNodePlanDumper.java
##
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils;
+
+import org.apache.flink.table.planner.plan.nodes.calcite.LegacySink;
+import org.apache.flink.table.planner.plan.nodes.calcite.Sink;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNode;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeVisitor;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeVisitorImpl;
+import org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An utility class for converting an exec node plan to a string as a tree
style.
+ */
+public class ExecNodePlanDumper {
+
+ /**
+* Converts an {@link ExecNode} tree to a string as a tree style.
+*
+* @param node the ExecNode to convert
+* @return explain plan of ExecNode
+*
+* The following tree of {@link ExecNode}
Review comment:
I think it is a dag not tree?
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/FlinkRelNode.scala
##
@@ -89,19 +89,31 @@ trait FlinkRelNode extends RelNode {
val ops = c.getOperands.map(
getExpressionString(_, inFields, localExprsTable, expressionFormat))
c.getOperator match {
- case _ : SqlAsOperator => ops.head
+ case _: SqlAsOperator => ops.head
case _ =>
-expressionFormat match {
- case ExpressionFormat.Infix if ops.size() == 1 =>
-val operand = ops.head
-c.getKind match {
- case IS_FALSE | IS_NOT_FALSE | IS_TRUE | IS_NOT_TRUE |
IS_UNKNOWN | IS_NULL |
- IS_NOT_NULL => s"$operand $op"
- case _ => s"$op($operand)"
-}
- case ExpressionFormat.Infix => s"(${ops.mkString(s" $op ")})"
- case ExpressionFormat.PostFix => s"(${ops.mkString(", ")})$op"
- case ExpressionFormat.Prefix => s"$op(${ops.mkString(", ")})"
+if (ops.size() == 1) {
+ val operand = ops.head
+ c.getKind match {
+case IS_FALSE | IS_NOT_FALSE | IS_TRUE | IS_NOT_TRUE |
IS_UNKNOWN
+ | IS_NULL | IS_NOT_NULL => s"$operand $op"
+case _ => s"$op($operand)"
+ }
+} else {
+ c.getKind match {
+case TIMES | DIVIDE | PLUS | MINUS
Review comment:
what is the purpose to change this?
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/utils/ExecNodePlanDumper.java
##
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils;
[GitHub] [flink] flinkbot edited a comment on pull request #14331: [FLINK-20514][python] Introduce StreamExecPythonGroupTableAggregateRule and StreamExecPythonGroupTableAggregate
flinkbot edited a comment on pull request #14331: URL: https://github.com/apache/flink/pull/14331#issuecomment-740479953 ## CI report: * b365d4d901d59b59e01b057ce331c64fbd1a5c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10707) * 827e99c471c5ab49c35e5a71c38357426a4f21a5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14315: [FLINK-20478][table-planner-blink] Adjust the explain result for blink planner
flinkbot edited a comment on pull request #14315: URL: https://github.com/apache/flink/pull/14315#issuecomment-739151875 ## CI report: * c89b7c0522e126bb172acaef9d182d602a4a7d34 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10712) * a1e0179a7a99f8f4783954d142b8135f810dfd71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #14347: [FLINK-20483][python] Support Pandas UDF for Map Operation in Python Table API
dianfu commented on a change in pull request #14347: URL: https://github.com/apache/flink/pull/14347#discussion_r539779543 ## File path: flink-python/pyflink/table/tests/test_row_based_operation.py ## @@ -49,6 +49,35 @@ def test_map(self): actual = source_sink_utils.results() self.assert_equals(actual, ["4,9", "3,4", "7,36", "10,81", "5,16"]) +def test_map_pandas(self): Review comment: ```suggestion def test_map_with_pandas_udf(self): ``` ## File path: flink-python/pyflink/table/utils.py ## @@ -38,16 +38,35 @@ def create_array(s, t): "pyarrow.Array (%s)." raise RuntimeError(error_msg % (s.dtype, t), e) -arrays = [create_array( -tz_convert_to_internal(series[i], field_types[i], timezone), -schema.types[i]) for i in range(0, len(schema))] +arrays = [] +for i in range(len(schema)): +s = series[i] +field_type = field_types[i] +schema_type = schema.types[i] +if type(field_type) == RowType: +array_names = [(create_array(s[s.columns[i]], field.type), field.name) Review comment: ```suggestion array_names = [(create_array(s[s.columns[j]], field.type), field.name) ``` ## File path: flink-python/pyflink/table/utils.py ## @@ -38,16 +38,35 @@ def create_array(s, t): "pyarrow.Array (%s)." raise RuntimeError(error_msg % (s.dtype, t), e) -arrays = [create_array( -tz_convert_to_internal(series[i], field_types[i], timezone), -schema.types[i]) for i in range(0, len(schema))] +arrays = [] +for i in range(len(schema)): +s = series[i] +field_type = field_types[i] +schema_type = schema.types[i] +if type(field_type) == RowType: +array_names = [(create_array(s[s.columns[i]], field.type), field.name) + for i, field in enumerate(schema_type)] Review comment: ```suggestion for j, field in enumerate(schema_type)] ``` ## File path: flink-python/pyflink/table/utils.py ## @@ -38,16 +38,35 @@ def create_array(s, t): "pyarrow.Array (%s)." raise RuntimeError(error_msg % (s.dtype, t), e) -arrays = [create_array( -tz_convert_to_internal(series[i], field_types[i], timezone), -schema.types[i]) for i in range(0, len(schema))] +arrays = [] +for i in range(len(schema)): +s = series[i] +field_type = field_types[i] +schema_type = schema.types[i] +if type(field_type) == RowType: +array_names = [(create_array(s[s.columns[i]], field.type), field.name) + for i, field in enumerate(schema_type)] +struct_arrays, struct_names = zip(*array_names) +arrays.append(pa.StructArray.from_arrays(struct_arrays, struct_names)) +else: +arrays.append(create_array( +tz_convert_to_internal(s, field_type, timezone), schema_type)) return pa.RecordBatch.from_arrays(arrays, schema) def arrow_to_pandas(timezone, field_types, batches): +def arrow_column_to_pandas(arrow_column, t: DataType): +if type(t) == RowType: +import pandas as pd +series = [column.to_pandas(date_as_object=True).rename(field.name) + for column, field in zip(arrow_column.flatten(), arrow_column.type)] +s = pd.concat(series, axis=1) +else: +s = arrow_column.to_pandas(date_as_object=True) Review comment: ```suggestion return arrow_column.to_pandas(date_as_object=True) ``` ## File path: flink-python/pyflink/table/utils.py ## @@ -38,16 +38,35 @@ def create_array(s, t): "pyarrow.Array (%s)." raise RuntimeError(error_msg % (s.dtype, t), e) -arrays = [create_array( -tz_convert_to_internal(series[i], field_types[i], timezone), -schema.types[i]) for i in range(0, len(schema))] +arrays = [] +for i in range(len(schema)): +s = series[i] +field_type = field_types[i] +schema_type = schema.types[i] +if type(field_type) == RowType: +array_names = [(create_array(s[s.columns[i]], field.type), field.name) + for i, field in enumerate(schema_type)] +struct_arrays, struct_names = zip(*array_names) +arrays.append(pa.StructArray.from_arrays(struct_arrays, struct_names)) +else: +arrays.append(create_array( +tz_convert_to_internal(s, field_type, timezone), schema_type)) return pa.RecordBatch.from_arrays(arrays, schema) def arrow_to_pandas(timezone, field_types, batches): +def arrow_column_to_pandas(arrow_column, t: DataType): +if type(t) == RowType: +
[jira] [Commented] (FLINK-20416) Need a cached catalog for batch SQL job
[
https://issues.apache.org/jira/browse/FLINK-20416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246973#comment-17246973
]
Rui Li commented on FLINK-20416:
I believe this is in general a useful feature, but I agree with [~jark] that
it's better to propose a design doc first which we can discuss about.
> Need a cached catalog for batch SQL job
> ---
>
> Key: FLINK-20416
> URL: https://issues.apache.org/jira/browse/FLINK-20416
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common, Connectors / Hive, Table SQL / API,
> Table SQL / Planner
>Reporter: Sebastian Liu
>Priority: Major
> Labels: pull-request-available
>
> For OLAP scenarios, There are usually some analytical queries which running
> time is relatively short. These queries are also sensitive to latency. In the
> current Blink sql processing, parse/validate/optimize stages are all need
> meta data from catalog API. But each request to the catalog requires re-run
> of the underlying meta query.
>
> We may need a cached catalog which can cache the table schema and statistic
> info to avoid unnecessary repeated meta requests.
> I have submitted a related PR for adding a genetic cached catalog, which can
> delegate other implementations of {{AbstractCatalog. }}
> {{[https://github.com/apache/flink/pull/14260]}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] Windber opened a new pull request #185: [hotfix] Fix typos in StatefulFunctionModule.java
Windber opened a new pull request #185: URL: https://github.com/apache/flink-statefun/pull/185 hi, find a tiny typo in StatefulFunctionModule.java and fix it: MEGA-INF -> META-INF This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-20492) The SourceOperatorStreamTask should implement cancelTask() and finishTask()
[
https://issues.apache.org/jira/browse/FLINK-20492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin resolved FLINK-20492.
--
Resolution: Fixed
> The SourceOperatorStreamTask should implement cancelTask() and finishTask()
> ---
>
> Key: FLINK-20492
> URL: https://issues.apache.org/jira/browse/FLINK-20492
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.12.0, 1.11.3
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.1
>
>
> The {{SourceOperatorStreamTask}} does not implement {{cancelTask}} and
> {{finishTask}} at this point. This causes resource leak on job cancellation
> of finish.
> Currently there are three cases that a Job may exit.
> # Canceled - {{cancelTask()}} method will be called to close the
> SourceOperator.
> # Stopped with a savepoint - {{finishTask()}} method will be called to close
> the SourceOperator.
> # Runs to the EndOfInput - the SourceOperator is closed in
> {{StreamTask.afterInvoke()}}.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20492) The SourceOperatorStreamTask should implement cancelTask() and finishTask()
[
https://issues.apache.org/jira/browse/FLINK-20492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin updated FLINK-20492:
-
Affects Version/s: 1.11.3
1.12.0
> The SourceOperatorStreamTask should implement cancelTask() and finishTask()
> ---
>
> Key: FLINK-20492
> URL: https://issues.apache.org/jira/browse/FLINK-20492
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.12.0, 1.11.3
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.1
>
>
> The {{SourceOperatorStreamTask}} does not implement {{cancelTask}} and
> {{finishTask}} at this point. This causes resource leak on job cancellation
> of finish.
> Currently there are three cases that a Job may exit.
> # Canceled - {{cancelTask()}} method will be called to close the
> SourceOperator.
> # Stopped with a savepoint - {{finishTask()}} method will be called to close
> the SourceOperator.
> # Runs to the EndOfInput - the SourceOperator is closed in
> {{StreamTask.afterInvoke()}}.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20492) The SourceOperatorStreamTask should implement cancelTask() and finishTask()
[
https://issues.apache.org/jira/browse/FLINK-20492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246933#comment-17246933
]
Jiangjie Qin commented on FLINK-20492:
--
Per discussion in the [PR|https://github.com/apache/flink/pull/14314]. The
correct solution here is closing the {{SourceReader}} in
{{SourceOperator.dispose()}} which will be called in all cases.
The PR has been merged:
master: 168124f99c75e873adc81437c700f85f703e2248
release-1.12: c22bb49443e6d70be5d312acee065fc9b4d95bb2
> The SourceOperatorStreamTask should implement cancelTask() and finishTask()
> ---
>
> Key: FLINK-20492
> URL: https://issues.apache.org/jira/browse/FLINK-20492
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.1
>
>
> The {{SourceOperatorStreamTask}} does not implement {{cancelTask}} and
> {{finishTask}} at this point. This causes resource leak on job cancellation
> of finish.
> Currently there are three cases that a Job may exit.
> # Canceled - {{cancelTask()}} method will be called to close the
> SourceOperator.
> # Stopped with a savepoint - {{finishTask()}} method will be called to close
> the SourceOperator.
> # Runs to the EndOfInput - the SourceOperator is closed in
> {{StreamTask.afterInvoke()}}.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] becketqin closed pull request #14314: [FLINK-20492][runtime] Add implementations of cancelTask() and finish…
becketqin closed pull request #14314: URL: https://github.com/apache/flink/pull/14314 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #14314: [FLINK-20492][runtime] Add implementations of cancelTask() and finish…
becketqin commented on pull request #14314: URL: https://github.com/apache/flink/pull/14314#issuecomment-742157315 Merged to master: 168124f99c75e873adc81437c700f85f703e2248 Merged to release-1.12: c22bb49443e6d70be5d312acee065fc9b4d95bb2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13964: [FLINK-19314][coordination] Add DeclarativeSlotPoolBridge
flinkbot edited a comment on pull request #13964: URL: https://github.com/apache/flink/pull/13964#issuecomment-723100396 ## CI report: * 1b82c38eed09f008282d49f2680257a3b96f0d55 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10726) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14350: [FLINK-20422] Remove site.baseurl from remaining .html templates
sjwiesman commented on pull request #14350: URL: https://github.com/apache/flink/pull/14350#issuecomment-742080380 I'm not entirely sure how to review this. It builds so +1 from me This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14350: [FLINK-20422] Remove site.baseurl from remaining .html templates
sjwiesman commented on a change in pull request #14350: URL: https://github.com/apache/flink/pull/14350#discussion_r539667952 ## File path: docs/_config.yml ## @@ -88,6 +88,7 @@ previous_docs: exclude: - "build_docs.sh" - "check_links.sh" + - "spider.log" Review comment: +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20456) Make streaming SQL concepts more approachable
[ https://issues.apache.org/jira/browse/FLINK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-20456. Resolution: Fixed > Make streaming SQL concepts more approachable > - > > Key: FLINK-20456 > URL: https://issues.apache.org/jira/browse/FLINK-20456 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Ecosystem >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20456) Make streaming SQL concepts more approachable
[ https://issues.apache.org/jira/browse/FLINK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246840#comment-17246840 ] Seth Wiesman commented on FLINK-20456: -- fixed in master: 6539e25cc291aa97f9ea62d8d7a3e74429b38baf > Make streaming SQL concepts more approachable > - > > Key: FLINK-20456 > URL: https://issues.apache.org/jira/browse/FLINK-20456 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Ecosystem >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman closed pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
sjwiesman closed pull request #14292: URL: https://github.com/apache/flink/pull/14292 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
sjwiesman commented on pull request #14292: URL: https://github.com/apache/flink/pull/14292#issuecomment-742074292 thanks for the review, will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
sjwiesman commented on a change in pull request #14292: URL: https://github.com/apache/flink/pull/14292#discussion_r539660475 ## File path: docs/redirects/temporal_table.md ## @@ -0,0 +1,24 @@ +--- +title: Temporal Tables +layout: redirect +redirect: /dev/table/streaming/versioned_table.html Review comment: - Now that you say it, I think legacy makes more sense. That's the content people are expecting to find. And that page already cross-links to versioned_tables. - The `zh` redirect happens automatically This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
sjwiesman commented on a change in pull request #14292: URL: https://github.com/apache/flink/pull/14292#discussion_r539658939 ## File path: docs/redirects/temporal_table.md ## @@ -0,0 +1,24 @@ +--- +title: Temporal Tables +layout: redirect +redirect: /dev/table/streaming/versioned_table.html Review comment: damn, thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #11480: [FLINK-16597] handle exception thrown from close method in OutputFormat
rkhachatryan commented on a change in pull request #11480:
URL: https://github.com/apache/flink/pull/11480#discussion_r539642775
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/DataSinkTask.java
##
@@ -295,7 +298,9 @@ public void cancel() throws Exception {
if (format != null) {
try {
this.format.close();
- } catch (Throwable t) {}
+ } catch (Throwable t) {
+ LOG.error("Error closing format.");
Review comment:
I think `tryCleanupOnError` should be moved here (like it's done in
`OutputFormatSinkFunction` for example).
(logging then need to be adjusted: if `tryCleanupOnError` succeeds then no
need to log `close()` error, at least on ERR level).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-16597) Handle exception thrown from close method in OutputFormat
[
https://issues.apache.org/jira/browse/FLINK-16597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-16597:
--
Fix Version/s: 1.13.0
> Handle exception thrown from close method in OutputFormat
> -
>
> Key: FLINK-16597
> URL: https://issues.apache.org/jira/browse/FLINK-16597
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.10.0
>Reporter: Jiayi Liao
>Assignee: Jiayi Liao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If {{DataSinkTask}} fails or throws an exception, the {{OutputFormat}} will
> call {{tryCleanupOnError}} before {{close}}. But the calling order is reverse
> when {{DataSinkTask}} is cancelled, which doesn't make much sense that we can
> still clean the output format after it's closed.
> I'm not very sure if this is a bug. But this does mislead our developers when
> implementing our customized {{OutputFormat}}. cc [~sewen]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16597) Handle exception thrown from close method in OutputFormat
[
https://issues.apache.org/jira/browse/FLINK-16597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-16597:
--
Affects Version/s: 1.11.0
1.12.0
> Handle exception thrown from close method in OutputFormat
> -
>
> Key: FLINK-16597
> URL: https://issues.apache.org/jira/browse/FLINK-16597
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.10.0, 1.11.0, 1.12.0
>Reporter: Jiayi Liao
>Assignee: Jiayi Liao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If {{DataSinkTask}} fails or throws an exception, the {{OutputFormat}} will
> call {{tryCleanupOnError}} before {{close}}. But the calling order is reverse
> when {{DataSinkTask}} is cancelled, which doesn't make much sense that we can
> still clean the output format after it's closed.
> I'm not very sure if this is a bug. But this does mislead our developers when
> implementing our customized {{OutputFormat}}. cc [~sewen]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on a change in pull request #11480: [FLINK-16597] handle exception thrown from close method in OutputFormat
rkhachatryan commented on a change in pull request #11480:
URL: https://github.com/apache/flink/pull/11480#discussion_r539647610
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/DataSinkTaskTest.java
##
@@ -276,6 +277,25 @@ public void testSortingDataSinkTask() {
}
}
+ @Test
+ public void testClosingErrorDataSinkTask() throws Exception {
Review comment:
Though the test does check that `close` was called, it doesn't test if
the original issues were resolved (order of `close` and `tryCleanupOnError` on
`cancel` and `invoke`).
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/DataSinkTask.java
##
@@ -295,7 +298,9 @@ public void cancel() throws Exception {
if (format != null) {
try {
this.format.close();
- } catch (Throwable t) {}
+ } catch (Throwable t) {
+ LOG.error("Error closing format.");
Review comment:
I think `tryCleanupOnError` should be moved here (like it's done in
`OutputFormatSinkFunction` for example).
(logging then need to be adjusted: if `tryCleanupOnError` no need to log
`close()` error, at least on ERR level).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
