[GitHub] [flink] WeiZhong94 commented on issue #9653: [FLINK-14014][python] Introduce PythonScalarFunctionRunner to handle the communication with Python worker for Python ScalarFunction execution
WeiZhong94 commented on issue #9653: [FLINK-14014][python] Introduce PythonScalarFunctionRunner to handle the communication with Python worker for Python ScalarFunction execution URL: https://github.com/apache/flink/pull/9653#issuecomment-532529973 @dianfu Thanks a lot for the update. LGTM and +1 to merge from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9653: [FLINK-14014][python] Introduce PythonScalarFunctionRunner to handle the communication with Python worker for Python ScalarFunction execution
flinkbot edited a comment on issue #9653: [FLINK-14014][python] Introduce PythonScalarFunctionRunner to handle the communication with Python worker for Python ScalarFunction execution URL: https://github.com/apache/flink/pull/9653#issuecomment-529525552 ## CI report: * 852555dbd31eb4e36009c99a22771a28544a0d5e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126471470) * 9b9156b15508efd40c5a8e5427a8cddb469913f9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126547856) * 48d9619d06297af85382bde44c5cff9688364459 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/127053703) * fde6b67e4060b486d32f6e63e5bd8458ac1e5ac5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127058420) * 6b4727221ed393d2d18a99ad220c87ee224093b4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127130673) * d367d3cbaa29d722741257e07603e9d33b03da28 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/127965086) * 3bac78e2cd1d22241dd82b5ec7c42cb66039412d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/12796) * a0876069203faaff034af1aa2fcad831d5fec343 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127975554) * 0475b1342c19c2831f42dad3e4f43162f72526ab : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932080#comment-16932080
]
lining commented on FLINK-13987:
Could you show your JIAR about it?
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log api returns all contents，it will block or not work when file is
> large.It's unfriendly for user.
> As application runs on jvm, sometime user need see log of gc, but there
> aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * see taskmanager log file by range
> ** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }
> {code}
> * see jobmanager log file by range
> ** /jobmanager/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931971#comment-16931971
]
lining edited comment on FLINK-13987 at 9/18/19 5:42 AM:
-
I have disscussed with till about that pr. It hasn't updated for a long time.
And see issue links, I have linked it.
was (Author: lining):
I hava disscussed with till about that pr. It hasn't updated for a long time.
And see issue links, I have linked it.
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log api returns all contents，it will block or not work when file is
> large.It's unfriendly for user.
> As application runs on jvm, sometime user need see log of gc, but there
> aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * see taskmanager log file by range
> ** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }
> {code}
> * see jobmanager log file by range
> ** /jobmanager/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931971#comment-16931971
]
lining edited comment on FLINK-13987 at 9/18/19 5:40 AM:
-
I hava disscussed with till about that pr. It hasn't updated for a long time.
And see issue links, I have linked it.
was (Author: lining):
I hava disscussed with till about that pr. It hasn‘t updated for a long time.
And see issue links, I have linked it.
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log api returns all contents，it will block or not work when file is
> large.It's unfriendly for user.
> As application runs on jvm, sometime user need see log of gc, but there
> aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * see taskmanager log file by range
> ** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }
> {code}
> * see jobmanager log file by range
> ** /jobmanager/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931971#comment-16931971
]
lining edited comment on FLINK-13987 at 9/18/19 5:39 AM:
-
I hava disscussed with till about that pr. It hasn‘t updated for a long time.
And see issue links, I have linked it.
was (Author: lining):
I hava disscussed with till about that pr. It has updated for a long time. And
see issue links, I have linked it.
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log api returns all contents，it will block or not work when file is
> large.It's unfriendly for user.
> As application runs on jvm, sometime user need see log of gc, but there
> aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * see taskmanager log file by range
> ** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }
> {code}
> * see jobmanager log file by range
> ** /jobmanager/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13656) Upgrade Calcite dependency to 1.21
[ https://issues.apache.org/jira/browse/FLINK-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932076#comment-16932076 ] Kurt Young commented on FLINK-13656: [~danny0405] Would you like to take this issue? > Upgrade Calcite dependency to 1.21 > -- > > Key: FLINK-13656 > URL: https://issues.apache.org/jira/browse/FLINK-13656 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Legacy Planner, Table SQL / Planner >Reporter: Jark Wu >Priority: Major > > Umbrella issue for all tasks related to the next Calcite upgrade to 1.21.x > release -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14092) Upgrade Calcite version to 1.21 for Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932074#comment-16932074 ] Kurt Young commented on FLINK-14092: I think we should stick to FLINK-13656, I will close this one. > Upgrade Calcite version to 1.21 for Flink SQL > - > > Key: FLINK-14092 > URL: https://issues.apache.org/jira/browse/FLINK-14092 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.9.0 >Reporter: Danny Chan >Priority: Major > Fix For: 1.10.0 > > > Calcite 1.21 has been released recently, we need to upgrade to version 1.21 > for these reasons: > # Previously we have made some temp code to support full data types in sql > parser, since CALCITE-3213 has been resolved, we can do some refactoring for > these codes; > # We also fixed some important bug for Join which bring in from Calcite 1.20 > join like expression promotion, such as CALCITE-3170, CALCITE-3171. > # CALCITE-2302 has been resolved, there is possibility we support implicit > type coercion for Flink now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14092) Upgrade Calcite version to 1.21 for Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14092. -- Resolution: Duplicate > Upgrade Calcite version to 1.21 for Flink SQL > - > > Key: FLINK-14092 > URL: https://issues.apache.org/jira/browse/FLINK-14092 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.9.0 >Reporter: Danny Chan >Priority: Major > Fix For: 1.10.0 > > > Calcite 1.21 has been released recently, we need to upgrade to version 1.21 > for these reasons: > # Previously we have made some temp code to support full data types in sql > parser, since CALCITE-3213 has been resolved, we can do some refactoring for > these codes; > # We also fixed some important bug for Join which bring in from Calcite 1.20 > join like expression promotion, such as CALCITE-3170, CALCITE-3171. > # CALCITE-2302 has been resolved, there is possibility we support implicit > type coercion for Flink now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14037) Deserializing the input/output formats failed: unread block data
[
https://issues.apache.org/jira/browse/FLINK-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932070#comment-16932070
]
Zhu Zhu edited comment on FLINK-14037 at 9/18/19 5:26 AM:
--
[~liupengcheng] I mocked this case by creating 2 jars. Jar1 contains class
x.y.z.A. Jar2 contains class x.y.z.A and its subclass x.y.z.B.
A Flink test case is added with classloader1 on Jar1 and its child classloader2
on Jar2. classloader2 can be a parent first classloader or a child first class
loader with "x.y.z." as alwaysParentFirstPatterns.
It turned out to be that the class A is always loaded from Jar1, no matter what
kind of class loader is used.
Therefore the
[result|https://issues.apache.org/jira/browse/FLINK-14037?focusedCommentId=16930345=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16930345]
with your fix does not seem to be expected, unless
"org.apache.hadoop.mapreduce.lib.input.FileInputFormat" does not exist in the
system classloader.
was (Author: zhuzh):
[~liupengcheng] I mocked this case by creating 2 jars. Jar1 contains class
x.y.z.A. Jar2 contains class x.y.z.A and its subclass z.y.z.B.
A Flink test case is added with classloader1 on Jar1 and its child classloader2
on Jar2. classloader2 can be a parent first classloader or a child first class
loader with "x.y.z." as alwaysParentFirstPatterns.
It turned out to be that the class A is always loaded from Jar1, no matter what
kind of class loader is used.
Therefore the
[result|https://issues.apache.org/jira/browse/FLINK-14037?focusedCommentId=16930345=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16930345]
with your fix does not seem to be expected, unless
"org.apache.hadoop.mapreduce.lib.input.FileInputFormat" does not exist in the
system classloader.
> Deserializing the input/output formats failed: unread block data
>
>
> Key: FLINK-14037
> URL: https://issues.apache.org/jira/browse/FLINK-14037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0
> Environment: flink 1.9.0
> app jar use `flink-shaded-hadoop-2` dependencies to avoid some confilicts
>
>Reporter: liupengcheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recently, we encountered the following issue when testing flink 1.9.0:
> {code:java}
> org.apache.flink.client.program.ProgramInvocationException: Could not
> retrieve the execution result. (JobID: 8ffbc071dda81d6f8005c02be8adde6b)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:255)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:539)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort$.main(FlinkTeraSort.scala:89)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort.main(FlinkTeraSort.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by:
[GitHub] [flink] dianfu commented on issue #9653: [FLINK-14014][python] Introduce PythonScalarFunctionRunner to handle the communication with Python worker for Python ScalarFunction execution
dianfu commented on issue #9653: [FLINK-14014][python] Introduce PythonScalarFunctionRunner to handle the communication with Python worker for Python ScalarFunction execution URL: https://github.com/apache/flink/pull/9653#issuecomment-532524299 @WeiZhong94 Thanks for the review. Make sense and have updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14037) Deserializing the input/output formats failed: unread block data
[
https://issues.apache.org/jira/browse/FLINK-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932070#comment-16932070
]
Zhu Zhu commented on FLINK-14037:
-
[~liupengcheng] I mocked this case by creating 2 jars. Jar1 contains class
x.y.z.A. Jar2 contains class x.y.z.A and its subclass z.y.z.B.
A Flink test case is added with classloader1 on Jar1 and its child classloader2
on Jar2. classloader2 can be a parent first classloader or a child first class
loader with "x.y.z." as alwaysParentFirstPatterns.
It turned out to be that the class A is always loaded from Jar1, no matter what
kind of class loader is used.
Therefore the
[result|https://issues.apache.org/jira/browse/FLINK-14037?focusedCommentId=16930345=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16930345]
with your fix does not seem to be expected, unless
"org.apache.hadoop.mapreduce.lib.input.FileInputFormat" does not exist in the
system classloader.
> Deserializing the input/output formats failed: unread block data
>
>
> Key: FLINK-14037
> URL: https://issues.apache.org/jira/browse/FLINK-14037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0
> Environment: flink 1.9.0
> app jar use `flink-shaded-hadoop-2` dependencies to avoid some confilicts
>
>Reporter: liupengcheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recently, we encountered the following issue when testing flink 1.9.0:
> {code:java}
> org.apache.flink.client.program.ProgramInvocationException: Could not
> retrieve the execution result. (JobID: 8ffbc071dda81d6f8005c02be8adde6b)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:255)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:539)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort$.main(FlinkTeraSort.scala:89)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort.main(FlinkTeraSort.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
> submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$8(RestClusterClient.java:382)
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:263)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
>
[GitHub] [flink] flinkbot edited a comment on issue #9689: [FLINK-7151] add a basic function ddl
flinkbot edited a comment on issue #9689: [FLINK-7151] add a basic function ddl URL: https://github.com/apache/flink/pull/9689#issuecomment-531747685 ## CI report: * bd2624914db1147588ea838ae542333c310290cc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127790175) * b5523d10152123f45cf883e446872b90532879c3 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128113059) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14093) Java8 lambdas and exceptions lead to compile error
[
https://issues.apache.org/jira/browse/FLINK-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932061#comment-16932061
]
TisonKun commented on FLINK-14093:
--
[~zhaoshijie] We cannot just introduce the workaround without any context.
Following contributor is likely to revert your commit regarding as improvement.
For whether or not to tolerate JDK bugs, I think you'd better start a
discussion thread on dev mailing list. We need to reach a consensus before
start any code working.
> Java8 lambdas and exceptions lead to compile error
> --
>
> Key: FLINK-14093
> URL: https://issues.apache.org/jira/browse/FLINK-14093
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.0
>Reporter: zzsmdfj
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> when compile flink master by using Java 1.8.0_77, got errors as follow:
> {code:java}
> // code placeholder
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:
> compile (default-compile) on project flink-table-api-java: Compilation failure
> /home/*/zzsmdfj/sflink/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/factories/Cal
> culatedTableFactory.java:[90,53] unreported exception X; must be caught or
> declared to be thrownat
> org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:213)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:146)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:117)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:81)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
> (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main
> (Launcher.java:356)
> Caused by: org.apache.maven.plugin.compiler.CompilationFailureException:
> Compilation failure
> {code}
> if using Java 1.8.0_102 to compile, it build success. it maybe a case of bug
> [JDK-8054569|https://bugs.openjdk.java.net/browse/JDK-8054569], although we
> can fix this by upgrading the JDK,but i think it is better of compatible with
> different versions of the JDK8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14111) Flink should be robust to a non-leader Zookeeper host going down
[
https://issues.apache.org/jira/browse/FLINK-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932059#comment-16932059
]
TisonKun commented on FLINK-14111:
--
Yes I think it is a duplication of FLINK-10052. Could you check it
[~aaronlevin]?
> Flink should be robust to a non-leader Zookeeper host going down
>
>
> Key: FLINK-14111
> URL: https://issues.apache.org/jira/browse/FLINK-14111
> Project: Flink
> Issue Type: Wish
> Components: Runtime / Coordination
>Affects Versions: 1.7.2, 1.8.0, 1.8.1, 1.9.0
> Environment: Linux
> JVM 8
> Flink {{1.7.2}}, {{1.8.1}}, {{1.9.0}}
> {{Zookeeper version 3.4.5}}
>Reporter: Aaron Levin
>Priority: Major
>
> I noticed that if a non-leader Zookeeper node goes down and there is still
> quorom in the zookeeper cluster , my flink application will restart anyway. I
> believe it should be possible for Flink applications not to require a restart
> in this scenario.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9689: [FLINK-7151] add a basic function ddl
flinkbot edited a comment on issue #9689: [FLINK-7151] add a basic function ddl URL: https://github.com/apache/flink/pull/9689#issuecomment-531747685 ## CI report: * bd2624914db1147588ea838ae542333c310290cc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127790175) * b5523d10152123f45cf883e446872b90532879c3 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-14112) Removing zookeeper state should cause the task manager and job managers to restart
[
https://issues.apache.org/jira/browse/FLINK-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932058#comment-16932058
]
TisonKun edited comment on FLINK-14112 at 9/18/19 4:56 AM:
---
Hi [~aaronlevin] thanks for creating this JIRA. Generally I think Flink owns
its znodes and the prerequisite here "delete all the znodes within {{/flink}}"
should not happen.
However, I can see your concern and ask you for the "massive amount of logging"
to see what we can improve in log scope.
was (Author: tison):
Hi [~aaronlevin] thanks for creating this JIRA. Generally I think Flink owns
its znodes and the prerequisite here "delete all the znodes within {{/flink}}"
should not happen.
However, I can see your concern and ask you for the "massive amount of logging"
to see what we can improve in log scope. Besides, I agree that JM and TM are
nice to crash if ZK is under an uncertain state.
> Removing zookeeper state should cause the task manager and job managers to
> restart
> --
>
> Key: FLINK-14112
> URL: https://issues.apache.org/jira/browse/FLINK-14112
> Project: Flink
> Issue Type: Wish
> Components: Runtime / Coordination
>Affects Versions: 1.8.1, 1.9.0
>Reporter: Aaron Levin
>Priority: Minor
>
> Suppose you have a flink application running on a cluster with the following
> configuration:
> {noformat}
> high-availability.zookeeper.path.root: /flink
> {noformat}
> Now suppose you delete all the znodes within {{/flink}}. I experienced the
> following:
> * massive amount of logging
> * application did not restart
> * task manager did not crash or restart
> * job manager did not crash or restart
> From this state I had to restart all the task managers and all the job
> managers in order for the flink application to recover.
> It would be desirable for the Task Managers and Job Managers to crash if the
> znode is not available (though perhaps you all have thought about this more
> deeply than I!)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14112) Removing zookeeper state should cause the task manager and job managers to restart
[
https://issues.apache.org/jira/browse/FLINK-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932058#comment-16932058
]
TisonKun commented on FLINK-14112:
--
Hi [~aaronlevin] thanks for creating this JIRA. Generally I think Flink owns
its znodes and the prerequisite here "delete all the znodes within {{/flink}}"
should not happen.
However, I can see your concern and ask you for the "massive amount of logging"
to see what we can improve in log scope. Besides, I agree that JM and TM are
nice to crash if ZK is under an uncertain state.
> Removing zookeeper state should cause the task manager and job managers to
> restart
> --
>
> Key: FLINK-14112
> URL: https://issues.apache.org/jira/browse/FLINK-14112
> Project: Flink
> Issue Type: Wish
> Components: Runtime / Coordination
>Affects Versions: 1.8.1, 1.9.0
>Reporter: Aaron Levin
>Priority: Minor
>
> Suppose you have a flink application running on a cluster with the following
> configuration:
> {noformat}
> high-availability.zookeeper.path.root: /flink
> {noformat}
> Now suppose you delete all the znodes within {{/flink}}. I experienced the
> following:
> * massive amount of logging
> * application did not restart
> * task manager did not crash or restart
> * job manager did not crash or restart
> From this state I had to restart all the task managers and all the job
> managers in order for the flink application to recover.
> It would be desirable for the Task Managers and Job Managers to crash if the
> znode is not available (though perhaps you all have thought about this more
> deeply than I!)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14109) Improve javadocs and tests for high-availability backend
[
https://issues.apache.org/jira/browse/FLINK-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932053#comment-16932053
]
TisonKun commented on FLINK-14109:
--
Hi [~bremac] thanks for reporting this!
There are two topics under this issue from my perspective.
1. Correct & enrich document.
2. Modify tests.
For 1, I it looks nice to have to me. For 2, maybe it's worth a separated
thread. So a suggestion is narrowing this issue to focus on 1 and starting a
separated issue focus on 2.
Besides, the community started a survey weeks ago on the topic "How do you use
high-availability services in Flink?"[1]. It is recommend to you sharing your
customization experience so that the ongoing refactor to high-availability
service[2] would take your use case into consideration :-)
[1]
https://lists.apache.org/x/thread.html/c0cc07197e6ba30b45d7709cc9e17d8497e5e3f33de504d58dfcafad@%3Cuser.flink.apache.org%3E
[2] https://issues.apache.org/jira/browse/FLINK-10333
> Improve javadocs and tests for high-availability backend
>
>
> Key: FLINK-14109
> URL: https://issues.apache.org/jira/browse/FLINK-14109
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Brendan MacDonell
>Priority: Minor
>
> My team at Sight Machine just finished building a custom HA backend for our
> infrastructure. The process was mostly painless, but we ran into a few pain
> points on the way:
> * {{CompletedCheckpointStore#getLatestCheckpoint}} is not marked as
> \{{@Nullable}}, so there was some confusion about whether the Javadocs or
> (lack of) annotation is correct. The interface would be clearer if the
> annotation was present.
> * The javadocs for {{CompletedCheckpointStore#recover}} disagree with the
> documentation for {{ZooKeeperCompletedCheckpointStore#recover}}. It's not
> immediately clear to someone working on the code that the ZK javadoc is
> outdated and the interface documentation is correct.
> * The base {{CompletedCheckpointStore}} tests only work with
> high-availability backends that keep a list of checkpoints in memory. If the
> backend persists and retrieves data from another source the tests will fail.
> It's fairly simple to change the tests to lift this requirement though. See
> [this gist|https://gist.github.com/bremac/1b3365bc0257dfbd33bcd0b7a7627c00]
> for an example.
> We've got patches for the points above that we'd be happy to contribute. :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932048#comment-16932048 ] Canbin Zheng commented on FLINK-14078: -- Thanks [~ykt836] ! > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor
flinkbot edited a comment on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor URL: https://github.com/apache/flink/pull/8546#issuecomment-532503099 ## CI report: * d3beb329bc438c726b0906ab8c97e02c084409b7 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128108187) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14092) Upgrade Calcite version to 1.21 for Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932043#comment-16932043 ] Danny Chan commented on FLINK-14092: Thanks for the reminder [~twalthr], there indeed is a issue FLINK-13656 about the Calcite 1.21 upgrade, can you help to bind the issue with this one ? > Upgrade Calcite version to 1.21 for Flink SQL > - > > Key: FLINK-14092 > URL: https://issues.apache.org/jira/browse/FLINK-14092 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.9.0 >Reporter: Danny Chan >Priority: Major > Fix For: 1.10.0 > > > Calcite 1.21 has been released recently, we need to upgrade to version 1.21 > for these reasons: > # Previously we have made some temp code to support full data types in sql > parser, since CALCITE-3213 has been resolved, we can do some refactoring for > these codes; > # We also fixed some important bug for Join which bring in from Calcite 1.20 > join like expression promotion, such as CALCITE-3170, CALCITE-3171. > # CALCITE-2302 has been resolved, there is possibility we support implicit > type coercion for Flink now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #9483: [FLINK-13767][task] Refactor StreamInputProcessor#processInput based on InputStatus
zhijiangW commented on a change in pull request #9483: [FLINK-13767][task]
Refactor StreamInputProcessor#processInput based on InputStatus
URL: https://github.com/apache/flink/pull/9483#discussion_r325477987
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessor.java
##
@@ -184,10 +184,6 @@ public boolean processInput() throws Exception {
}
checkFinished(status, lastReadInputIndex);
- if (status != InputStatus.MORE_AVAILABLE) {
Review comment:
Let me summary the changes a bit. There are mainly three changes:
1. Migrate `inputSelectionHandler.setUnavailableInput` to integrate with
`selectNextReadingInputIndex#updateAvailability`. I think it has no behavior
different with before. Just to put similar logics together for better tracing.
2. Return status instead of boolean for `StreamInputProcessor#processInput`.
It is for replacing the `isFinished` method and keeping the consistent form
with `emitNext` return type. It also has no behavior change, and only
`StreamTask#processInput` adds two additional conditions check instead of
previous `while` way. But we can also use `while` way or add the hot path to
avoid performance concern.
```
if (status == InputStatus.MORE_AVAILABLE) {
return;
}
```
3. The value returned by `StreamTwoInputProcessor#processInput` is different
from before which might cause different behaviors. In the past it might return
invalid status after current call, and then in the next call it might return -1
while `selectNextReadingInputIndex()` to trigger `isAvailable()` access. But
now it would always return valid status by judging the next selected index with
its status. So if it is not available for next selected input, it would trigger
`isAvailable()` access in current call directly, no need to trigger it in next
call. From this point, my previous thought was that it might save some
unnecessary cost from mailbox schedule aspect.
Anyway, it is hard to verify the above points because of unstable benchmark.
So we can only analyze the possible impacts in theory.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Issue Comment Deleted] (FLINK-13708) transformations should be cleared because a table environment could execute multiple job
[ https://issues.apache.org/jira/browse/FLINK-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated FLINK-13708: --- Comment: was deleted (was: Do you have sample code to reproduce this issue ? Because I don't see this issue if I run multiple sql jobs. ) > transformations should be cleared because a table environment could execute > multiple job > > > Key: FLINK-13708 > URL: https://issues.apache.org/jira/browse/FLINK-13708 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 10m > Remaining Estimate: 0h > > currently, if a table environment execute more than one sql jobs, the > following job contains transformations about the previous job. the reason is > the transformations is not cleared after execution -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor
flinkbot edited a comment on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor URL: https://github.com/apache/flink/pull/8546#issuecomment-532503099 ## CI report: * d3beb329bc438c726b0906ab8c97e02c084409b7 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128108187) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW edited a comment on issue #9646: [FLINK-14004][task] Define SourceReader interface to verify the integration with StreamOneInputProcessor
zhijiangW edited a comment on issue #9646: [FLINK-14004][task] Define SourceReader interface to verify the integration with StreamOneInputProcessor URL: https://github.com/apache/flink/pull/9646#issuecomment-532503656 Sorry for missing the following comments. I agree with the general above points. On runtime side it does not care about the specific `SourceReader` interface. The runtime only interacts with source operator via `#emitNext` and `#available` methods atm. As for how the source operator further interacts with the internal source reader, the runtime does not need to understand it. We can ignore the `SourceReader` related details in this PR. The possible meaningful point in this PR is that I try to pass through whether the runtime proposed `DataOutput` argument via `operator#emitNext` can satisfy well for the future `SourceReader`. So I construct a unit test for verifying it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on issue #9646: [FLINK-14004][task] Define SourceReader interface to verify the integration with StreamOneInputProcessor
zhijiangW commented on issue #9646: [FLINK-14004][task] Define SourceReader interface to verify the integration with StreamOneInputProcessor URL: https://github.com/apache/flink/pull/9646#issuecomment-532503656 Sorry for missing the following comments. I agree with the general above points. On runtime side it does not care about the specific `SourceReader` interface. The runtime only interacts with source operator via `#emitNext` and `#available` methods atm. As for how the source operator further interacts with the internal source reader, the runtime does not need to understand it. We can ignore the SourceReader related details in this PR. The possible meaningful point in this PR is that I try to pass through whether the runtime proposed `DataOutput` argument via `operator#emitNext` can satisfy well for the future `SourceReader`. So I construct a unit test for verifying it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14037) Deserializing the input/output formats failed: unread block data
[
https://issues.apache.org/jira/browse/FLINK-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932012#comment-16932012
]
liupengcheng commented on FLINK-14037:
--
[~zhuzh] Can you share your test code? Do you include `FileInputFormat` in the
user jar?
> Deserializing the input/output formats failed: unread block data
>
>
> Key: FLINK-14037
> URL: https://issues.apache.org/jira/browse/FLINK-14037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0
> Environment: flink 1.9.0
> app jar use `flink-shaded-hadoop-2` dependencies to avoid some confilicts
>
>Reporter: liupengcheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recently, we encountered the following issue when testing flink 1.9.0:
> {code:java}
> org.apache.flink.client.program.ProgramInvocationException: Could not
> retrieve the execution result. (JobID: 8ffbc071dda81d6f8005c02be8adde6b)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:255)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:539)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort$.main(FlinkTeraSort.scala:89)
> at
> com.github.ehiggs.spark.terasort.FlinkTeraSort.main(FlinkTeraSort.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
> submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$8(RestClusterClient.java:382)
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:263)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
> at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.runtime.rest.util.RestClientException:
[jira] [Comment Edited] (FLINK-14099) SQL supports timestamp in Long
[
https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932011#comment-16932011
]
Zijie Lu edited comment on FLINK-14099 at 9/18/19 3:30 AM:
---
Maybe we can add a parameter in the function timestampFromField with its
timestamp format and convert it to the stand SQL timestamp.
The code looks like this:
*new Schema()*
*.field("rowtime", Types.SQL_TIMESTAMP)*
*.rowtime(new Rowtime().timestampsFromField("timestamp",
"timestampFormat").watermarksPeriodicAscending())*
Do you think this idea work?
*[~felixzheng] [~twalthr]*
was (Author: alfredlu):
Maybe we can add a parameter in the function timestampFromField with its
timestamp format and convert it to the stand SQL timestamp.
The code looks like this:
*new Schema()*
*.field("rowtime", Types.SQL_TIMESTAMP)*
*.rowtime(new Rowtime().timestampsFromField("timestamp",
"timestampFormat").watermarksPeriodicAscending())*
**Do you think this idea work?
*[~felixzheng] [~twalthr]*
> SQL supports timestamp in Long
> --
>
> Key: FLINK-14099
> URL: https://issues.apache.org/jira/browse/FLINK-14099
> Project: Flink
> Issue Type: New Feature
>Reporter: Zijie Lu
>Priority: Major
>
> The rowtime only supports sql timestamp but not long. Can the rowtime field
> supports long?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14099) SQL supports timestamp in Long
[
https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932011#comment-16932011
]
Zijie Lu edited comment on FLINK-14099 at 9/18/19 3:30 AM:
---
Maybe we can add a parameter in the function timestampFromField with its
timestamp format and convert it to the stand SQL timestamp.
The code looks like this:
*new Schema()*
*.field("rowtime", Types.SQL_TIMESTAMP)*
*.rowtime(new Rowtime().timestampsFromField("timestamp",
"timestampFormat").watermarksPeriodicAscending())*
**Do you think this idea work?
*[~felixzheng] [~twalthr]*
was (Author: alfredlu):
Maybe we can add a parameter in the function timestampFromField with its
timestamp format and convert it to the stand SQL timestamp.
The code looks like this:
*new Schema()*
*.field("rowtime", Types.SQL_TIMESTAMP)*
*.rowtime(new Rowtime().timestampsFromField("timestamp",
"timestampFormat").watermarksPeriodicAscending())*
> SQL supports timestamp in Long
> --
>
> Key: FLINK-14099
> URL: https://issues.apache.org/jira/browse/FLINK-14099
> Project: Flink
> Issue Type: New Feature
>Reporter: Zijie Lu
>Priority: Major
>
> The rowtime only supports sql timestamp but not long. Can the rowtime field
> supports long?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14099) SQL supports timestamp in Long
[
https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932011#comment-16932011
]
Zijie Lu commented on FLINK-14099:
--
Maybe we can add a parameter in the function timestampFromField with its
timestamp format and convert it to the stand SQL timestamp.
The code looks like this:
*new Schema()*
*.field("rowtime", Types.SQL_TIMESTAMP)*
*.rowtime(new Rowtime().timestampsFromField("timestamp",
"timestampFormat").watermarksPeriodicAscending())*
> SQL supports timestamp in Long
> --
>
> Key: FLINK-14099
> URL: https://issues.apache.org/jira/browse/FLINK-14099
> Project: Flink
> Issue Type: New Feature
>Reporter: Zijie Lu
>Priority: Major
>
> The rowtime only supports sql timestamp but not long. Can the rowtime field
> supports long?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor
flinkbot commented on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor URL: https://github.com/apache/flink/pull/8546#issuecomment-532503099 ## CI report: * d3beb329bc438c726b0906ab8c97e02c084409b7 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk edited a comment on issue #9688: [FLINK-13056] [runtime] Introduce FastRestartPipelinedRegionStrategy which has better failover handling performance at the cost of slower reg
zhuzhurk edited a comment on issue #9688: [FLINK-13056] [runtime] Introduce
FastRestartPipelinedRegionStrategy which has better failover handling
performance at the cost of slower region building and more memory used
URL: https://github.com/apache/flink/pull/9688#issuecomment-532262379
> How do you configure the `FastRestartPipelinedRegionStrategy`?
One option is a boolean config
"jobmanager.execution.failover-strategy.region.fast-mode". This reveals to
users that we have a fast mode for region failover.
Another option is a new supported value "region-fast" for the failover
strategy specification config ("jobmanager.execution.failover-strategy"). This
reveals to users a new region failover strategy.
I prefer the first option which hides the implementation and makes it easy
if later we'd like to keep only one region failover strategy.
What's your opinion?
I planned to add the config when using it for the legacy failover strategy
creation. But as you mentioned we can also add it in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor
TisonKun commented on issue #8546: [FLINK-12621]. Use MiniCluster instead of JobExecutor URL: https://github.com/apache/flink/pull/8546#issuecomment-532500733 ref #9643 and [FLINK-13961](https://issues.apache.org/jira/browse/FLINK-13961). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14099) SQL supports timestamp in Long
[ https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932007#comment-16932007 ] Zijie Lu commented on FLINK-14099: -- I agree with [~felixzheng], So does the rowtime field only support the stand SQL Timestamp now? > SQL supports timestamp in Long > -- > > Key: FLINK-14099 > URL: https://issues.apache.org/jira/browse/FLINK-14099 > Project: Flink > Issue Type: New Feature >Reporter: Zijie Lu >Priority: Major > > The rowtime only supports sql timestamp but not long. Can the rowtime field > supports long? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932004#comment-16932004
]
vinoyang commented on FLINK-13987:
--
[~lining] Right, IMO, the existed implementation is not meaningless. It would
be better reuse it(line range) and some exist implementation(for example, our
log file list feature) to promote this big feature fastly. WDYT?
[~till.rohrmann]
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log api returns all contents，it will block or not work when file is
> large.It's unfriendly for user.
> As application runs on jvm, sometime user need see log of gc, but there
> aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * see taskmanager log file by range
> ** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }
> {code}
> * see jobmanager log file by range
> ** /jobmanager/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9025: [FLINK-13149][Web Frontend] support configuration and update Angular to 8.0 version
flinkbot edited a comment on issue #9025: [FLINK-13149][Web Frontend] support configuration and update Angular to 8.0 version URL: https://github.com/apache/flink/pull/9025#issuecomment-512173118 ## CI report: * e2ae53781367b527fd4d77a4a63ae3b8f354a810 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119439998) * 6f2b4c282a8d3f88b58eb6e3946dca5fc6ed9752 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/127981266) * a5367c6eff2e56223de725b5fe8b74afd1a0301c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128106324) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13746) Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932003#comment-16932003
]
Zijie Lu commented on FLINK-13746:
--
Is [Terror] logged in the $FLINK_DIR/log? In the travis log, *Test exited with
exit code 0 but the logs contained errors, exceptions or non-empty .out files.*
And I don't found any function in the common.sh which is used to count
[Terror]. So where is [Terror] counted?
> Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
> ---
>
> Key: FLINK-13746
> URL: https://issues.apache.org/jira/browse/FLINK-13746
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch, Tests
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0, 1.9.1
>
>
> The {{Elasticsearch (v2.3.5) sink end-to-end test}} fails on Travis because
> it logs contain the following line:
> {code}
> INFO org.elasticsearch.plugins - [Terror] modules [], plugins [], sites []
> {code}
> Due to this, the error check is triggered.
> https://api.travis-ci.org/v3/job/572255901/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13149) improve flink ui to make it easier to be used by other projects
[
https://issues.apache.org/jira/browse/FLINK-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie TANG updated FLINK-13149:
-
Description:
The new web UI looks nice, but there are still some problems when I try to
integrate it into the other frontend project, I think we can make some changes
in order to make it easier to be customized.
!20190709113558.jpg|width=596,height=185!
*These changes will not bring break changes and it will also not affect the
user interface.*
* Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it
configurable in angular DI system.
* Add property named {{JOB_PREFIX}} to {{config.service.ts}} for
{{job.service.ts}} and {{metrics.service.ts}} to make them configurable
* Add optional param to the url to hide menu and header to make it possible
for users want to embed the flink ui as an iframe in other website
* Update Angular version to 8.0 (no break changes)
* set `paths` of `tsconfig.json` ("@flink-runtime-web/*": ["./src/app/*",
"./src/environments/*"]) to ensure that it does not have the same name as the
personal project path.
was:
The new web UI looks nice, but there are still some problems when I try to
integrate it into the other frontend project, I think we can make some changes
in order to make it easier to be customized.
!20190709113558.jpg|width=596,height=185!
*These changes will not bring break changes and it will also not affect the
user interface.*
* Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it
configurable in angular DI system.
* Add property named {{JOB_PREFIX}} to {{config.service.ts}} for
{{job.service.ts}} and {{metrics.service.ts}} to make them configurable
* Add optional param to the url to hide menu and header to make it possible
for users want to embed the flink ui as an iframe in other website
* Update Angular version to 8.0 (no break changes)
* set `paths` of `tsconfig.json`
```
"paths": {
"@flink-runtime-web/*": ["./src/app/*", "./src/environments/*"]
},
```
> improve flink ui to make it easier to be used by other projects
> ---
>
> Key: FLINK-13149
> URL: https://issues.apache.org/jira/browse/FLINK-13149
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Affects Versions: 1.9.0
>Reporter: Jie TANG
>Priority: Major
> Labels: performance, pull-request-available
> Fix For: 1.10.0
>
> Attachments: 20190709113558.jpg, snapshot-1.png, snapshot-2.png
>
> Original Estimate: 120h
> Time Spent: 10m
> Remaining Estimate: 119h 50m
>
> The new web UI looks nice, but there are still some problems when I try to
> integrate it into the other frontend project, I think we can make some
> changes in order to make it easier to be customized.
> !20190709113558.jpg|width=596,height=185!
> *These changes will not bring break changes and it will also not affect the
> user interface.*
> * Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it
> configurable in angular DI system.
> * Add property named {{JOB_PREFIX}} to {{config.service.ts}} for
> {{job.service.ts}} and {{metrics.service.ts}} to make them configurable
> * Add optional param to the url to hide menu and header to make it possible
> for users want to embed the flink ui as an iframe in other website
> * Update Angular version to 8.0 (no break changes)
> * set `paths` of `tsconfig.json` ("@flink-runtime-web/*": ["./src/app/*",
> "./src/environments/*"]) to ensure that it does not have the same name as the
> personal project path.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14099) SQL supports timestamp in Long
[ https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931999#comment-16931999 ] Canbin Zheng edited comment on FLINK-14099 at 9/18/19 3:09 AM: --- I think computed column is a more general and better way. Maybe in future we can provide some generic and easy-use built-in functions for converting different rowtime field type into standard SQL Timestamp. In production environment, rowtime field from source usually has many different modes, cases like 1. second(String): "1568774938" 2. second(Long): 1568774938 3. millisecond(String): "1568774938143" 4. millisecond(Long): 1568774938143 5. custom date format: MMddHHmmss: "20190918112311" -MM-dd HH:mm:ss "2019-09-18 11:23:11" -MM-dd HH:mm:ss.SSS "2019-09-18 11:23:11.131" was (Author: felixzheng): I think Computed column is a more general and better way. Maybe in future we can provide some generic and easy-use built-in functions for converting different rowtime field type into standard SQL Timestamp. In production environment, rowtime field from source usually has many different modes, cases like 1. second(String): "1568774938" 2. second(Long): 1568774938 3. millisecond(String): "1568774938143" 4. millisecond(Long): 1568774938143 5. custom date format: MMddHHmmss: "20190918112311" -MM-dd HH:mm:ss "2019-09-18 11:23:11" -MM-dd HH:mm:ss.SSS "2019-09-18 11:23:11.131" > SQL supports timestamp in Long > -- > > Key: FLINK-14099 > URL: https://issues.apache.org/jira/browse/FLINK-14099 > Project: Flink > Issue Type: New Feature >Reporter: Zijie Lu >Priority: Major > > The rowtime only supports sql timestamp but not long. Can the rowtime field > supports long? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14099) SQL supports timestamp in Long
[ https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931999#comment-16931999 ] Canbin Zheng edited comment on FLINK-14099 at 9/18/19 3:08 AM: --- I think Computed column is a more general and better way. Maybe in future we can provide some generic and easy-use built-in functions for converting different rowtime field type into standard SQL Timestamp. In production environment, rowtime field from source usually has many different modes, cases like 1. second(String): "1568774938" 2. second(Long): 1568774938 3. millisecond(String): "1568774938143" 4. millisecond(Long): 1568774938143 5. custom date format: MMddHHmmss: "20190918112311" -MM-dd HH:mm:ss "2019-09-18 11:23:11" -MM-dd HH:mm:ss.SSS "2019-09-18 11:23:11.131" was (Author: felixzheng): +1. I think Computed column is a more general and better way. Maybe in future we can provide some generic and easy-use built-in functions for converting different rowtime field type into standard SQL Timestamp. In production environment, rowtime field from source usually has many different modes, cases like 1. second(String): "1568774938" 2. second(Long): 1568774938 3. millisecond(String): "1568774938143" 4. millisecond(Long): 1568774938143 5. custom date format: MMddHHmmss: "20190918112311" -MM-dd HH:mm:ss "2019-09-18 11:23:11" -MM-dd HH:mm:ss.SSS "2019-09-18 11:23:11.131" > SQL supports timestamp in Long > -- > > Key: FLINK-14099 > URL: https://issues.apache.org/jira/browse/FLINK-14099 > Project: Flink > Issue Type: New Feature >Reporter: Zijie Lu >Priority: Major > > The rowtime only supports sql timestamp but not long. Can the rowtime field > supports long? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13149) improve flink ui to make it easier to be used by other projects
[
https://issues.apache.org/jira/browse/FLINK-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie TANG updated FLINK-13149:
-
Description:
The new web UI looks nice, but there are still some problems when I try to
integrate it into the other frontend project, I think we can make some changes
in order to make it easier to be customized.
!20190709113558.jpg|width=596,height=185!
*These changes will not bring break changes and it will also not affect the
user interface.*
* Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it
configurable in angular DI system.
* Add property named {{JOB_PREFIX}} to {{config.service.ts}} for
{{job.service.ts}} and {{metrics.service.ts}} to make them configurable
* Add optional param to the url to hide menu and header to make it possible
for users want to embed the flink ui as an iframe in other website
* Update Angular version to 8.0 (no break changes)
* set `paths` of `tsconfig.json`
```
"paths": {
"@flink-runtime-web/*": ["./src/app/*", "./src/environments/*"]
},
```
was:
The new web UI looks nice, but there are still some problems when I try to
integrate it into the other frontend project, I think we can make some changes
in order to make it easier to be customized.
!20190709113558.jpg|width=596,height=185!
*These changes will not bring break changes and it will also not affect the
user interface.*
* Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it
configurable in angular DI system.
* Add property named {{JOB_PREFIX}} to {{config.service.ts}} for
{{job.service.ts}} and {{metrics.service.ts}} to make them configurable
* Add optional param to the url to hide menu and header to make it possible
for users want to embed the flink ui as an iframe in other website
* Update Angular version to 8.0 (no break changes)
> improve flink ui to make it easier to be used by other projects
> ---
>
> Key: FLINK-13149
> URL: https://issues.apache.org/jira/browse/FLINK-13149
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Affects Versions: 1.9.0
>Reporter: Jie TANG
>Priority: Major
> Labels: performance, pull-request-available
> Fix For: 1.10.0
>
> Attachments: 20190709113558.jpg, snapshot-1.png, snapshot-2.png
>
> Original Estimate: 120h
> Time Spent: 10m
> Remaining Estimate: 119h 50m
>
> The new web UI looks nice, but there are still some problems when I try to
> integrate it into the other frontend project, I think we can make some
> changes in order to make it easier to be customized.
> !20190709113558.jpg|width=596,height=185!
> *These changes will not bring break changes and it will also not affect the
> user interface.*
> * Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it
> configurable in angular DI system.
> * Add property named {{JOB_PREFIX}} to {{config.service.ts}} for
> {{job.service.ts}} and {{metrics.service.ts}} to make them configurable
> * Add optional param to the url to hide menu and header to make it possible
> for users want to embed the flink ui as an iframe in other website
> * Update Angular version to 8.0 (no break changes)
> * set `paths` of `tsconfig.json`
> ```
> "paths": {
> "@flink-runtime-web/*": ["./src/app/*", "./src/environments/*"]
> },
> ```
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14099) SQL supports timestamp in Long
[ https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931999#comment-16931999 ] Canbin Zheng commented on FLINK-14099: -- +1. I think Computed column is a more general and better way. Maybe in future we can provide some generic and easy-use built-in functions for converting different rowtime field type into standard SQL Timestamp. In production environment, rowtime field from source usually has many different modes, cases like 1. second(String): "1568774938" 2. second(Long): 1568774938 3. millisecond(String): "1568774938143" 4. millisecond(Long): 1568774938143 5. custom date format: MMddHHmmss: "20190918112311" -MM-dd HH:mm:ss "2019-09-18 11:23:11" -MM-dd HH:mm:ss.SSS "2019-09-18 11:23:11.131" > SQL supports timestamp in Long > -- > > Key: FLINK-14099 > URL: https://issues.apache.org/jira/browse/FLINK-14099 > Project: Flink > Issue Type: New Feature >Reporter: Zijie Lu >Priority: Major > > The rowtime only supports sql timestamp but not long. Can the rowtime field > supports long? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14100) Introduce OracleDialect
[ https://issues.apache.org/jira/browse/FLINK-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-14100: -- Assignee: Canbin Zheng > Introduce OracleDialect > --- > > Key: FLINK-14100 > URL: https://issues.apache.org/jira/browse/FLINK-14100 > Project: Flink > Issue Type: Sub-task >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14102) Introduce DB2Dialect
[ https://issues.apache.org/jira/browse/FLINK-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-14102: -- Assignee: Canbin Zheng > Introduce DB2Dialect > > > Key: FLINK-14102 > URL: https://issues.apache.org/jira/browse/FLINK-14102 > Project: Flink > Issue Type: Sub-task >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931993#comment-16931993 ] Kurt Young commented on FLINK-14078: Thanks [~felixzheng], I will assign these jiras to you. > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-14078: -- Assignee: Canbin Zheng > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14101) Introduce SqlServerDialect
[ https://issues.apache.org/jira/browse/FLINK-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-14101: -- Assignee: Canbin Zheng > Introduce SqlServerDialect > -- > > Key: FLINK-14101 > URL: https://issues.apache.org/jira/browse/FLINK-14101 > Project: Flink > Issue Type: Sub-task >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9025: [FLINK-13149][Web Frontend] support configuration and update Angular to 8.0 version
flinkbot edited a comment on issue #9025: [FLINK-13149][Web Frontend] support configuration and update Angular to 8.0 version URL: https://github.com/apache/flink/pull/9025#issuecomment-512173118 ## CI report: * e2ae53781367b527fd4d77a4a63ae3b8f354a810 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119439998) * 6f2b4c282a8d3f88b58eb6e3946dca5fc6ed9752 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/127981266) * a5367c6eff2e56223de725b5fe8b74afd1a0301c : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #9653: [FLINK-14014][python] Introduce PythonScalarFunctionRunner to handle the communication with Python worker for Python ScalarFunc
WeiZhong94 commented on a change in pull request #9653: [FLINK-14014][python]
Introduce PythonScalarFunctionRunner to handle the communication with Python
worker for Python ScalarFunction execution
URL: https://github.com/apache/flink/pull/9653#discussion_r325458729
##
File path:
flink-python/src/main/java/org/apache/flink/python/AbstractPythonFunctionRunner.java
##
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.python;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.core.memory.ByteArrayInputStreamWithPos;
+import org.apache.flink.core.memory.ByteArrayOutputStreamWithPos;
+import org.apache.flink.table.functions.python.PythonEnv;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.Environments;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
+import org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.nio.file.Files;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * An base class for {@link PythonFunctionRunner}.
+ *
+ * @param Type of the input elements.
+ * @param Type of the execution results.
+ */
+@Internal
+public abstract class AbstractPythonFunctionRunner implements
PythonFunctionRunner {
+
+ private static final String MAIN_INPUT_ID = "input";
+
+ private final String taskName;
+
+ /**
+* The Python function execution result receiver.
+*/
+ private final FnDataReceiver resultReceiver;
+
+ /**
+* The Python execution environment.
+*/
+ private final PythonEnv pythonEnv;
+
+ /**
+* The bundle factory which has all job-scoped information and can be
used to create a {@link StageBundleFactory}.
+*/
+ private transient JobBundleFactory jobBundleFactory;
+
+ /**
+* The bundle factory which has all of the resources it needs to
provide new {@link RemoteBundle}.
+*/
+ private transient StageBundleFactory stageBundleFactory;
+
+ /**
+* Handler for state requests.
+*/
+ private final StateRequestHandler stateRequestHandler;
+
+ /**
+* Handler for bundle progress messages, both during bundle execution
and on its completion.
+*/
+ private transient BundleProgressHandler progressHandler;
+
+ /**
+* A bundle handler for handling input elements by forwarding them to a
remote environment for processing.
+* It holds a collection of {@link FnDataReceiver}s which actually
perform the data forwarding work.
+*
+* When a RemoteBundle is closed, it will block until bundle
processing is finished on remote resources,
+* and throw an exception if bundle processing has failed.
+*/
+ private transient RemoteBundle remoteBundle;
+
+ /**
+* The receiver which forwards the input elements to a remote
environment for processing.
+*/
+ private transient FnDataReceiver> mainInputReceiver;
+
+ /**
+* The coder for input elements.
+*/
+
[jira] [Commented] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931980#comment-16931980 ] Canbin Zheng commented on FLINK-14078: -- Thanks for the quick response [~ykt836] ! I will add plenty of tests when start to work on this issue. I plan to make some progress for OracleDialect first. > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14099) SQL supports timestamp in Long
[
https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931977#comment-16931977
]
Zijie Lu commented on FLINK-14099:
--
I use a kafka connector and describe a schema as rowtime which is from field
"timestamp". And the timestamp in the kafka message is unix timestamp in second
which means it is a Long.
And the code is as following:
*new Schema()*
*.field("rowtime", Types.SQL_TIMESTAMP)*
*.rowtime(new
Rowtime().timestampsFromField("timestamp").watermarksPeriodicAscending())*
*.field("requestId", Types.STRING)*
And the following expections are throwed.
*Caused by: java.time.format.DateTimeParseException: Text '1568772929' could
not be parsed at index 0*
*at
java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)*
*at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1777)*
*at
org.apache.flink.formats.json.JsonRowDeserializationSchema.lambda$createTimestampConverter$1dee6515$1(JsonRowDeserializationSchema.java:334)*
*at
org.apache.flink.formats.json.JsonRowDeserializationSchema.lambda$wrapIntoNullableConverter$d586c97$1(JsonRowDeserializationSchema.java:232)*
*at
org.apache.flink.formats.json.JsonRowDeserializationSchema.convertField(JsonRowDeserializationSchema.java:403)*
*at
org.apache.flink.formats.json.JsonRowDeserializationSchema.lambda$assembleRowConverter$77f7700$1(JsonRowDeserializationSchema.java:382)*
*at
org.apache.flink.formats.json.JsonRowDeserializationSchema.lambda$wrapIntoNullableConverter$d586c97$1(JsonRowDeserializationSchema.java:232)*
*at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:127)*
*... 7 more*
> SQL supports timestamp in Long
> --
>
> Key: FLINK-14099
> URL: https://issues.apache.org/jira/browse/FLINK-14099
> Project: Flink
> Issue Type: New Feature
>Reporter: Zijie Lu
>Priority: Major
>
> The rowtime only supports sql timestamp but not long. Can the rowtime field
> supports long?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jinglining edited a comment on issue #9699: [FLINK-13949][rest]Delete deduplicating JobVertexDetailsInfo.VertexTa…
jinglining edited a comment on issue #9699: [FLINK-13949][rest]Delete deduplicating JobVertexDetailsInfo.VertexTa… URL: https://github.com/apache/flink/pull/9699#issuecomment-532483390 > Thanks for this PR @jinglining. LGTM. Have you tried whether the web UI is still displaying all subtask execution attempt details? I have built it. And this is the screenshots  It's ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931971#comment-16931971
]
lining commented on FLINK-13987:
I hava disscussed with till about that pr. It has updated for a long time. And
see issue links, I have linked it.
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log api returns all contents，it will block or not work when file is
> large.It's unfriendly for user.
> As application runs on jvm, sometime user need see log of gc, but there
> aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * see taskmanager log file by range
> ** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }
> {code}
> * see jobmanager log file by range
> ** /jobmanager/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-13856) Reduce the delete file api when the checkpoint is completed
[ https://issues.apache.org/jira/browse/FLINK-13856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] apus66 updated FLINK-13856: --- Comment: was deleted (was: Hi [~StephanEwen] Can you help me review the code and assign this task to me? We have already verified in the production environment, the effect is very obvious！ The general idea is the following: * In full checkpoint, to delete the snapshot(chk-) folder directly * In incremental checkpoint, find IncrementalRemoteKeyedStateHandle and delete the shared state first, and then dropping the exclusive location as a whole. [reduce delete api | [https://github.com/apache/flink/pull/9602]]) > Reduce the delete file api when the checkpoint is completed > --- > > Key: FLINK-13856 > URL: https://issues.apache.org/jira/browse/FLINK-13856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.8.1, 1.9.0 >Reporter: andrew.D.lin >Priority: Major > Labels: pull-request-available > Attachments: after.png, before.png, > f6cc56b7-2c74-4f4b-bb6a-476d28a22096.png > > Original Estimate: 48h > Time Spent: 10m > Remaining Estimate: 47h 50m > > When the new checkpoint is completed, an old checkpoint will be deleted by > calling CompletedCheckpoint.discardOnSubsume(). > When deleting old checkpoints, follow these steps: > 1, drop the metadata > 2, discard private state objects > 3, discard location as a whole > In some cases, is it possible to delete the checkpoint folder recursively by > one call? > As far as I know the full amount of checkpoint, it should be possible to > delete the folder directly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] KurtYoung commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate
KurtYoung commented on a change in pull request #8468:
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r325453512
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/plan/nodes/logical/FlinkLogicalTableSourceScan.scala
##
@@ -76,9 +76,25 @@ class FlinkLogicalTableSourceScan(
val terms = super.explainTerms(pw)
.item("fields", tableSource.getTableSchema.getFieldNames.mkString(",
"))
+val auxiliarySourceDesc = tableSource match {
+ case fts: FilterableTableSource[_] =>
+s"FilterPushDown=${fts.isFilterPushedDown.toString}"
Review comment:
I'm not sure if we can do this for users.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-14096) Merge NewClusterClient into ClusterClient
[
https://issues.apache.org/jira/browse/FLINK-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931962#comment-16931962
]
TisonKun commented on FLINK-14096:
--
FYI
https://lists.apache.org/x/thread.html/e6536ad385ae1fe41006309dfb3e808c12195aeb8d425c52c86d4ce6@%3Cdev.flink.apache.org%3E
> Merge NewClusterClient into ClusterClient
> -
>
> Key: FLINK-14096
> URL: https://issues.apache.org/jira/browse/FLINK-14096
> Project: Flink
> Issue Type: Sub-task
> Components: Client / Job Submission
>Reporter: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With the effort under FLINK-10392 we don't need the bridge class
> {{NewClusterClient}} any more. We can just merge {{NewClusterClient}} into
> {{ClusterClient}} towards an interface-ized {{ClusterClient}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jinglining commented on issue #9699: [FLINK-13949][rest]Delete deduplicating JobVertexDetailsInfo.VertexTa…
jinglining commented on issue #9699: [FLINK-13949][rest]Delete deduplicating JobVertexDetailsInfo.VertexTa… URL: https://github.com/apache/flink/pull/9699#issuecomment-532483390 > Thanks for this PR @jinglining. LGTM. Have you tried whether the web UI is still displaying all subtask execution attempt details? I have built it. And this is the screenshots  It's ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931960#comment-16931960 ] Kurt Young commented on FLINK-14078: I think split into some subtasks is a good idea. Even if there is might be a few dozen lines of code, I believe we need add plenty of tests for each dialect. > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931959#comment-16931959 ] Canbin Zheng commented on FLINK-14078: -- [~ykt836] [~lzljs3620320] Yes, so there's no need to split them into subtasks, right? > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931955#comment-16931955 ] Jingsong Lee commented on FLINK-14078: -- > Just want to clarify, does this only take effect to JDBC connector? I think yes, Like Mysql Dialect, each Dialect is estimated to have only a few dozen lines of code. Right? [~felixzheng] > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14099) SQL supports timestamp in Long
[ https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931954#comment-16931954 ] Jingsong Lee commented on FLINK-14099: -- The problem I've encountered is that I want to mark a column in Source's schema as rowtime, but that column is BIGINT. I think we can support it by computed columns? > SQL supports timestamp in Long > -- > > Key: FLINK-14099 > URL: https://issues.apache.org/jira/browse/FLINK-14099 > Project: Flink > Issue Type: New Feature >Reporter: Zijie Lu >Priority: Major > > The rowtime only supports sql timestamp but not long. Can the rowtime field > supports long? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14113) Remove class JobWithJars
TisonKun created FLINK-14113:
Summary: Remove class JobWithJars
Key: FLINK-14113
URL: https://issues.apache.org/jira/browse/FLINK-14113
Project: Flink
Issue Type: Sub-task
Components: Client / Job Submission
Reporter: TisonKun
Fix For: 1.10.0
{{JobWithJars}} is a batch-only concept, acts as a POJO consists of {{Plan}}
and {{URL}}s of libs. We can
1. inline the usage of {{Plan}} and {{URL}}s as we do in streaming case.
2. extract static methods into a utility class said {{ClientUtils}}.
The main purpose here is towards no batch specific concept that doesn't bring
too much good.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931944#comment-16931944 ] Kurt Young commented on FLINK-14078: Just want to clarify, does this only take effect to JDBC connector? > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Priority: Major > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14092) Upgrade Calcite version to 1.21 for Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931917#comment-16931917 ] Timo Walther commented on FLINK-14092: -- [~danny0405] can you check if this is a duplicate issue? Usually we already have Calcite version Jira issue with subtasks of the previous release. > Upgrade Calcite version to 1.21 for Flink SQL > - > > Key: FLINK-14092 > URL: https://issues.apache.org/jira/browse/FLINK-14092 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.9.0 >Reporter: Danny Chan >Priority: Major > Fix For: 1.10.0 > > > Calcite 1.21 has been released recently, we need to upgrade to version 1.21 > for these reasons: > # Previously we have made some temp code to support full data types in sql > parser, since CALCITE-3213 has been resolved, we can do some refactoring for > these codes; > # We also fixed some important bug for Join which bring in from Calcite 1.20 > join like expression promotion, such as CALCITE-3170, CALCITE-3171. > # CALCITE-2302 has been resolved, there is possibility we support implicit > type coercion for Flink now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14099) SQL supports timestamp in Long
[
https://issues.apache.org/jira/browse/FLINK-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931915#comment-16931915
]
Timo Walther commented on FLINK-14099:
--
Could you share a use case/example with us? Actually, we aim to support Long
timestamps for the {{TIMESTAMP WITH LOCAL TIME ZONE}} data type.
> SQL supports timestamp in Long
> --
>
> Key: FLINK-14099
> URL: https://issues.apache.org/jira/browse/FLINK-14099
> Project: Flink
> Issue Type: New Feature
>Reporter: Zijie Lu
>Priority: Major
>
> The rowtime only supports sql timestamp but not long. Can the rowtime field
> supports long?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14111) Flink should be robust to a non-leader Zookeeper host going down
[
https://issues.apache.org/jira/browse/FLINK-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931901#comment-16931901
]
Elias Levy commented on FLINK-14111:
Probable related to FLINK-10052.
> Flink should be robust to a non-leader Zookeeper host going down
>
>
> Key: FLINK-14111
> URL: https://issues.apache.org/jira/browse/FLINK-14111
> Project: Flink
> Issue Type: Wish
> Components: Runtime / Coordination
>Affects Versions: 1.7.2, 1.8.0, 1.8.1, 1.9.0
> Environment: Linux
> JVM 8
> Flink {{1.7.2}}, {{1.8.1}}, {{1.9.0}}
> {{Zookeeper version 3.4.5}}
>Reporter: Aaron Levin
>Priority: Major
>
> I noticed that if a non-leader Zookeeper node goes down and there is still
> quorom in the zookeeper cluster , my flink application will restart anyway. I
> believe it should be possible for Flink applications not to require a restart
> in this scenario.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-14112) Removing zookeeper state should cause the task manager and job managers to restart
Aaron Levin created FLINK-14112:
---
Summary: Removing zookeeper state should cause the task manager
and job managers to restart
Key: FLINK-14112
URL: https://issues.apache.org/jira/browse/FLINK-14112
Project: Flink
Issue Type: Wish
Components: Runtime / Coordination
Affects Versions: 1.9.0, 1.8.1
Reporter: Aaron Levin
Suppose you have a flink application running on a cluster with the following
configuration:
{noformat}
high-availability.zookeeper.path.root: /flink
{noformat}
Now suppose you delete all the znodes within {{/flink}}. I experienced the
following:
* massive amount of logging
* application did not restart
* task manager did not crash or restart
* job manager did not crash or restart
>From this state I had to restart all the task managers and all the job
>managers in order for the flink application to recover.
It would be desirable for the Task Managers and Job Managers to crash if the
znode is not available (though perhaps you all have thought about this more
deeply than I!)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14111) Flink should be robust to a non-leader Zookeeper host going down
Aaron Levin created FLINK-14111:
---
Summary: Flink should be robust to a non-leader Zookeeper host
going down
Key: FLINK-14111
URL: https://issues.apache.org/jira/browse/FLINK-14111
Project: Flink
Issue Type: Wish
Components: Runtime / Coordination
Affects Versions: 1.9.0, 1.8.1, 1.8.0, 1.7.2
Environment: Linux
JVM 8
Flink {{1.7.2}}, {{1.8.1}}, {{1.9.0}}
{{Zookeeper version 3.4.5}}
Reporter: Aaron Levin
I noticed that if a non-leader Zookeeper node goes down and there is still
quorom in the zookeeper cluster , my flink application will restart anyway. I
believe it should be possible for Flink applications not to require a restart
in this scenario.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-14110) Deleting state.backend.rocksdb.localdir causes silent failure
[
https://issues.apache.org/jira/browse/FLINK-14110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Levin updated FLINK-14110:
Description:
Suppose {{state.backend.rocksdb.localdir}} is configured as:
{noformat}
state.backend.rocksdb.localdir: /flink/tmp
{noformat}
If I then run \{{rm -rf /flink/tmp/job_*}} on a host while a Flink application
is running, I will observe the following:
* throughput of my operators running on that host will drop to zero
* the application will not fail or restart
* the task manager will not fail or restart
* in most cases there is nothing in the logs to indicate a failure (I've run
this several times and only once seen an exception - I believe I was lucky and
deleted those directories during a checkpoint or something)
The desired behaviour here would be to throw an exception and crash, instead of
silently dropping throughput to zero. Restarting the Task Manager will resolve
the issues.
I only tried this on Flink {{1.8.1}} and {{1.9.0}}.
was:
Suppose {{state.backend.rocksdb.localdir}} is configured as:
{noformat}
state.backend.rocksdb.localdir: /flink/tmp
{noformat}
If I then run{{rm -rf /flink/tmp/job_*}} on a host while a Flink application is
running, I will observe the following:
* throughput of my operators running on that host will drop to zero
* the application will not fail or restart
* the task manager will not fail or restart
* in most cases there is nothing in the logs to indicate a failure (I've run
this several times and only once seen an exception - I believe I was lucky and
deleted those directories during a checkpoint or something)
The desired behaviour here would be to throw an exception and crash, instead of
silently dropping throughput to zero. Restarting the Task Manager will resolve
the issues.
I only tried this on Flink {{1.8.1}} and {{1.9.0}}.
> Deleting state.backend.rocksdb.localdir causes silent failure
> -
>
> Key: FLINK-14110
> URL: https://issues.apache.org/jira/browse/FLINK-14110
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.8.1, 1.9.0
> Environment: Flink {{1.8.1}} and {{1.9.0}}.
> JVM 8
>Reporter: Aaron Levin
>Priority: Minor
>
> Suppose {{state.backend.rocksdb.localdir}} is configured as:
> {noformat}
> state.backend.rocksdb.localdir: /flink/tmp
> {noformat}
> If I then run \{{rm -rf /flink/tmp/job_*}} on a host while a Flink
> application is running, I will observe the following:
> * throughput of my operators running on that host will drop to zero
> * the application will not fail or restart
> * the task manager will not fail or restart
> * in most cases there is nothing in the logs to indicate a failure (I've run
> this several times and only once seen an exception - I believe I was lucky
> and deleted those directories during a checkpoint or something)
> The desired behaviour here would be to throw an exception and crash, instead
> of silently dropping throughput to zero. Restarting the Task Manager will
> resolve the issues.
> I only tried this on Flink {{1.8.1}} and {{1.9.0}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14110) Deleting state.backend.rocksdb.localdir causes silent failure
Aaron Levin created FLINK-14110:
---
Summary: Deleting state.backend.rocksdb.localdir causes silent
failure
Key: FLINK-14110
URL: https://issues.apache.org/jira/browse/FLINK-14110
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.9.0, 1.8.1
Environment: Flink {{1.8.1}} and {{1.9.0}}.
JVM 8
Reporter: Aaron Levin
Suppose {{state.backend.rocksdb.localdir}} is configured as:
{noformat}
state.backend.rocksdb.localdir: /flink/tmp
{noformat}
If I then run{{rm -rf /flink/tmp/job_*}} on a host while a Flink application is
running, I will observe the following:
* throughput of my operators running on that host will drop to zero
* the application will not fail or restart
* the task manager will not fail or restart
* in most cases there is nothing in the logs to indicate a failure (I've run
this several times and only once seen an exception - I believe I was lucky and
deleted those directories during a checkpoint or something)
The desired behaviour here would be to throw an exception and crash, instead of
silently dropping throughput to zero. Restarting the Task Manager will resolve
the issues.
I only tried this on Flink {{1.8.1}} and {{1.9.0}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-14109) Improve javadocs and tests for high-availability backend
[
https://issues.apache.org/jira/browse/FLINK-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brendan MacDonell updated FLINK-14109:
--
Description:
My team at Sight Machine just finished building a custom HA backend for our
infrastructure. The process was mostly painless, but we ran into a few pain
points on the way:
* {{CompletedCheckpointStore#getLatestCheckpoint}} is not marked as
\{{@Nullable}}, so there was some confusion about whether the Javadocs or (lack
of) annotation is correct. The interface would be clearer if the annotation was
present.
* The javadocs for {{CompletedCheckpointStore#recover}} disagree with the
documentation for {{ZooKeeperCompletedCheckpointStore#recover}}. It's not
immediately clear to someone working on the code that the ZK javadoc is
outdated and the interface documentation is correct.
* The base {{CompletedCheckpointStore}} tests only work with high-availability
backends that keep a list of checkpoints in memory. If the backend persists and
retrieves data from another source the tests will fail. It's fairly simple to
change the tests to lift this requirement though. See [this
gist|https://gist.github.com/bremac/1b3365bc0257dfbd33bcd0b7a7627c00] for an
example.
We've got patches for the points above that we'd be happy to contribute. :)
was:
My team at Sight Machine just finished building a custom HA backend for our
infrastructure. The process was mostly painless, but we ran into a few pain
points on the way:
* {{CompletedCheckpointStore#getLatestCheckpoint}} is not marked as
{{@Nullable, }}so there was some confusion about whether the Javadocs or (lack
of) annotation is correct. The interface would be clearer if the annotation was
present.
* The javadocs for {{CompletedCheckpointStore#recover}} disagree with the
documentation for {{ZooKeeperCompletedCheckpointStore#recover}}. It's not
immediately clear to someone working on the code that the ZK javadoc is
outdated and the interface documentation is correct.
* The base {{CompletedCheckpointStore}} tests only work with high-availability
backends that keep a list of checkpoints in memory. If the backend persists and
retrieves data from another source the tests will fail. It's fairly simple to
change the tests to lift this requirement though. See [this
gist|https://gist.github.com/bremac/1b3365bc0257dfbd33bcd0b7a7627c00] for an
example.
We've got patches for the points above that we'd be happy to contribute. :)
> Improve javadocs and tests for high-availability backend
>
>
> Key: FLINK-14109
> URL: https://issues.apache.org/jira/browse/FLINK-14109
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Brendan MacDonell
>Priority: Minor
>
> My team at Sight Machine just finished building a custom HA backend for our
> infrastructure. The process was mostly painless, but we ran into a few pain
> points on the way:
> * {{CompletedCheckpointStore#getLatestCheckpoint}} is not marked as
> \{{@Nullable}}, so there was some confusion about whether the Javadocs or
> (lack of) annotation is correct. The interface would be clearer if the
> annotation was present.
> * The javadocs for {{CompletedCheckpointStore#recover}} disagree with the
> documentation for {{ZooKeeperCompletedCheckpointStore#recover}}. It's not
> immediately clear to someone working on the code that the ZK javadoc is
> outdated and the interface documentation is correct.
> * The base {{CompletedCheckpointStore}} tests only work with
> high-availability backends that keep a list of checkpoints in memory. If the
> backend persists and retrieves data from another source the tests will fail.
> It's fairly simple to change the tests to lift this requirement though. See
> [this gist|https://gist.github.com/bremac/1b3365bc0257dfbd33bcd0b7a7627c00]
> for an example.
> We've got patches for the points above that we'd be happy to contribute. :)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14109) Improve javadocs and tests for high-availability backend
Brendan MacDonell created FLINK-14109:
-
Summary: Improve javadocs and tests for high-availability backend
Key: FLINK-14109
URL: https://issues.apache.org/jira/browse/FLINK-14109
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Reporter: Brendan MacDonell
My team at Sight Machine just finished building a custom HA backend for our
infrastructure. The process was mostly painless, but we ran into a few pain
points on the way:
* {{CompletedCheckpointStore#getLatestCheckpoint}} is not marked as
{{@Nullable, }}so there was some confusion about whether the Javadocs or (lack
of) annotation is correct. The interface would be clearer if the annotation was
present.
* The javadocs for {{CompletedCheckpointStore#recover}} disagree with the
documentation for {{ZooKeeperCompletedCheckpointStore#recover}}. It's not
immediately clear to someone working on the code that the ZK javadoc is
outdated and the interface documentation is correct.
* The base {{CompletedCheckpointStore}} tests only work with high-availability
backends that keep a list of checkpoints in memory. If the backend persists and
retrieves data from another source the tests will fail. It's fairly simple to
change the tests to lift this requirement though. See [this
gist|https://gist.github.com/bremac/1b3365bc0257dfbd33bcd0b7a7627c00] for an
example.
We've got patches for the points above that we'd be happy to contribute. :)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-14107) Kinesis consumer record emitter deadlock under event time alignment
[ https://issues.apache.org/jira/browse/FLINK-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated FLINK-14107: - Summary: Kinesis consumer record emitter deadlock under event time alignment (was: Kinesis consumer record emitter may deadlock under event time alignment) > Kinesis consumer record emitter deadlock under event time alignment > --- > > Key: FLINK-14107 > URL: https://issues.apache.org/jira/browse/FLINK-14107 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.8.2, 1.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When the emitter reaches the max timestamp for the current queue, it stops > emitting and waits for the max timestamp to advance. Since it simultaneously > selects the next queue as the new "minimum" queue, it may deadlock if the > previous min queue represents the new global lower bound after the max > timestamp advanced. This occurs very infrequently and we were finally able to > reproduce. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-14076) 'ClassNotFoundException: KafkaException' on Flink v1.9 w/ checkpointing
[ https://issues.apache.org/jira/browse/FLINK-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931775#comment-16931775 ] Jeffrey Martin commented on FLINK-14076: Confirmed: the issue is that FlinkKafkaProducer#checkErroneous throws a wrapped KafkaException, which causes a ClassNotFoundException when it hits the JobManager. Not sure how/why this is different on Flink v1.9 vs v1.8. > 'ClassNotFoundException: KafkaException' on Flink v1.9 w/ checkpointing > --- > > Key: FLINK-14076 > URL: https://issues.apache.org/jira/browse/FLINK-14076 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.9.0 >Reporter: Jeffrey Martin >Priority: Major > Attachments: error.txt > > > A Flink job that worked with checkpointing on a Flink v1.8.0 cluster fails on > a Flink v1.9.0 cluster with checkpointing. It works on a Flink v1.9.0 cluster > _without_ checkpointing. It is specifically _enabling checkpointing on > v1.9.0_ that causes the JM to start throwing ClassNotFoundExceptions. Full > stacktrace: [^error.txt] > The job reads from Kafka via FlinkKafkaConsumer and writes to Kafka via > FlinkKafkaProducer. > The jobmanagers and taskmanagers are standalone. > The exception is being raised deep in some Flink serialization code, so I'm > not sure how to go about stepping through this in a debugger. The issue is > happening in an internal repository at my job, but I can try to get a minimal > repro on GitHub if it's not obvious from the error message alone what's > broken. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot edited a comment on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9702#issuecomment-532344762 ## CI report: * b5d14cefce1ec689f991fcc0e02bb229195f4418 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128056494) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot edited a comment on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9702#issuecomment-532344762 ## CI report: * b5d14cefce1ec689f991fcc0e02bb229195f4418 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128056494) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] yxu-valleytider commented on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink
yxu-valleytider commented on issue #9581: [FLINK-13864][streaming]: Modify the StreamingFileSink Builder interface to allow for easier subclassing of StreamingFileSink URL: https://github.com/apache/flink/pull/9581#issuecomment-532348389 @kl0u Agree with most of the review comments. Just want to re-iterate the main reason why introducing a third generics parameter in `RowFormatBuilder` and `BulkFormatBuilder` is due to the above https://github.com/apache/flink/pull/9581#issuecomment-527534328. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] jgrier commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
jgrier commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9702#issuecomment-532347487 @flinkbot approve all This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9702#issuecomment-532344762 ## CI report: * b5d14cefce1ec689f991fcc0e02bb229195f4418 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9691: [FLINK-13965] Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
flinkbot edited a comment on issue #9691: [FLINK-13965] Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation URL: https://github.com/apache/flink/pull/9691#issuecomment-531758345 ## CI report: * 1152df6719c529003d2532cbf6f7a3d01ff6da72 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127794827) * ec28147c7b26085ccd69dde18be2733df755b466 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127978593) * ff5db06fbcc94f73626202a8fe840d6f96aa1521 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128046774) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9673: [FLINK-14031][examples][table]Added the blink planner dependency and …
flinkbot edited a comment on issue #9673: [FLINK-14031][examples][table]Added the blink planner dependency and … URL: https://github.com/apache/flink/pull/9673#issuecomment-530444375 ## CI report: * e7868053366c8881900ae4740519c8d1487d0e91 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127034348) * 715e9fe05903b28a4017ff83fae4b69d59c93d61 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128044332) * ce23eac7f354a95674d1bfc0553ea3d8d47780fc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128046740) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14033) Distributed caches are not registered in Yarn Per Job Cluster Mode
[ https://issues.apache.org/jira/browse/FLINK-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931732#comment-16931732 ] Zhenqiu Huang commented on FLINK-14033: --- Agree. It is not for a jar. The issue happens on another path. Basically, if a user wants to put a hdfs file as resource in the distributed cache, the file is not accessible in runtime due to the blob item is not prepared for per job mode. For session mode, the cache files will be created through rest-client before job graph submission. > Distributed caches are not registered in Yarn Per Job Cluster Mode > -- > > Key: FLINK-14033 > URL: https://issues.apache.org/jira/browse/FLINK-14033 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.0 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Minor > > CacheFiles in StreamExecutionEnvironment is not used in Job Submission in the > Yarn per job cluster mode. Compare to the job submission in session cluster > mode that will upload distributed cache files onto http server in application > master, we should get the cache files in job graph and register them into > blob store in YarnJobClusterEntrypoint. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] tweise commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
tweise commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9702#issuecomment-532339408 @haltwise PTAL. I added a unit test on top of the changes we tested yesterday. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
flinkbot commented on issue #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9702#issuecomment-532338878 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b5d14cefce1ec689f991fcc0e02bb229195f4418 (Tue Sep 17 18:14:37 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14107) Kinesis consumer record emitter may deadlock under event time alignment
[ https://issues.apache.org/jira/browse/FLINK-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14107: --- Labels: pull-request-available (was: ) > Kinesis consumer record emitter may deadlock under event time alignment > --- > > Key: FLINK-14107 > URL: https://issues.apache.org/jira/browse/FLINK-14107 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.8.2, 1.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > > When the emitter reaches the max timestamp for the current queue, it stops > emitting and waits for the max timestamp to advance. Since it simultaneously > selects the next queue as the new "minimum" queue, it may deadlock if the > previous min queue represents the new global lower bound after the max > timestamp advanced. This occurs very infrequently and we were finally able to > reproduce. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] tweise opened a new pull request #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock
tweise opened a new pull request #9702: [FLINK-14107][kinesis] Erroneous queue selection in record emitter may lead to deadlock URL: https://github.com/apache/flink/pull/9702 ## What is the purpose of the change When the emitter reaches the max timestamp for the current queue, it stops emitting and waits for the max timestamp to advance. Since it simultaneously selects the next queue as new "minimum" queue, it may subsequently deadlock if the previous min queue represents the new global lower bound after the max timestamp advanced. This occurs very infrequently and we were finally able to reproduce. ## Verifying this change This change was verified on our internal system and reproduced with the unit test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14076) 'ClassNotFoundException: KafkaException' on Flink v1.9 w/ checkpointing
[ https://issues.apache.org/jira/browse/FLINK-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931704#comment-16931704 ] Jeffrey Martin commented on FLINK-14076: It looks like this is due to a KafkaException getting thrown while Flink is trying to checkpoint. The KafkaException gets wrapped in a 'DeclineCheckpoint' and sent back to the JobManager. The JobManager can't deserialize it because the JobManager's classpath does not include org.apache.kafka:kafka-clients by default. Gonna dig in some more and see if I can figure out which exception is at fault. > 'ClassNotFoundException: KafkaException' on Flink v1.9 w/ checkpointing > --- > > Key: FLINK-14076 > URL: https://issues.apache.org/jira/browse/FLINK-14076 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.9.0 >Reporter: Jeffrey Martin >Priority: Major > Attachments: error.txt > > > A Flink job that worked with checkpointing on a Flink v1.8.0 cluster fails on > a Flink v1.9.0 cluster with checkpointing. It works on a Flink v1.9.0 cluster > _without_ checkpointing. It is specifically _enabling checkpointing on > v1.9.0_ that causes the JM to start throwing ClassNotFoundExceptions. Full > stacktrace: [^error.txt] > The job reads from Kafka via FlinkKafkaConsumer and writes to Kafka via > FlinkKafkaProducer. > The jobmanagers and taskmanagers are standalone. > The exception is being raised deep in some Flink serialization code, so I'm > not sure how to go about stepping through this in a debugger. The issue is > happening in an internal repository at my job, but I can try to get a minimal > repro on GitHub if it's not obvious from the error message alone what's > broken. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-14108) Support for Confluent Kafka schema registry for Avro serialisation
Lasse Nedergaard created FLINK-14108: - Summary: Support for Confluent Kafka schema registry for Avro serialisation Key: FLINK-14108 URL: https://issues.apache.org/jira/browse/FLINK-14108 Project: Flink Issue Type: New Feature Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.10.0 Reporter: Lasse Nedergaard The current implementation in flink-avro-confluent-registry support deserialization with schema lookup in Confluent Kafka schema registry. I would like support for serialization as well, following the same structure as deserialization. With the feature it would be possible to use Confluent schema registry in Sink writing Avro to Kafka and at the same time register the schema used. The test in TestAvroConsumerConfluent need to be updated together with the comment as it indicate it use Confluent schema registry for write, but the example code use SimpleStringSchema. We have a running version, that we would like to give back to the community. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuyi Chen reassigned FLINK-14055:
--
Assignee: Zhenqiu Huang
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9691: [FLINK-13965] Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
flinkbot edited a comment on issue #9691: [FLINK-13965] Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation URL: https://github.com/apache/flink/pull/9691#issuecomment-531758345 ## CI report: * 1152df6719c529003d2532cbf6f7a3d01ff6da72 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127794827) * ec28147c7b26085ccd69dde18be2733df755b466 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127978593) * ff5db06fbcc94f73626202a8fe840d6f96aa1521 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128046774) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9675: [FLINK-13953] [runtime] Facilitate enabling new scheduler in MiniCluster Tests
flinkbot edited a comment on issue #9675: [FLINK-13953] [runtime] Facilitate enabling new scheduler in MiniCluster Tests URL: https://github.com/apache/flink/pull/9675#issuecomment-530686777 ## CI report: * d15f9751632d4c7897f68a0f8829d5facbdfe14e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127138398) * 1c172249b8e66e88d6e3fb90cbc3802fe7521955 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128039253) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-14098) Support multiple sql statements splitting by semicolon for TableEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931685#comment-16931685 ] Xuefu Zhang edited comment on FLINK-14098 at 9/17/19 5:34 PM: -- Yeah, this appears indeed a dupe of or at least related to FLINK-12845, but thanks for pointing out a plausible solution. was (Author: xuefuz): Yeah, this is indeed a dupe of FLINK-12845, but thanks for pointing out a plausible solution. > Support multiple sql statements splitting by semicolon for TableEnvironment > --- > > Key: FLINK-14098 > URL: https://issues.apache.org/jira/browse/FLINK-14098 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Canbin Zheng >Priority: Major > > Currently TableEnvironment.sqlUpdate supports single sql statement parsing by > invoking SqlParser.parseStmt. > Actually, after the work of > [CALCITE-2453|https://issues.apache.org/jira/browse/CALCITE-2453], Calcite’s > SqlParser is able to parse multiple sql statements split by semicolon, IMO, > it’s useful to refactor TableEnvironment.sqlUpdate to support multiple sql > statements too, by invoking SqlParser.parseStmtList instead. > I am not sure whether this is a duplicated ticket, if it is, let me know, > thanks. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9673: [FLINK-14031][examples][table]Added the blink planner dependency and …
flinkbot edited a comment on issue #9673: [FLINK-14031][examples][table]Added the blink planner dependency and … URL: https://github.com/apache/flink/pull/9673#issuecomment-530444375 ## CI report: * e7868053366c8881900ae4740519c8d1487d0e91 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127034348) * 715e9fe05903b28a4017ff83fae4b69d59c93d61 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128044332) * ce23eac7f354a95674d1bfc0553ea3d8d47780fc : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128046740) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14098) Support multiple sql statements splitting by semicolon for TableEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931685#comment-16931685 ] Xuefu Zhang commented on FLINK-14098: - Yeah, this is indeed a dupe of FLINK-12845, but thanks for pointing out a plausible solution. > Support multiple sql statements splitting by semicolon for TableEnvironment > --- > > Key: FLINK-14098 > URL: https://issues.apache.org/jira/browse/FLINK-14098 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Canbin Zheng >Priority: Major > > Currently TableEnvironment.sqlUpdate supports single sql statement parsing by > invoking SqlParser.parseStmt. > Actually, after the work of > [CALCITE-2453|https://issues.apache.org/jira/browse/CALCITE-2453], Calcite’s > SqlParser is able to parse multiple sql statements split by semicolon, IMO, > it’s useful to refactor TableEnvironment.sqlUpdate to support multiple sql > statements too, by invoking SqlParser.parseStmtList instead. > I am not sure whether this is a duplicated ticket, if it is, let me know, > thanks. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] zhuzhurk commented on a change in pull request #9675: [FLINK-13953] [runtime] Facilitate enabling new scheduler in MiniCluster Tests
zhuzhurk commented on a change in pull request #9675: [FLINK-13953] [runtime]
Facilitate enabling new scheduler in MiniCluster Tests
URL: https://github.com/apache/flink/pull/9675#discussion_r325292832
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java
##
@@ -810,15 +812,24 @@ private void startHeartbeatServices() {
log);
}
- private void assignScheduler(
+ private void reassignScheduler(
SchedulerNG newScheduler,
JobManagerJobMetricGroup newJobManagerJobMetricGroup) {
validateRunsInMainThread();
checkState(schedulerNG.requestJobStatus().isTerminalState());
checkState(jobManagerJobMetricGroup == null);
- schedulerNG = newScheduler;
- jobManagerJobMetricGroup = newJobManagerJobMetricGroup;
+ assignScheduler(newScheduler, newJobManagerJobMetricGroup);
+ }
+
+ private void assignScheduler(
+ SchedulerNG scheduler,
+ JobManagerJobMetricGroup jobManagerJobMetricGroup) {
+
+ this.schedulerNG = scheduler;
+ this.jobManagerJobMetricGroup = jobManagerJobMetricGroup;
+
+ log.info("Scheduler {} is used for job {}.", schedulerNG,
jobGraph.getJobID());
Review comment:
Agreed that we can infer from the cluster config log for the scheduler used.
But it is not very straight forward, especially when it is not configured and
the default config is used.
As we are already logging the failover strategy for each job, I think
logging the scheduler for each job is acceptable.
To avoid logging it for multiple times, we can just log it the JobMaster
constructor.
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9675: [FLINK-13953] [runtime] Facilitate enabling new scheduler in MiniCluster Tests
zhuzhurk commented on a change in pull request #9675: [FLINK-13953] [runtime]
Facilitate enabling new scheduler in MiniCluster Tests
URL: https://github.com/apache/flink/pull/9675#discussion_r325292832
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java
##
@@ -810,15 +812,24 @@ private void startHeartbeatServices() {
log);
}
- private void assignScheduler(
+ private void reassignScheduler(
SchedulerNG newScheduler,
JobManagerJobMetricGroup newJobManagerJobMetricGroup) {
validateRunsInMainThread();
checkState(schedulerNG.requestJobStatus().isTerminalState());
checkState(jobManagerJobMetricGroup == null);
- schedulerNG = newScheduler;
- jobManagerJobMetricGroup = newJobManagerJobMetricGroup;
+ assignScheduler(newScheduler, newJobManagerJobMetricGroup);
+ }
+
+ private void assignScheduler(
+ SchedulerNG scheduler,
+ JobManagerJobMetricGroup jobManagerJobMetricGroup) {
+
+ this.schedulerNG = scheduler;
+ this.jobManagerJobMetricGroup = jobManagerJobMetricGroup;
+
+ log.info("Scheduler {} is used for job {}.", schedulerNG,
jobGraph.getJobID());
Review comment:
Agreed that we can infer from the cluster config log for the scheduler used.
But it is not very straight forward, especially when it is not configured and
the default config is used.
As we are already logging the failover strategy for each job, I think
logging the scheduler for each job is acceptable.
To avoid logging it for multiple times, we can just log it the JobMaster
constructor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #9483: [FLINK-13767][task] Refactor StreamInputProcessor#processInput based on InputStatus
pnowojski commented on a change in pull request #9483: [FLINK-13767][task]
Refactor StreamInputProcessor#processInput based on InputStatus
URL: https://github.com/apache/flink/pull/9483#discussion_r325290219
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessor.java
##
@@ -184,10 +184,6 @@ public boolean processInput() throws Exception {
}
checkFinished(status, lastReadInputIndex);
- if (status != InputStatus.MORE_AVAILABLE) {
Review comment:
> are there any purpose to update the selection immediately before?
I'm not sure if there is any difference now, with the updated return status
code. Previously it was returning boolean "more available" for hot looping.
Without up to date input availability, it could attempt to read from incorrect
(empty) input and that would trigger a `volatile` `isAvailable()` access.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9691: [FLINK-13965] Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
flinkbot edited a comment on issue #9691: [FLINK-13965] Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation URL: https://github.com/apache/flink/pull/9691#issuecomment-531758345 ## CI report: * 1152df6719c529003d2532cbf6f7a3d01ff6da72 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127794827) * ec28147c7b26085ccd69dde18be2733df755b466 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127978593) * ff5db06fbcc94f73626202a8fe840d6f96aa1521 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9673: [FLINK-14031][examples][table]Added the blink planner dependency and …
flinkbot edited a comment on issue #9673: [FLINK-14031][examples][table]Added the blink planner dependency and … URL: https://github.com/apache/flink/pull/9673#issuecomment-530444375 ## CI report: * e7868053366c8881900ae4740519c8d1487d0e91 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127034348) * 715e9fe05903b28a4017ff83fae4b69d59c93d61 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/128044332) * ce23eac7f354a95674d1bfc0553ea3d8d47780fc : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
