[GitHub] [flink] flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient
flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient URL: https://github.com/apache/flink/pull/9935#issuecomment-543796546 ## CI report: * c4aef47f26bb9fe9cde0291046c365533ec5b08d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132555336) * ab926c8b9a9235489c35c7f05b3a7ebde3696558 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132564151) * 02cb7f81f32c17d1d1fd6cc2e9713b20bb5fe733 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132622502) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient
flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient URL: https://github.com/apache/flink/pull/9935#issuecomment-543781814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 02cb7f81f32c17d1d1fd6cc2e9713b20bb5fe733 (Sat Oct 19 05:00:13 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient
TisonKun commented on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient URL: https://github.com/apache/flink/pull/9935#issuecomment-544099034 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed
flinkbot edited a comment on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed URL: https://github.com/apache/flink/pull/9940#issuecomment-544080123 ## CI report: * baacecb92f11cd367dc89bc48744fea6de94670b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132623058) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #3672: [Flink-6072] TestCase CheckpointStateRestoreTest::testSetState should be fail
flinkbot edited a comment on issue #3672: [Flink-6072] TestCase CheckpointStateRestoreTest::testSetState should be fail URL: https://github.com/apache/flink/pull/3672#issuecomment-544083819 ## CI report: * c498dea78675a529d032857a00f6fbaf36c0c16e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132623694) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed
flinkbot edited a comment on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed URL: https://github.com/apache/flink/pull/9940#issuecomment-544080123 ## CI report: * baacecb92f11cd367dc89bc48744fea6de94670b : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132623058) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient
flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient URL: https://github.com/apache/flink/pull/9935#issuecomment-543796546 ## CI report: * c4aef47f26bb9fe9cde0291046c365533ec5b08d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132555336) * ab926c8b9a9235489c35c7f05b3a7ebde3696558 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132564151) * 02cb7f81f32c17d1d1fd6cc2e9713b20bb5fe733 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132622502) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #3672: [Flink-6072] TestCase CheckpointStateRestoreTest::testSetState should be fail
flinkbot commented on issue #3672: [Flink-6072] TestCase CheckpointStateRestoreTest::testSetState should be fail URL: https://github.com/apache/flink/pull/3672#issuecomment-544083819 ## CI report: * c498dea78675a529d032857a00f6fbaf36c0c16e : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed
flinkbot commented on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed URL: https://github.com/apache/flink/pull/9940#issuecomment-544080123 ## CI report: * baacecb92f11cd367dc89bc48744fea6de94670b : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient
flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient URL: https://github.com/apache/flink/pull/9935#issuecomment-543796546 ## CI report: * c4aef47f26bb9fe9cde0291046c365533ec5b08d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132555336) * ab926c8b9a9235489c35c7f05b3a7ebde3696558 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132564151) * 02cb7f81f32c17d1d1fd6cc2e9713b20bb5fe733 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132622502) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on issue #3672: [Flink-6072] TestCase CheckpointStateRestoreTest::testSetState should be fail
TisonKun commented on issue #3672: [Flink-6072] TestCase CheckpointStateRestoreTest::testSetState should be fail URL: https://github.com/apache/flink/pull/3672#issuecomment-544077697 @guoweiM Is this issue still valid? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed
flinkbot commented on issue #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed URL: https://github.com/apache/flink/pull/9940#issuecomment-544075551 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit baacecb92f11cd367dc89bc48744fea6de94670b (Sat Oct 19 03:25:13 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14434) Dispatcher#createJobManagerRunner should returns on creation succeed, not after startJobManagerRunner
[
https://issues.apache.org/jira/browse/FLINK-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-14434:
---
Labels: pull-request-available (was: )
> Dispatcher#createJobManagerRunner should returns on creation succeed, not
> after startJobManagerRunner
> -
>
> Key: FLINK-14434
> URL: https://issues.apache.org/jira/browse/FLINK-14434
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Attachments: patch.diff
>
>
> In an edge case, let's said
> 1) job finished nearly immediately
> 2) Dispatcher has been suspended in {{#startJobManagerRunner}} after
> {{jobManagerRunner.start();}} but before {{return jobManagerRunner;}}
> due to
> 1) we put {{jobManagerRunnerFutures}} with {{#startJobManagerRunner}}
> finished.
> 2) the creation of JobManagerRunner doesn't happen in MainThread.
> it is a possible execution order
> 1) JobManagerRunner created in akka-dispatcher thread
> 2) then apply {{Dispatcher#startJobManagerRunner}}
> 3) until {{jobManagerRunner.start();}} and before {{return jobManagerRunner;}}
> 4) this thread suspended
> 5) job finished, execute callback on MainThread
> 6) {{jobManagerRunnerFutures.get(jobID).getNow(null)}} returns {{null}}
> because akka-dispatcher thread doesn't {{return jobManagerRunner;}}
> 7) it report {{There is a newer JobManagerRunner for the job}} but actually
> not.
> **Solution**
> Two perspective but we can even have them both.
> 1. return {{jobManagerRunnerFuture}} in {{#createJobManagerRunner}}, let
> {{#startJobManagerRunner}} an action
> 2. on JobManagerRunner created, execute {{#startJobManagerRunner}} in
> MainThread.
> CC [~trohrmann]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] TisonKun opened a new pull request #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed
TisonKun opened a new pull request #9940: [FLINK-14434][coordination] Dispatcher#createJobManagerRunner returns on creation succeed URL: https://github.com/apache/flink/pull/9940 ## What is the purpose of the change Fixes a race condition described detailedly in [FLINK-14434](https://issues.apache.org/jira/browse/FLINK-14434#). ## Brief change log Let `Dispatcher#createJobManagerRunner` returns on creation succeed. ## Verifying this change I think this change is straightforward and current behavior doesn't break guarded by existing tests. The diff attached on corresponding JIRA can reproduce the issue but I'm afraid it is always invasive to emulate this case(add a debug latch between `jobManagerRunner.start` and `return` for testing hook in my mind). Due to the change is straightforward and easy to reason, I tend to avoid invasive testing hook. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-14434) Dispatcher#createJobManagerRunner should returns on creation succeed, not after startJobManagerRunner
[
https://issues.apache.org/jira/browse/FLINK-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955058#comment-16955058
]
Zili Chen edited comment on FLINK-14434 at 10/19/19 3:16 AM:
-
Thanks for your reply [~trohrmann]. I final a possibly better option3 for this
issue. The diff is tiny and expressive.
https://github.com/TisonKun/flink/commit/baacecb92f11cd367dc89bc48744fea6de94670b
In short, the problem described above is mainly caused by when "job manager
runner result future" called back, "job manager runner created future"
conceptually finished but not completed. Revisit the semantic of
{{#createJobManagerRunner}} we are able to just return a future represent the
creation and let the caller take care of the start.
Compared with option 2, this approach has a clear semantic and a subtle
difference that it execute {{JobManagerRunner}} in akka-dispatcher thread, not
in MainThread. We internally have some issue if starting jm runner happens in
dispatcher MainThread[1] but it doesn't exist in community codebase.
I'm glad to help with FLINK-11843 and FLINK-11719 on the review side. For this
issue I'd like to send this tiny patch as a pull request so that you can
coordinate patches depending on your schedule.
[1] FYI: It is an interesting case but bias a bit from community codebase. We
move the access to job registry totally to Dispatcher so that when job manager
runner granted leadership it send a RPC to Dispatcher for querying what job
scheduling status now. Our fork is currently based on 1.7 so that there is a
dead-lock execution order with solution option 2 above.
1. job manager runner called {{#start}} in Dispatcher MainThread
2. job manager runner leader election service started, and if it is
standalone(non-ha), it directly calls grantLeadership
3. job manager runner on granted leadership, send a RPC to Dispatcher for
querying and wait for the result.
4. because {{#start}} occupied the MainThread, the later RPC cannot be
processed.
We can workaround this case in many ways such as dispatch action a bit, but it
might infer that if we can schedule an action out of Dispatcher MainThread
without worry about synchronization provided by single-thread, we'd better to
do it.
was (Author: tison):
Thanks for your reply [~trohrmann]. I final a possibly better option3 for this
issue. The diff is tiny and expressive.
https://github.com/TisonKun/flink/commit/baacecb92f11cd367dc89bc48744fea6de94670b
In short, the problem described above is mainly caused by when "job manager
runner result future" called back, "job manager runner created future"
conceptually finished but not completed. Revisit the semantic of
{{#createJobManagerRunner}} we are able to just return a future represent the
creation and let the caller take care of the start.
Compared with option 2, this approach has a clear semantic and a subtle
difference that it execute {{JobManagerRunner}} in akka-dispatcher thread, not
in MainThread. We internally have some issue if starting jm runner happens in
dispatcher MainThread[1] but it doesn't exist in community codebase.
I'm glad to help with FLINK-11843 and FLINK-11719 on the review side. For this
issue I'd like to send this tiny patch as a pull request so that you can
coordinate patches depending on your schedule.
[1] FYI: It is an interesting case but bias a bit from community codebase. We
move the job registry totally to Dispatcher so that when job manager runner
granted leadership it send a RPC to Dispatcher for querying what job scheduling
status now. Our fork is currently based on 1.7 so that there is a dead-lock
execution order with solution option 2 above.
1. job manager runner called {{#start}} in Dispatcher MainThread
2. job manager runner leader election service started, and if it is
standalone(non-ha), it directly calls grantLeadership
3. job manager runner on granted leadership, send a RPC to Dispatcher for
querying and wait for the result.
4. because {{#start}} occupied the MainThread, the later RPC cannot be
processed.
We can workaround this case in many ways such as dispatch action a bit, but it
might infer that if we can schedule an action out of Dispatcher MainThread
without worry about synchronization provided by single-thread, we'd better to
do it.
> Dispatcher#createJobManagerRunner should returns on creation succeed, not
> after startJobManagerRunner
> -
>
> Key: FLINK-14434
> URL: https://issues.apache.org/jira/browse/FLINK-14434
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Fix For: 1.10.0
>
>
[GitHub] [flink] flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient
flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient URL: https://github.com/apache/flink/pull/9935#issuecomment-543796546 ## CI report: * c4aef47f26bb9fe9cde0291046c365533ec5b08d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132555336) * ab926c8b9a9235489c35c7f05b3a7ebde3696558 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132564151) * 02cb7f81f32c17d1d1fd6cc2e9713b20bb5fe733 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14434) Dispatcher#createJobManagerRunner should returns on creation succeed, not after startJobManagerRunner
[
https://issues.apache.org/jira/browse/FLINK-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955058#comment-16955058
]
Zili Chen commented on FLINK-14434:
---
Thanks for your reply [~trohrmann]. I final a possibly better option3 for this
issue. The diff is tiny and expressive.
https://github.com/TisonKun/flink/commit/baacecb92f11cd367dc89bc48744fea6de94670b
In short, the problem described above is mainly caused by when "job manager
runner result future" called back, "job manager runner created future"
conceptually finished but not completed. Revisit the semantic of
{{#createJobManagerRunner}} we are able to just return a future represent the
creation and let the caller take care of the start.
Compared with option 2, this approach has a clear semantic and a subtle
difference that it execute {{JobManagerRunner}} in akka-dispatcher thread, not
in MainThread. We internally have some issue if starting jm runner happens in
dispatcher MainThread[1] but it doesn't exist in community codebase.
I'm glad to help with FLINK-11843 and FLINK-11719 on the review side. For this
issue I'd like to send this tiny patch as a pull request so that you can
coordinate patches depending on your schedule.
[1] FYI: It is an interesting case but bias a bit from community codebase. We
move the job registry totally to Dispatcher so that when job manager runner
granted leadership it send a RPC to Dispatcher for querying what job scheduling
status now. Our fork is currently based on 1.7 so that there is a dead-lock
execution order with solution option 2 above.
1. job manager runner called {{#start}} in Dispatcher MainThread
2. job manager runner leader election service started, and if it is
standalone(non-ha), it directly calls grantLeadership
3. job manager runner on granted leadership, send a RPC to Dispatcher for
querying and wait for the result.
4. because {{#start}} occupied the MainThread, the later RPC cannot be
processed.
We can workaround this case in many ways such as dispatch action a bit, but it
might infer that if we can schedule an action out of Dispatcher MainThread
without worry about synchronization provided by single-thread, we'd better to
do it.
> Dispatcher#createJobManagerRunner should returns on creation succeed, not
> after startJobManagerRunner
> -
>
> Key: FLINK-14434
> URL: https://issues.apache.org/jira/browse/FLINK-14434
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: patch.diff
>
>
> In an edge case, let's said
> 1) job finished nearly immediately
> 2) Dispatcher has been suspended in {{#startJobManagerRunner}} after
> {{jobManagerRunner.start();}} but before {{return jobManagerRunner;}}
> due to
> 1) we put {{jobManagerRunnerFutures}} with {{#startJobManagerRunner}}
> finished.
> 2) the creation of JobManagerRunner doesn't happen in MainThread.
> it is a possible execution order
> 1) JobManagerRunner created in akka-dispatcher thread
> 2) then apply {{Dispatcher#startJobManagerRunner}}
> 3) until {{jobManagerRunner.start();}} and before {{return jobManagerRunner;}}
> 4) this thread suspended
> 5) job finished, execute callback on MainThread
> 6) {{jobManagerRunnerFutures.get(jobID).getNow(null)}} returns {{null}}
> because akka-dispatcher thread doesn't {{return jobManagerRunner;}}
> 7) it report {{There is a newer JobManagerRunner for the job}} but actually
> not.
> **Solution**
> Two perspective but we can even have them both.
> 1. return {{jobManagerRunnerFuture}} in {{#createJobManagerRunner}}, let
> {{#startJobManagerRunner}} an action
> 2. on JobManagerRunner created, execute {{#startJobManagerRunner}} in
> MainThread.
> CC [~trohrmann]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14327) Getting "Could not forward element to next operator" error
[
https://issues.apache.org/jira/browse/FLINK-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955057#comment-16955057
]
Zhu Zhu edited comment on FLINK-14327 at 10/19/19 3:07 AM:
---
Hi [~ASK5], the NPE happens in flinkBroadcast1.scala which seems to be user
code.
Could you check flinkBroadcast1.scala line 41?
If it's the code you attached, I guess it's these two lines in case that the
property is null.
{code:java}
val loc = v.get("locationID").asInstanceOf[String]
val temperature = v.get("temp").asDouble()
{code}
was (Author: zhuzh):
Hi [~ASK5], the NPE to happen in flinkBroadcast1.scala which seems to be user
code.
Could you check flinkBroadcast1.scala line 41?
If it's the code you attached, I guess it's these two lines in case that the
property is null.
{code:java}
val loc = v.get("locationID").asInstanceOf[String]
val temperature = v.get("temp").asDouble()
{code}
> Getting "Could not forward element to next operator" error
> --
>
> Key: FLINK-14327
> URL: https://issues.apache.org/jira/browse/FLINK-14327
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.9.0
>Reporter: ASK5
>Priority: Major
> Fix For: 1.9.2
>
> Attachments: so2.png
>
>
> val TEMPERATURE_THRESHOLD: Double = 50.00
> val see: StreamExecutionEnvironment =
> StreamExecutionEnvironment.getExecutionEnvironment
> val properties = new Properties()
> properties.setProperty("zookeeper.connect", "localhost:2181")
> properties.setProperty("bootstrap.servers", "localhost:9092")
> val src = see.addSource(new FlinkKafkaConsumer010[ObjectNode]("broadcast",
> new JSONKeyValueDeserializationSchema(false),
> properties)).name("kafkaSource")
> case class Event(locationID: String, temp: Double)
> var data = src.map { v => {
> val loc = v.get("locationID").asInstanceOf[String]
> val temperature = v.get("temp").asDouble()
> (loc, temperature)
> }}
> data = data
> .keyBy(
> v => v._1
> )
> data.print()
> see.execute()
> ---*
> And I'm getting the following error while consuming json file from Kafka:-
>
> {{Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException: Job execution
> failed at flinkBroadcast1$.main(flinkBroadcast1.scala:59) at
> flinkBroadcast1.main(flinkBroadcast1.scala)Caused by: java.lang.Exception:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator...Caused by:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator...Caused by:
> java.lang.NullPointerException}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14327) Getting "Could not forward element to next operator" error
[
https://issues.apache.org/jira/browse/FLINK-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955057#comment-16955057
]
Zhu Zhu commented on FLINK-14327:
-
Hi [~ASK5], the NPE to happen in flinkBroadcast1.scala which seems to be user
code.
Could you check flinkBroadcast1.scala line 41?
If it's the code you attached, I guess it's these two lines in case that the
property is null.
{code:java}
val loc = v.get("locationID").asInstanceOf[String]
val temperature = v.get("temp").asDouble()
{code}
> Getting "Could not forward element to next operator" error
> --
>
> Key: FLINK-14327
> URL: https://issues.apache.org/jira/browse/FLINK-14327
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.9.0
>Reporter: ASK5
>Priority: Major
> Fix For: 1.9.2
>
> Attachments: so2.png
>
>
> val TEMPERATURE_THRESHOLD: Double = 50.00
> val see: StreamExecutionEnvironment =
> StreamExecutionEnvironment.getExecutionEnvironment
> val properties = new Properties()
> properties.setProperty("zookeeper.connect", "localhost:2181")
> properties.setProperty("bootstrap.servers", "localhost:9092")
> val src = see.addSource(new FlinkKafkaConsumer010[ObjectNode]("broadcast",
> new JSONKeyValueDeserializationSchema(false),
> properties)).name("kafkaSource")
> case class Event(locationID: String, temp: Double)
> var data = src.map { v => {
> val loc = v.get("locationID").asInstanceOf[String]
> val temperature = v.get("temp").asDouble()
> (loc, temperature)
> }}
> data = data
> .keyBy(
> v => v._1
> )
> data.print()
> see.execute()
> ---*
> And I'm getting the following error while consuming json file from Kafka:-
>
> {{Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException: Job execution
> failed at flinkBroadcast1$.main(flinkBroadcast1.scala:59) at
> flinkBroadcast1.main(flinkBroadcast1.scala)Caused by: java.lang.Exception:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator...Caused by:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator...Caused by:
> java.lang.NullPointerException}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538885706 ## CI report: * dd22d245973b8caf25f8f26a392a1ca95f863736 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/13062) * 6ae59be715e9454885ea94df650b61a5adc9d9f1 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/131049194) * d0c22910828121d32e0a847ff33bd705fb9aae35 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131050758) * 538f0528ff44a648a878e7c9eb5bac62b005c39c : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/131190746) * 62f76f23a0dae45d3efa9a773d37155f16565df8 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131224128) * 3f43b51e9e9c8d3121786043901a4cbe3e4822a3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131246054) * 0c687c49cfa6d37560873461babdf460cee09303 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131266147) * f92953f34e8c0bd59b684fd823b59cb13b68c689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132074402) * 542324b1fbe108ce517b3234c0a5b7867caf13f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132618133) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-14401) create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf
[ https://issues.apache.org/jira/browse/FLINK-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li closed FLINK-14401. Resolution: Fixed master: d1b06ac4e8ab033c699b656cb0f1d924b6bafc56 > create FunctionDefinitionUtil to instantiate regular java class-based udf and > add HiveFunctionDefinitionFactory to instantiate both flink and hive udf > -- > > Key: FLINK-14401 > URL: https://issues.apache.org/jira/browse/FLINK-14401 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / API >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0b025af1643af49ec0a2214f8e669f2d66c42ec5 (Sat Oct 19 01:25:34 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] asfgit closed pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instant
asfgit closed pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0b025af1643af49ec0a2214f8e669f2d66c42ec5 (Sat Oct 19 01:23:32 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to insta
bowenli86 commented on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-544038106 @xuefuz thanks. Merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542451844 ## CI report: * 293c5d2734301dca498db30d053dce971eccebc3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132068083) * 7474de440d60cc72d5d92c2a2f3986d3553cd496 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132183772) * 50f378f1a0d69ec5570c76a9fc6352ed904fc174 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132203495) * 304b713773632b779a993d33c485dc9dcfa092ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132415887) * 0b025af1643af49ec0a2214f8e669f2d66c42ec5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132615982) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538885706 ## CI report: * dd22d245973b8caf25f8f26a392a1ca95f863736 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/13062) * 6ae59be715e9454885ea94df650b61a5adc9d9f1 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/131049194) * d0c22910828121d32e0a847ff33bd705fb9aae35 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131050758) * 538f0528ff44a648a878e7c9eb5bac62b005c39c : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/131190746) * 62f76f23a0dae45d3efa9a773d37155f16565df8 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131224128) * 3f43b51e9e9c8d3121786043901a4cbe3e4822a3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131246054) * 0c687c49cfa6d37560873461babdf460cee09303 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131266147) * f92953f34e8c0bd59b684fd823b59cb13b68c689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132074402) * 542324b1fbe108ce517b3234c0a5b7867caf13f2 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132618133) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538874419 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 542324b1fbe108ce517b3234c0a5b7867caf13f2 (Sat Oct 19 00:51:56 UTC 2019) **Warnings:** * **4 pom.xml files were touched**: Check for build and licensing issues. * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14332).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] lukess commented on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
lukess commented on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-544029415 @zentol 1. yes, I think I will follow community decision. Both self release or shared release work to me. One thing in my mind is how to control quality and release between upstream and many repos (or one shared repo) 2. yes, I can bump flink-metrics-dropwizard, and let them depends on the same codahale. 3. ok cool, I just follow the JMXReporter to create new factory, and updated docs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538885706 ## CI report: * dd22d245973b8caf25f8f26a392a1ca95f863736 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/13062) * 6ae59be715e9454885ea94df650b61a5adc9d9f1 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/131049194) * d0c22910828121d32e0a847ff33bd705fb9aae35 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131050758) * 538f0528ff44a648a878e7c9eb5bac62b005c39c : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/131190746) * 62f76f23a0dae45d3efa9a773d37155f16565df8 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131224128) * 3f43b51e9e9c8d3121786043901a4cbe3e4822a3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131246054) * 0c687c49cfa6d37560873461babdf460cee09303 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131266147) * f92953f34e8c0bd59b684fd823b59cb13b68c689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132074402) * 542324b1fbe108ce517b3234c0a5b7867caf13f2 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538874419 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 542324b1fbe108ce517b3234c0a5b7867caf13f2 (Sat Oct 19 00:28:32 UTC 2019) **Warnings:** * **4 pom.xml files were touched**: Check for build and licensing issues. * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14332).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542451844 ## CI report: * 293c5d2734301dca498db30d053dce971eccebc3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132068083) * 7474de440d60cc72d5d92c2a2f3986d3553cd496 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132183772) * 50f378f1a0d69ec5570c76a9fc6352ed904fc174 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132203495) * 304b713773632b779a993d33c485dc9dcfa092ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132415887) * 0b025af1643af49ec0a2214f8e669f2d66c42ec5 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132615982) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-14211) Add jobManager address configuration for SqlClient
[
https://issues.apache.org/jira/browse/FLINK-14211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangwei closed FLINK-14211.
Resolution: Incomplete
> Add jobManager address configuration for SqlClient
> --
>
> Key: FLINK-14211
> URL: https://issues.apache.org/jira/browse/FLINK-14211
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.9.0
>Reporter: zhangwei
>Priority: Trivial
>
> Add jobmanager option in deployment configuration, allow SQL clients to
> submit jobs to the remote flink cluster
> {code:java}
> deployment:
> # general cluster communication timeout in ms
> response-timeout: 5000
> # (optional) address from cluster to gateway
> gateway-address: ""
> # (optional) port from cluster to gateway
> gateway-port: 0
> # (optional) jobmanager address
> jobmanager: 127.0.0.1:8081
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543850869 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e7fb90f82b660fc906b569933820e7a1859bc6ba (Sat Oct 19 00:05:06 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add
Module interface and ModuleManager
URL: https://github.com/apache/flink/pull/9937#discussion_r336709310
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableEnvironment.java
##
@@ -237,6 +257,13 @@ static TableEnvironment create(EnvironmentSettings
settings) {
*/
String[] listCatalogs();
+ /**
+* Gets the names of all modules registered in this environment.
+*
+* @return A list of the names of all registered modules.
+*/
+ String[] listModules();
Review comment:
Not sure if it was in the discussion. If the order of modules matters (such
as in function resolution), maybe we should return something has the order in
it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add
Module interface and ModuleManager
URL: https://github.com/apache/flink/pull/9937#discussion_r336710263
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.module;
+
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.module.exceptions.ModuleAlreadyExistException;
+import org.apache.flink.table.module.exceptions.ModuleNotFoundException;
+import org.apache.flink.util.StringUtils;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Responsible for loading/unloading modules, managing their life cycles, and
resolving module objects.
+ */
+public class ModuleManager {
+ private LinkedHashMap modules;
+
+ public ModuleManager() {
+ this.modules = new LinkedHashMap<>();
+
+ // TODO: Add Core module to modules
+ }
+
+ /**
+* Load a module with given name.
+*
+* @param name name of the module
+* @param module the module instance
+* @throws ModuleAlreadyExistException thrown when there is already a
module with the same name
+*/
+ public void loadModule(String name, Module module) throws
ModuleAlreadyExistException {
+ checkArgument(!StringUtils.isNullOrWhitespaceOnly(name), "name
cannot be null or empty string");
+ checkNotNull(module, "module cannot be null");
+
+ if (!modules.containsKey(name)) {
+ modules.put(name, module);
+ } else {
+ throw new ModuleAlreadyExistException(name);
+ }
+ }
+
+ /**
+* Unload a module with given name.
+*
+* @param name name of the module
+* @throws ModuleNotFoundException thrown when there is no module with
the given name
+*/
+ public void unloadModule(String name) throws ModuleNotFoundException {
+ if (modules.containsKey(name)) {
+ modules.remove(name);
+ } else {
+ throw new ModuleNotFoundException(name);
+ }
+ }
+
+ /**
+* Get names of all modules registered.
+*
+* @return a list of names of registered modules
+*/
+ public List listModules() {
+ return new ArrayList<>(modules.keySet());
+ }
+
+ /**
+* Get names of all functions from all modules.
+*
+* @return a set of names of registered modules.
+*/
+ public Set listFunctions() {
+ return modules.values().stream()
+ .map(m -> m.listFunctions())
+ .flatMap(n -> n.stream())
+ .collect(Collectors.toSet());
+ }
+
+ /**
+* Get an optional of {@link FunctionDefinition} by a given name.
+*
+* @param name name of the function
+* @return an optional of {@link FunctionDefinition}
+*/
+ public Optional getFunctionDefinition(String name) {
+ Optional module = modules.values().stream()
Review comment:
I'm not sure if I'm mistaken, but This might not be efficient, as it
accumulates all functions from all modules and then find the first one.
A cleaner one might be calling lookupFunction in each module until one is
found.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add
Module interface and ModuleManager
URL: https://github.com/apache/flink/pull/9937#discussion_r33670
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -118,7 +125,8 @@ public static TableEnvironmentImpl
create(EnvironmentSettings settings) {
settings.getBuiltInCatalogName(),
new
GenericInMemoryCatalog(settings.getBuiltInCatalogName(),
settings.getBuiltInDatabaseName()));
- FunctionCatalog functionCatalog = new
FunctionCatalog(catalogManager);
+ ModuleManager moduleManager = new ModuleManager();
Review comment:
Shouldn't we use the instance passed from the constructor instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add
Module interface and ModuleManager
URL: https://github.com/apache/flink/pull/9937#discussion_r336710366
##
File path:
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/module/ModuleManagerTest.java
##
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.module;
+
+/**
+ * Test for {@link ModuleManager}.
+ */
+public class ModuleManagerTest {
+
Review comment:
Interesting that this contains no test at all.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add
Module interface and ModuleManager
URL: https://github.com/apache/flink/pull/9937#discussion_r336708662
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableEnvironment.java
##
@@ -103,6 +106,23 @@ static TableEnvironment create(EnvironmentSettings
settings) {
*/
Optional getCatalog(String catalogName);
+ /**
+* Load a module with given name.
+*
+* @param moduleName name of the module
+* @param module the module instance
+* @throws ModuleAlreadyExistException thrown when there is already a
module with the same name
+*/
+ void loadModule(String moduleName, Module module) throws
ModuleAlreadyExistException;
+
+ /**
Review comment:
Same as above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add
Module interface and ModuleManager
URL: https://github.com/apache/flink/pull/9937#discussion_r336709983
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.module;
+
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.module.exceptions.ModuleAlreadyExistException;
+import org.apache.flink.table.module.exceptions.ModuleNotFoundException;
+import org.apache.flink.util.StringUtils;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Responsible for loading/unloading modules, managing their life cycles, and
resolving module objects.
+ */
+public class ModuleManager {
+ private LinkedHashMap modules;
+
+ public ModuleManager() {
+ this.modules = new LinkedHashMap<>();
+
+ // TODO: Add Core module to modules
+ }
+
+ /**
+* Load a module with given name.
+*
+* @param name name of the module
+* @param module the module instance
+* @throws ModuleAlreadyExistException thrown when there is already a
module with the same name
+*/
+ public void loadModule(String name, Module module) throws
ModuleAlreadyExistException {
+ checkArgument(!StringUtils.isNullOrWhitespaceOnly(name), "name
cannot be null or empty string");
+ checkNotNull(module, "module cannot be null");
+
+ if (!modules.containsKey(name)) {
+ modules.put(name, module);
+ } else {
+ throw new ModuleAlreadyExistException(name);
+ }
+ }
+
+ /**
+* Unload a module with given name.
+*
+* @param name name of the module
+* @throws ModuleNotFoundException thrown when there is no module with
the given name
+*/
+ public void unloadModule(String name) throws ModuleNotFoundException {
+ if (modules.containsKey(name)) {
+ modules.remove(name);
+ } else {
+ throw new ModuleNotFoundException(name);
+ }
+ }
+
+ /**
+* Get names of all modules registered.
+*
+* @return a list of names of registered modules
+*/
+ public List listModules() {
+ return new ArrayList<>(modules.keySet());
+ }
+
+ /**
+* Get names of all functions from all modules.
+*
+* @return a set of names of registered modules.
+*/
+ public Set listFunctions() {
+ return modules.values().stream()
+ .map(m -> m.listFunctions())
+ .flatMap(n -> n.stream())
+ .collect(Collectors.toSet());
+ }
+
+ /**
+* Get an optional of {@link FunctionDefinition} by a given name.
+*
+* @param name name of the function
+* @return an optional of {@link FunctionDefinition}
+*/
+ public Optional getFunctionDefinition(String name) {
Review comment:
get -> lookup?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542451844 ## CI report: * 293c5d2734301dca498db30d053dce971eccebc3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132068083) * 7474de440d60cc72d5d92c2a2f3986d3553cd496 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132183772) * 50f378f1a0d69ec5570c76a9fc6352ed904fc174 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132203495) * 304b713773632b779a993d33c485dc9dcfa092ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132415887) * 0b025af1643af49ec0a2214f8e669f2d66c42ec5 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 304b713773632b779a993d33c485dc9dcfa092ee (Fri Oct 18 23:38:36 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinit
bowenli86 commented on a change in pull request #9908:
[FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular
java class-based udf and add HiveFunctionDefinitionFactory to instantiate both
flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336707901
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/factories/HiveFunctionDefinitionFactory.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog.hive.factories;
+
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.connectors.hive.HiveTableFactory;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.config.CatalogConfig;
+import org.apache.flink.table.catalog.hive.client.HiveShim;
+import org.apache.flink.table.catalog.hive.client.HiveShimLoader;
+import org.apache.flink.table.catalog.hive.descriptors.HiveCatalogValidator;
+import org.apache.flink.table.factories.FunctionDefinitionFactory;
+import org.apache.flink.table.functions.FunctionDefinitionUtil;
+import org.apache.flink.table.functions.AggregateFunctionDefinition;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.ScalarFunctionDefinition;
+import org.apache.flink.table.functions.TableFunctionDefinition;
+import org.apache.flink.table.functions.hive.HiveFunctionWrapper;
+import org.apache.flink.table.functions.hive.HiveGenericUDAF;
+import org.apache.flink.table.functions.hive.HiveGenericUDF;
+import org.apache.flink.table.functions.hive.HiveGenericUDTF;
+import org.apache.flink.table.functions.hive.HiveSimpleUDF;
+import org.apache.flink.types.Row;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.UDAF;
+import org.apache.hadoop.hive.ql.exec.UDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A factory to instantiate Hive UDFs as Flink UDFs.
+ */
+public class HiveFunctionDefinitionFactory implements
FunctionDefinitionFactory {
+ private static final Logger LOG =
LoggerFactory.getLogger(HiveTableFactory.class);
+
+ private final String hiveVersion;
+ private final HiveShim hiveShim;
+
+ public HiveFunctionDefinitionFactory(HiveConf hiveConf) {
+ // this has to come from hiveConf, otherwise we may lose what
user specifies in the yaml file
+ this.hiveVersion =
checkNotNull(hiveConf.get(HiveCatalogValidator.CATALOG_HIVE_VERSION),
+ "Hive version is not defined");
+ this.hiveShim = HiveShimLoader.loadHiveShim(hiveVersion);
+ }
+
+ @Override
+ public FunctionDefinition createFunctionDefinition(String name,
CatalogFunction catalogFunction) {
Review comment:
sure, I created https://issues.apache.org/jira/browse/FLINK-14458 for
tracking purpose
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543850869 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e7fb90f82b660fc906b569933820e7a1859bc6ba (Fri Oct 18 23:35:34 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add
Module interface and ModuleManager
URL: https://github.com/apache/flink/pull/9937#discussion_r336707535
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableEnvironment.java
##
@@ -103,6 +106,23 @@ static TableEnvironment create(EnvironmentSettings
settings) {
*/
Optional getCatalog(String catalogName);
+ /**
+* Load a module with given name.
Review comment:
Nit: use a similar comment as those in existing methods.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
xuefuz commented on a change in pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#discussion_r336707182 ## File path: flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/table/api/java/internal/StreamTableEnvironmentImpl.java ## @@ -104,7 +106,9 @@ public static StreamTableEnvironment create( settings.getBuiltInCatalogName(), new GenericInMemoryCatalog(settings.getBuiltInCatalogName(), settings.getBuiltInDatabaseName())); - FunctionCatalog functionCatalog = new FunctionCatalog(catalogManager); + ModuleManager moduleManager = new ModuleManager(); + + FunctionCatalog functionCatalog = new FunctionCatalog(catalogManager, moduleManager); Review comment: I saw repeated pattern like this. It's probably better to have a constructor that doesn't take a module manager and instead creates it inside the constructor. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543850869 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e7fb90f82b660fc906b569933820e7a1859bc6ba (Fri Oct 18 23:31:29 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132576286) * 4487cf268c806093138280a257cb32a61f607bf5 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/132599005) * 8f5a9adec72bd450aa0edc6c8cc3934300ae47ed : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/132602552) * e7fb90f82b660fc906b569933820e7a1859bc6ba : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132604853) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132576286) * 4487cf268c806093138280a257cb32a61f607bf5 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/132599005) * 8f5a9adec72bd450aa0edc6c8cc3934300ae47ed : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/132602552) * e7fb90f82b660fc906b569933820e7a1859bc6ba : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132604853) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132576286) * 4487cf268c806093138280a257cb32a61f607bf5 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/132599005) * 8f5a9adec72bd450aa0edc6c8cc3934300ae47ed : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132602552) * e7fb90f82b660fc906b569933820e7a1859bc6ba : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-14458) Assert implementation classes of catalog objects
Bowen Li created FLINK-14458: Summary: Assert implementation classes of catalog objects Key: FLINK-14458 URL: https://issues.apache.org/jira/browse/FLINK-14458 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Bowen Li Assignee: Bowen Li We need to assert implementation classes of catalog objects, including table/view/function/partition/stats/db, to make sure they are of the right implementations -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 304b713773632b779a993d33c485dc9dcfa092ee (Fri Oct 18 21:43:32 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinition
xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive]
create FunctionDefinitionUtil to instantiate regular java class-based udf and
add HiveFunctionDefinitionFactory to instantiate both flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336686199
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/factories/HiveFunctionDefinitionFactory.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog.hive.factories;
+
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.connectors.hive.HiveTableFactory;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.config.CatalogConfig;
+import org.apache.flink.table.catalog.hive.client.HiveShim;
+import org.apache.flink.table.catalog.hive.client.HiveShimLoader;
+import org.apache.flink.table.catalog.hive.descriptors.HiveCatalogValidator;
+import org.apache.flink.table.factories.FunctionDefinitionFactory;
+import org.apache.flink.table.functions.FunctionDefinitionUtil;
+import org.apache.flink.table.functions.AggregateFunctionDefinition;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.ScalarFunctionDefinition;
+import org.apache.flink.table.functions.TableFunctionDefinition;
+import org.apache.flink.table.functions.hive.HiveFunctionWrapper;
+import org.apache.flink.table.functions.hive.HiveGenericUDAF;
+import org.apache.flink.table.functions.hive.HiveGenericUDF;
+import org.apache.flink.table.functions.hive.HiveGenericUDTF;
+import org.apache.flink.table.functions.hive.HiveSimpleUDF;
+import org.apache.flink.types.Row;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.UDAF;
+import org.apache.hadoop.hive.ql.exec.UDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A factory to instantiate Hive UDFs as Flink UDFs.
+ */
+public class HiveFunctionDefinitionFactory implements
FunctionDefinitionFactory {
+ private static final Logger LOG =
LoggerFactory.getLogger(HiveTableFactory.class);
+
+ private final String hiveVersion;
+ private final HiveShim hiveShim;
+
+ public HiveFunctionDefinitionFactory(HiveConf hiveConf) {
+ // this has to come from hiveConf, otherwise we may lose what
user specifies in the yaml file
+ this.hiveVersion =
checkNotNull(hiveConf.get(HiveCatalogValidator.CATALOG_HIVE_VERSION),
+ "Hive version is not defined");
+ this.hiveShim = HiveShimLoader.loadHiveShim(hiveVersion);
+ }
+
+ @Override
+ public FunctionDefinition createFunctionDefinition(String name,
CatalogFunction catalogFunction) {
Review comment:
ok. Maybe we can create a JIRA for this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinition
xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive]
create FunctionDefinitionUtil to instantiate regular java class-based udf and
add HiveFunctionDefinitionFactory to instantiate both flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336685999
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/FunctionCatalog.java
##
@@ -242,21 +242,17 @@ public void
registerTempCatalogScalarFunction(ObjectIdentifier oi, ScalarFunctio
CatalogFunction catalogFunction = catalog.getFunction(
new
ObjectPath(catalogManager.getCurrentDatabase(), functionName));
- if (catalog.getTableFactory().isPresent() &&
- catalog.getTableFactory().get() instanceof
FunctionDefinitionFactory) {
-
- FunctionDefinitionFactory factory =
(FunctionDefinitionFactory) catalog.getTableFactory().get();
-
- userCandidate =
factory.createFunctionDefinition(functionName, catalogFunction);
-
- return Optional.of(
- new FunctionLookup.Result(
-
ObjectIdentifier.of(catalogManager.getCurrentCatalog(),
catalogManager.getCurrentDatabase(), name),
- userCandidate)
- );
+ if (catalog.getFunctionDefinitionFactory().isPresent())
{
+ userCandidate =
catalog.getFunctionDefinitionFactory().get().createFunctionDefinition(functionName,
catalogFunction);
} else {
- // TODO: should go through function definition
discover service
+ userCandidate =
FunctionDefinitionUtil.createFunctionDefinition(functionName, catalogFunction);
Review comment:
uhn, I guess this is fine. Especially, the API is consistent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132576286) * 4487cf268c806093138280a257cb32a61f607bf5 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132599005) * 8f5a9adec72bd450aa0edc6c8cc3934300ae47ed : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
bowenli86 commented on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543958736 cc @xuefuz @lirui-apache @zjuwangg This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543850869 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e7fb90f82b660fc906b569933820e7a1859bc6ba (Fri Oct 18 21:32:20 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543850869 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8f5a9adec72bd450aa0edc6c8cc3934300ae47ed (Fri Oct 18 21:20:07 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot edited a comment on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9939#issuecomment-543914792 ## CI report: * 68f395d06159abf1c80b412fc835c851be733e64 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132590678) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132576286) * 4487cf268c806093138280a257cb32a61f607bf5 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha commented on issue #5609: [FLINK-8822] RotateLogFile may not work well when sed version is belo…
aljoscha commented on issue #5609: [FLINK-8822] RotateLogFile may not work well when sed version is belo… URL: https://github.com/apache/flink/pull/5609#issuecomment-543947233 The solution would be to just not use extended RE, i.e. the `sed` call becomes `sed s/\.[0-9][0-9]*$//`. I'll fix that next week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot edited a comment on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9938#issuecomment-543904749 ## CI report: * b21455ce25efcc61aa51bf00a51d569621028723 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132588016) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543850869 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4487cf268c806093138280a257cb32a61f607bf5 (Fri Oct 18 20:43:28 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot edited a comment on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9939#issuecomment-543914792 ## CI report: * 68f395d06159abf1c80b412fc835c851be733e64 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132590678) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot commented on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9939#issuecomment-543914792 ## CI report: * 68f395d06159abf1c80b412fc835c851be733e64 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot edited a comment on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9938#issuecomment-543904749 ## CI report: * b21455ce25efcc61aa51bf00a51d569621028723 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132588016) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot commented on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9938#issuecomment-543904749 ## CI report: * b21455ce25efcc61aa51bf00a51d569621028723 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14442) Add time based interval execution to JDBC connectors.
[ https://issues.apache.org/jira/browse/FLINK-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954927#comment-16954927 ] None none commented on FLINK-14442: --- Ok so it will flush on which ever is reached first? > Add time based interval execution to JDBC connectors. > - > > Key: FLINK-14442 > URL: https://issues.apache.org/jira/browse/FLINK-14442 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0 >Reporter: None none >Priority: Minor > > Hi, currently the JDBC sink/output only supports batch interval execution. > For data to be streamed/committed to the JDBC database we need to wait for > the batch interval to be filled up. > For example if you set a batch interval of 100 but only get 99 records then > no data will be committed to the database. > The JDBC driver should maybe also have a time based interval so that data is > eventually pushed to the database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot commented on issue #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9939#issuecomment-543899220 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 68f395d06159abf1c80b412fc835c851be733e64 (Fri Oct 18 19:23:54 UTC 2019) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14175).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14175) Upgrade KPL version in flink-connector-kinesis to fix application OOM
[ https://issues.apache.org/jira/browse/FLINK-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954923#comment-16954923 ] Abhilasha Seth commented on FLINK-14175: The master branch already has the updated KPL version as it was updated in https://issues.apache.org/jira/browse/FLINK-12847. PR for release-1.9 branch: [https://github.com/apache/flink/pull/9938] PR for release-1.8 branch: [https://github.com/apache/flink/pull/9939] > Upgrade KPL version in flink-connector-kinesis to fix application OOM > - > > Key: FLINK-14175 > URL: https://issues.apache.org/jira/browse/FLINK-14175 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.6.3, 1.6.4, 1.6.5, 1.7.2, 1.7.3, 1.8.0, 1.8.1, 1.8.2, > 1.9.0 > Environment: [link title|http://example.com][link > title|http://example.com] >Reporter: Abhilasha Seth >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The [KPL > version|https://github.com/apache/flink/blob/release-1.9/flink-connectors/flink-connector-kinesis/pom.xml#L38] > (0.12.9) used by flink-connector-kinesis in the affected Flink versions has > a thread leak bug that causes applications to run out of memory after > frequent restarts: > KPL Issue - [https://github.com/awslabs/amazon-kinesis-producer/issues/224] > Fix - [https://github.com/awslabs/amazon-kinesis-producer/pull/225/files] > Upgrading KPL to 0.12.10 or higher is necessary to avoid this issue. The > recommended version to upgrade would be the latest (0.13.1) > Note that KPL version in Flink 1.10.0 has been updated to the latest version > (0.13.1): https://issues.apache.org/jira/browse/FLINK-12847 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] abhilashaseth opened a new pull request #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
abhilashaseth opened a new pull request #9939: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9939 This fixes https://github.com/awslabs/amazon-kinesis-producer/issues/224. * JIRA - https://issues.apache.org/jira/browse/FLINK-14175 * mvn clean package succeeds after this change * The version upgrade has already been performed (hence tested) in Flink 1.10.0 as part of FLINK-12847. ## What is the purpose of the change This pull request upgrades the KPL version in flink-kinesis-connector to fix thread leak mentioned in ttps://github.com/awslabs/amazon-kinesis-producer/issues/224. ## Brief change log Update the KPL version from 0.12.9 to 0.13.1 ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
flinkbot commented on issue #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9938#issuecomment-543896275 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b21455ce25efcc61aa51bf00a51d569621028723 (Fri Oct 18 19:17:16 UTC 2019) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14175).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14175) Upgrade KPL version in flink-connector-kinesis to fix application OOM
[ https://issues.apache.org/jira/browse/FLINK-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14175: --- Labels: pull-request-available (was: ) > Upgrade KPL version in flink-connector-kinesis to fix application OOM > - > > Key: FLINK-14175 > URL: https://issues.apache.org/jira/browse/FLINK-14175 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.6.3, 1.6.4, 1.6.5, 1.7.2, 1.7.3, 1.8.0, 1.8.1, 1.8.2, > 1.9.0 > Environment: [link title|http://example.com][link > title|http://example.com] >Reporter: Abhilasha Seth >Priority: Major > Labels: pull-request-available > > The [KPL > version|https://github.com/apache/flink/blob/release-1.9/flink-connectors/flink-connector-kinesis/pom.xml#L38] > (0.12.9) used by flink-connector-kinesis in the affected Flink versions has > a thread leak bug that causes applications to run out of memory after > frequent restarts: > KPL Issue - [https://github.com/awslabs/amazon-kinesis-producer/issues/224] > Fix - [https://github.com/awslabs/amazon-kinesis-producer/pull/225/files] > Upgrading KPL to 0.12.10 or higher is necessary to avoid this issue. The > recommended version to upgrade would be the latest (0.13.1) > Note that KPL version in Flink 1.10.0 has been updated to the latest version > (0.13.1): https://issues.apache.org/jira/browse/FLINK-12847 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] abhilashaseth opened a new pull request #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1
abhilashaseth opened a new pull request #9938: [FLINK-14175] [flink-connector-kinesis] Update KPL version to 0.13.1 URL: https://github.com/apache/flink/pull/9938 This fixes https://github.com/awslabs/amazon-kinesis-producer/issues/224. * JIRA - https://issues.apache.org/jira/browse/FLINK-14175 * mvn clean package succeeds after this change * The version upgrade has already been performed (hence tested) in Flink 1.10.0 as part of FLINK-12847. ## What is the purpose of the change This pull request upgrades the KPL version in flink-kinesis-connector to fix thread leak mentioned in ttps://github.com/awslabs/amazon-kinesis-producer/issues/224. ## Brief change log Update the KPL version from 0.12.9 to 0.13.1 ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132576286) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9936: [FLINK-14450][runtime] Change SchedulingTopology to extend base topology
flinkbot edited a comment on issue #9936: [FLINK-14450][runtime] Change SchedulingTopology to extend base topology URL: https://github.com/apache/flink/pull/9936#issuecomment-543851152 ## CI report: * 8c26cb49219cd022f5c7187b3b4c9b012295d798 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132573450) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session
TisonKun commented on a change in pull request #9832: [FLINK-11843] Bind
lifespan of Dispatcher to leader session
URL: https://github.com/apache/flink/pull/9832#discussion_r336620347
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##
@@ -197,19 +191,37 @@ public void onStart() throws Exception {
onFatalError(exception);
throw exception;
}
+
+ startRecoveredJobs();
Review comment:
Well I notice that `submitJob` actually have a persisting step which isn't
required for a recovered job. But if so, why we do this `onAddedJobGraph`...As
is the code works due to we support update job graph(rewrite with the same in
all cases though) in job graph store.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session
flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session URL: https://github.com/apache/flink/pull/9832#issuecomment-537039262 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 271703eda6f6c55b1641a54206109ef659f62854 (Fri Oct 18 18:26:00 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session
TisonKun commented on a change in pull request #9832: [FLINK-11843] Bind
lifespan of Dispatcher to leader session
URL: https://github.com/apache/flink/pull/9832#discussion_r336620347
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##
@@ -197,19 +191,37 @@ public void onStart() throws Exception {
onFatalError(exception);
throw exception;
}
+
+ startRecoveredJobs();
Review comment:
Well I notice that `submitJob` actually have a persisting step which isn't
required for a recovered job. But if so, why we do this `onAddedJobGraph`...As
is the code it works due to we support update job graph in job graph store.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session
TisonKun commented on a change in pull request #9832: [FLINK-11843] Bind
lifespan of Dispatcher to leader session
URL: https://github.com/apache/flink/pull/9832#discussion_r336620347
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##
@@ -197,19 +191,37 @@ public void onStart() throws Exception {
onFatalError(exception);
throw exception;
}
+
+ startRecoveredJobs();
Review comment:
Well I notice that `submitJob` actually have a persisting step which isn't
required for a recovered job. But if so, why we do this `onAddedJobGraph`...As
is the code works due to we support update job graph in job graph store.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot edited a comment on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132576286) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot commented on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543861679 ## CI report: * 969a3e2e079661951786ddd22cac7b3e78e6e9d0 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9936: [FLINK-14450][runtime] Change SchedulingTopology to extend base topology
flinkbot edited a comment on issue #9936: [FLINK-14450][runtime] Change SchedulingTopology to extend base topology URL: https://github.com/apache/flink/pull/9936#issuecomment-543851152 ## CI report: * 8c26cb49219cd022f5c7187b3b4c9b012295d798 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132573450) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535848177 ## CI report: * 5d9e3113e10bbf0220b4f8e080ecd1cddd9c9059 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129411044) * 55c766190d205798534f6d09170e66463c832082 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129544930) * 8c7b264e3d17dfa2eb2019ecb470cbe4ef9fa339 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/13191) * dd55f2f66aec3cbcee7211e2769496d0a5c873e2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131961829) * 69dbf0fa215fa7cb7aaadd596fcaadd3747697d2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132096181) * 16077687afe140b71bc91a99315c50219ac8c732 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132279941) * 0332dd139f1b2857efe5de3852cf4bb4573575a3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325960) * 7542ca7c45594de8ac6f7112e43d0a7bcd8615b6 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132476281) * 416f6a5535f1efe984fc57f13ac0d2a84c9b5fa8 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132537704) * 1e44c409a68a5746d40a573c754ae7805dc75e2f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132570553) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 304b713773632b779a993d33c485dc9dcfa092ee (Fri Oct 18 17:45:14 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinit
bowenli86 commented on a change in pull request #9908:
[FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular
java class-based udf and add HiveFunctionDefinitionFactory to instantiate both
flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336603833
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/FunctionCatalog.java
##
@@ -242,21 +242,17 @@ public void
registerTempCatalogScalarFunction(ObjectIdentifier oi, ScalarFunctio
CatalogFunction catalogFunction = catalog.getFunction(
new
ObjectPath(catalogManager.getCurrentDatabase(), functionName));
- if (catalog.getTableFactory().isPresent() &&
- catalog.getTableFactory().get() instanceof
FunctionDefinitionFactory) {
-
- FunctionDefinitionFactory factory =
(FunctionDefinitionFactory) catalog.getTableFactory().get();
-
- userCandidate =
factory.createFunctionDefinition(functionName, catalogFunction);
-
- return Optional.of(
- new FunctionLookup.Result(
-
ObjectIdentifier.of(catalogManager.getCurrentCatalog(),
catalogManager.getCurrentDatabase(), name),
- userCandidate)
- );
+ if (catalog.getFunctionDefinitionFactory().isPresent())
{
+ userCandidate =
catalog.getFunctionDefinitionFactory().get().createFunctionDefinition(functionName,
catalogFunction);
} else {
- // TODO: should go through function definition
discover service
+ userCandidate =
FunctionDefinitionUtil.createFunctionDefinition(functionName, catalogFunction);
Review comment:
not sure if I understand it from an end-to-end perspective, can you provide
some code example?
if the catalog doesn't provide a func def factory, what would planner do
then? FunctionDefinitionUtil.createFunctionDefinition() serves as the default
logic if users don't provide a func def factory to instantiate func def; if
they want customized func def instantiation logic, they should provide a func
def factory.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 304b713773632b779a993d33c485dc9dcfa092ee (Fri Oct 18 17:34:04 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinit
bowenli86 commented on a change in pull request #9908:
[FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular
java class-based udf and add HiveFunctionDefinitionFactory to instantiate both
flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336598770
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/factories/HiveFunctionDefinitionFactory.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog.hive.factories;
+
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.connectors.hive.HiveTableFactory;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.config.CatalogConfig;
+import org.apache.flink.table.catalog.hive.client.HiveShim;
+import org.apache.flink.table.catalog.hive.client.HiveShimLoader;
+import org.apache.flink.table.catalog.hive.descriptors.HiveCatalogValidator;
+import org.apache.flink.table.factories.FunctionDefinitionFactory;
+import org.apache.flink.table.functions.FunctionDefinitionUtil;
+import org.apache.flink.table.functions.AggregateFunctionDefinition;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.ScalarFunctionDefinition;
+import org.apache.flink.table.functions.TableFunctionDefinition;
+import org.apache.flink.table.functions.hive.HiveFunctionWrapper;
+import org.apache.flink.table.functions.hive.HiveGenericUDAF;
+import org.apache.flink.table.functions.hive.HiveGenericUDF;
+import org.apache.flink.table.functions.hive.HiveGenericUDTF;
+import org.apache.flink.table.functions.hive.HiveSimpleUDF;
+import org.apache.flink.types.Row;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.UDAF;
+import org.apache.hadoop.hive.ql.exec.UDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A factory to instantiate Hive UDFs as Flink UDFs.
+ */
+public class HiveFunctionDefinitionFactory implements
FunctionDefinitionFactory {
+ private static final Logger LOG =
LoggerFactory.getLogger(HiveTableFactory.class);
+
+ private final String hiveVersion;
+ private final HiveShim hiveShim;
+
+ public HiveFunctionDefinitionFactory(HiveConf hiveConf) {
+ // this has to come from hiveConf, otherwise we may lose what
user specifies in the yaml file
+ this.hiveVersion =
checkNotNull(hiveConf.get(HiveCatalogValidator.CATALOG_HIVE_VERSION),
+ "Hive version is not defined");
+ this.hiveShim = HiveShimLoader.loadHiveShim(hiveVersion);
+ }
+
+ @Override
+ public FunctionDefinition createFunctionDefinition(String name,
CatalogFunction catalogFunction) {
Review comment:
I'm leaning towards not doing that now because we don't assert on any
catalog table/view/partition/db objects currently.
If it turns out we need to go that route, we'd better update them all at
once in the future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9936: [FLINK-14450][runtime] Change SchedulingTopology to extend base topology
flinkbot commented on issue #9936: [FLINK-14450][runtime] Change SchedulingTopology to extend base topology URL: https://github.com/apache/flink/pull/9936#issuecomment-543851152 ## CI report: * 8c26cb49219cd022f5c7187b3b4c9b012295d798 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager
flinkbot commented on issue #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937#issuecomment-543850869 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 969a3e2e079661951786ddd22cac7b3e78e6e9d0 (Fri Oct 18 17:31:22 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient
flinkbot edited a comment on issue #9935: [FLINK-14456][client] Remove or shift down field from ClusterClient URL: https://github.com/apache/flink/pull/9935#issuecomment-543796546 ## CI report: * c4aef47f26bb9fe9cde0291046c365533ec5b08d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132555336) * ab926c8b9a9235489c35c7f05b3a7ebde3696558 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132564151) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 opened a new pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager
bowenli86 opened a new pull request #9937: [FLINK-14416][table] Add Module interface and ModuleManager URL: https://github.com/apache/flink/pull/9937 ## What is the purpose of the change Add Module interface and ModuleManager to Flink SQL. ## Brief change log - developed Module interface and ModuleManager class - added ModuleManager to impls of TableEnvironment and FunctionCatalog - added module related APIs to TableEnvironment - adapted lots of code (prod and testing) to adopt ModuleManager ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) Docs will be added later This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14416) Add Module interface and ModuleManager
[ https://issues.apache.org/jira/browse/FLINK-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14416: --- Labels: pull-request-available (was: ) > Add Module interface and ModuleManager > -- > > Key: FLINK-14416 > URL: https://issues.apache.org/jira/browse/FLINK-14416 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 304b713773632b779a993d33c485dc9dcfa092ee (Fri Oct 18 17:27:56 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinition
xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive]
create FunctionDefinitionUtil to instantiate regular java class-based udf and
add HiveFunctionDefinitionFactory to instantiate both flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336596387
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/FunctionCatalog.java
##
@@ -242,21 +242,17 @@ public void
registerTempCatalogScalarFunction(ObjectIdentifier oi, ScalarFunctio
CatalogFunction catalogFunction = catalog.getFunction(
new
ObjectPath(catalogManager.getCurrentDatabase(), functionName));
- if (catalog.getTableFactory().isPresent() &&
- catalog.getTableFactory().get() instanceof
FunctionDefinitionFactory) {
-
- FunctionDefinitionFactory factory =
(FunctionDefinitionFactory) catalog.getTableFactory().get();
-
- userCandidate =
factory.createFunctionDefinition(functionName, catalogFunction);
-
- return Optional.of(
- new FunctionLookup.Result(
-
ObjectIdentifier.of(catalogManager.getCurrentCatalog(),
catalogManager.getCurrentDatabase(), name),
- userCandidate)
- );
+ if (catalog.getFunctionDefinitionFactory().isPresent())
{
+ userCandidate =
catalog.getFunctionDefinitionFactory().get().createFunctionDefinition(functionName,
catalogFunction);
} else {
- // TODO: should go through function definition
discover service
+ userCandidate =
FunctionDefinitionUtil.createFunctionDefinition(functionName, catalogFunction);
Review comment:
My point is, if the catalog doesn't provide a FunctionDefFactory, we should
not call FunctionDefinitionUtil.createFunctionDefinition() because we don't
know if that's a catalog implementation wants. It's better for a catalog
implementation to explicitly invoke this call, if that's what it needs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to
flinkbot edited a comment on issue #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinitionFactory to instantiate both flink and hive udf URL: https://github.com/apache/flink/pull/9908#issuecomment-542444814 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 304b713773632b779a993d33c485dc9dcfa092ee (Fri Oct 18 17:22:52 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9542: [FLINK-13873][metrics] Change the column family as tags for influxdb …
flinkbot edited a comment on issue #9542: [FLINK-13873][metrics] Change the column family as tags for influxdb … URL: https://github.com/apache/flink/pull/9542#issuecomment-525276537 ## CI report: * e8636926351f3d406962dcadba275e20e49aff39 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124736151) * d23ea97e8419bbacea0698b3ba82a459d940cf38 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128606376) * 05977cd49eb306d768668be4e8cb31034343df02 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128606947) * 057d453e5e00656bcfcb87d1d69172f614b2d11f : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128681895) * 5001042b5fc5202da14c06e2d21faf1427f50a66 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128683546) * 3a1e39e889daf0fca82c54982911ded35df6cb77 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128692102) * 2745a3cb02e601a1a26360e9d6b3f0af5428c66a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129027621) * e108aa24e739d6f48f63efc560c3fc049b509860 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131172797) * 24a270d468dea677379713d5cf402ea453d9f222 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131246042) * acf1c2b9add8c3b903a8485ed41c9f0b18d97729 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/131260202) * 2f5f28b6bc2c24e81b330a3ad65cde2a23a1af95 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132564124) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535848177 ## CI report: * 5d9e3113e10bbf0220b4f8e080ecd1cddd9c9059 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129411044) * 55c766190d205798534f6d09170e66463c832082 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129544930) * 8c7b264e3d17dfa2eb2019ecb470cbe4ef9fa339 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/13191) * dd55f2f66aec3cbcee7211e2769496d0a5c873e2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131961829) * 69dbf0fa215fa7cb7aaadd596fcaadd3747697d2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132096181) * 16077687afe140b71bc91a99315c50219ac8c732 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132279941) * 0332dd139f1b2857efe5de3852cf4bb4573575a3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325960) * 7542ca7c45594de8ac6f7112e43d0a7bcd8615b6 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132476281) * 416f6a5535f1efe984fc57f13ac0d2a84c9b5fa8 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132537704) * 1e44c409a68a5746d40a573c754ae7805dc75e2f : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132570553) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinition
xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive]
create FunctionDefinitionUtil to instantiate regular java class-based udf and
add HiveFunctionDefinitionFactory to instantiate both flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336594274
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/factories/HiveFunctionDefinitionFactory.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog.hive.factories;
+
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.connectors.hive.HiveTableFactory;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.config.CatalogConfig;
+import org.apache.flink.table.catalog.hive.client.HiveShim;
+import org.apache.flink.table.catalog.hive.client.HiveShimLoader;
+import org.apache.flink.table.catalog.hive.descriptors.HiveCatalogValidator;
+import org.apache.flink.table.factories.FunctionDefinitionFactory;
+import org.apache.flink.table.functions.FunctionDefinitionUtil;
+import org.apache.flink.table.functions.AggregateFunctionDefinition;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.ScalarFunctionDefinition;
+import org.apache.flink.table.functions.TableFunctionDefinition;
+import org.apache.flink.table.functions.hive.HiveFunctionWrapper;
+import org.apache.flink.table.functions.hive.HiveGenericUDAF;
+import org.apache.flink.table.functions.hive.HiveGenericUDF;
+import org.apache.flink.table.functions.hive.HiveGenericUDTF;
+import org.apache.flink.table.functions.hive.HiveSimpleUDF;
+import org.apache.flink.types.Row;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.UDAF;
+import org.apache.hadoop.hive.ql.exec.UDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A factory to instantiate Hive UDFs as Flink UDFs.
+ */
+public class HiveFunctionDefinitionFactory implements
FunctionDefinitionFactory {
+ private static final Logger LOG =
LoggerFactory.getLogger(HiveTableFactory.class);
+
+ private final String hiveVersion;
+ private final HiveShim hiveShim;
+
+ public HiveFunctionDefinitionFactory(HiveConf hiveConf) {
+ // this has to come from hiveConf, otherwise we may lose what
user specifies in the yaml file
+ this.hiveVersion =
checkNotNull(hiveConf.get(HiveCatalogValidator.CATALOG_HIVE_VERSION),
+ "Hive version is not defined");
+ this.hiveShim = HiveShimLoader.loadHiveShim(hiveVersion);
+ }
+
+ @Override
+ public FunctionDefinition createFunctionDefinition(String name,
CatalogFunction catalogFunction) {
Review comment:
Yeah, that's the case today, but in the future there might be other
implementations. If Hive catalog can only handle CatalogFunctionImpl, it's good
to make an assertion. This is minor though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive] create FunctionDefinitionUtil to instantiate regular java class-based udf and add HiveFunctionDefinition
xuefuz commented on a change in pull request #9908: [FLINK-14401][table][hive]
create FunctionDefinitionUtil to instantiate regular java class-based udf and
add HiveFunctionDefinitionFactory to instantiate both flink and hive udf
URL: https://github.com/apache/flink/pull/9908#discussion_r336594274
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/factories/HiveFunctionDefinitionFactory.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog.hive.factories;
+
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.connectors.hive.HiveTableFactory;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.config.CatalogConfig;
+import org.apache.flink.table.catalog.hive.client.HiveShim;
+import org.apache.flink.table.catalog.hive.client.HiveShimLoader;
+import org.apache.flink.table.catalog.hive.descriptors.HiveCatalogValidator;
+import org.apache.flink.table.factories.FunctionDefinitionFactory;
+import org.apache.flink.table.functions.FunctionDefinitionUtil;
+import org.apache.flink.table.functions.AggregateFunctionDefinition;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.ScalarFunctionDefinition;
+import org.apache.flink.table.functions.TableFunctionDefinition;
+import org.apache.flink.table.functions.hive.HiveFunctionWrapper;
+import org.apache.flink.table.functions.hive.HiveGenericUDAF;
+import org.apache.flink.table.functions.hive.HiveGenericUDF;
+import org.apache.flink.table.functions.hive.HiveGenericUDTF;
+import org.apache.flink.table.functions.hive.HiveSimpleUDF;
+import org.apache.flink.types.Row;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.UDAF;
+import org.apache.hadoop.hive.ql.exec.UDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A factory to instantiate Hive UDFs as Flink UDFs.
+ */
+public class HiveFunctionDefinitionFactory implements
FunctionDefinitionFactory {
+ private static final Logger LOG =
LoggerFactory.getLogger(HiveTableFactory.class);
+
+ private final String hiveVersion;
+ private final HiveShim hiveShim;
+
+ public HiveFunctionDefinitionFactory(HiveConf hiveConf) {
+ // this has to come from hiveConf, otherwise we may lose what
user specifies in the yaml file
+ this.hiveVersion =
checkNotNull(hiveConf.get(HiveCatalogValidator.CATALOG_HIVE_VERSION),
+ "Hive version is not defined");
+ this.hiveShim = HiveShimLoader.loadHiveShim(hiveVersion);
+ }
+
+ @Override
+ public FunctionDefinition createFunctionDefinition(String name,
CatalogFunction catalogFunction) {
Review comment:
Yeah, that's the case today, but in the future there might be other
implementations. If Hive catalog can only handle CatalogFunctionImpl.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
