[jira] [Commented] (FLINK-14436) Add getter for ContextEnvironment instance in StreamContextEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954321#comment-16954321 ] Zili Chen commented on FLINK-14436: --- Thanks for your reply [~KevinZwx]. I'll appreciate it if you participant in the discussion thread and share your user case so that our evolution in client API can satisfy your user case also. Feel free to start your own discussion thread on this topic if you prefer it. > Add getter for ContextEnvironment instance in StreamContextEnvironment > -- > > Key: FLINK-14436 > URL: https://issues.apache.org/jira/browse/FLINK-14436 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.9.0 >Reporter: Kevin Zhang >Priority: Major > Labels: easyfix > > Currently in StreamContextEnvironment the member variable ctx, an instance of > ContextEnvironment, is private. It's very helpful to add a getter for ctx > because through the ContextEnvironment we can access the ClusterClient object > of the application, and then get the information of the flink jobs or > submit/cancel jobs. It's especially useful for developers building a server > upon flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9906: [FLINK-14395] Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6
flinkbot edited a comment on issue #9906: [FLINK-14395] Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6 URL: https://github.com/apache/flink/pull/9906#issuecomment-542220479 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 42e6df0f54fb098572d925acb0b203ce2642a9ea (Fri Oct 18 06:56:41 UTC 2019) **Warnings:** * **2 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14395).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] yanghua commented on issue #9906: [FLINK-14395] Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6
yanghua commented on issue #9906: [FLINK-14395] Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6 URL: https://github.com/apache/flink/pull/9906#issuecomment-543551865 This PR go to wrong direction, closing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] yanghua closed pull request #9906: [FLINK-14395] Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6
yanghua closed pull request #9906: [FLINK-14395] Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6 URL: https://github.com/apache/flink/pull/9906 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. URL: https://github.com/apache/flink/pull/9910#issuecomment-542644991 ## CI report: * 8d1696b97872fdd9fd3aca725c8b58ad8227f6f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132137194) * bd54289054f8b4d144cdb7cdffb7f3d0dee274b4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325988) * 7df654afc7b6f8da2fdc6eee5a516b9bd789d0da : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/132481274) * 90c4d2d3c6859de464622c8934b92f007c71734a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132482718) * d2493bf0f15605fd0ec1aee8afa2ba47dc4267a6 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-14436) Add getter for ContextEnvironment instance in StreamContextEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang closed FLINK-14436. --- Resolution: Won't Fix > Add getter for ContextEnvironment instance in StreamContextEnvironment > -- > > Key: FLINK-14436 > URL: https://issues.apache.org/jira/browse/FLINK-14436 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.9.0 >Reporter: Kevin Zhang >Priority: Major > Labels: easyfix > > Currently in StreamContextEnvironment the member variable ctx, an instance of > ContextEnvironment, is private. It's very helpful to add a getter for ctx > because through the ContextEnvironment we can access the ClusterClient object > of the application, and then get the information of the flink jobs or > submit/cancel jobs. It's especially useful for developers building a server > upon flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9255: [FLINK-13034] Introduce isEmpty method for MapState
flinkbot edited a comment on issue #9255: [FLINK-13034] Introduce isEmpty method for MapState URL: https://github.com/apache/flink/pull/9255#issuecomment-515868641 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7a533f9e7472c73bbef42da35599df295c4f7666 (Fri Oct 18 06:48:33 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❗ 3. Needs [attention] from. - Needs attention by @aljoscha [PMC], @wuchong [committer] * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-13195) Add create table support for SqlClient
[ https://issues.apache.org/jira/browse/FLINK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-13195: --- Assignee: Zheng Hu (was: Danny Chen) > Add create table support for SqlClient > -- > > Key: FLINK-13195 > URL: https://issues.apache.org/jira/browse/FLINK-13195 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.9.0 >Reporter: Danny Chen >Assignee: Zheng Hu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Myasuka commented on issue #9255: [FLINK-13034] Introduce isEmpty method for MapState
Myasuka commented on issue #9255: [FLINK-13034] Introduce isEmpty method for MapState URL: https://github.com/apache/flink/pull/9255#issuecomment-543547724 Failed due to [FLINK-14445](https://issues.apache.org/jira/browse/FLINK-14445) @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14436) Add getter for ContextEnvironment instance in StreamContextEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954317#comment-16954317 ] Kevin Zhang commented on FLINK-14436: - Thanks [~tison] for the information, I'll close this issue. > Add getter for ContextEnvironment instance in StreamContextEnvironment > -- > > Key: FLINK-14436 > URL: https://issues.apache.org/jira/browse/FLINK-14436 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.9.0 >Reporter: Kevin Zhang >Priority: Major > Labels: easyfix > > Currently in StreamContextEnvironment the member variable ctx, an instance of > ContextEnvironment, is private. It's very helpful to add a getter for ctx > because through the ContextEnvironment we can access the ClusterClient object > of the application, and then get the information of the flink jobs or > submit/cancel jobs. It's especially useful for developers building a server > upon flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14434) Dispatcher#createJobManagerRunner should returns on creation succeed, not after startJobManagerRunner
[
https://issues.apache.org/jira/browse/FLINK-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954316#comment-16954316
]
Zili Chen commented on FLINK-14434:
---
Now I would prefer 2 only because with 1 we possibly miss exception in
JobManagerRunner#start so that user receive submission success but request job
result with job not found because start failed and the job manager runner
future removed without a result.
> Dispatcher#createJobManagerRunner should returns on creation succeed, not
> after startJobManagerRunner
> -
>
> Key: FLINK-14434
> URL: https://issues.apache.org/jira/browse/FLINK-14434
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: patch.diff
>
>
> In an edge case, let's said
> 1) job finished nearly immediately
> 2) Dispatcher has been suspended in {{#startJobManagerRunner}} after
> {{jobManagerRunner.start();}} but before {{return jobManagerRunner;}}
> due to
> 1) we put {{jobManagerRunnerFutures}} with {{#startJobManagerRunner}}
> finished.
> 2) the creation of JobManagerRunner doesn't happen in MainThread.
> it is a possible execution order
> 1) JobManagerRunner created in akka-dispatcher thread
> 2) then apply {{Dispatcher#startJobManagerRunner}}
> 3) until {{jobManagerRunner.start();}} and before {{return jobManagerRunner;}}
> 4) this thread suspended
> 5) job finished, execute callback on MainThread
> 6) {{jobManagerRunnerFutures.get(jobID).getNow(null)}} returns {{null}}
> because akka-dispatcher thread doesn't {{return jobManagerRunner;}}
> 7) it report {{There is a newer JobManagerRunner for the job}} but actually
> not.
> **Solution**
> Two perspective but we can even have them both.
> 1. return {{jobManagerRunnerFuture}} in {{#createJobManagerRunner}}, let
> {{#startJobManagerRunner}} an action
> 2. on JobManagerRunner created, execute {{#startJobManagerRunner}} in
> MainThread.
> CC [~trohrmann]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535435111 ## CI report: * 1aad21713df6970025319183b3774aec674fefde : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129246538) * 7c3fd57d89d6dc77c3629758a95c4cbf94414d34 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132084942) * cfe6acdc0038aca92ff72cd201076eb07e78cd0d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132477620) * 6d323bf2657f8f82eac28d2f529fa1dbe36fcde6 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13195) Add create table support for SqlClient
[ https://issues.apache.org/jira/browse/FLINK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954315#comment-16954315 ] Zheng Hu commented on FLINK-13195: -- Talked with [~danny0405] offline, he is working on some other high-priority issues, I will pick up this task. Thanks. > Add create table support for SqlClient > -- > > Key: FLINK-13195 > URL: https://issues.apache.org/jira/browse/FLINK-13195 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.9.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14446) Design a Java end to end test framework for supporting more external system
[ https://issues.apache.org/jira/browse/FLINK-14446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14446. -- Resolution: Duplicate > Design a Java end to end test framework for supporting more external system > --- > > Key: FLINK-14446 > URL: https://issues.apache.org/jira/browse/FLINK-14446 > Project: Flink > Issue Type: Improvement >Reporter: Zheng Hu >Priority: Major > > I'm trying to design an end-to-end tests for hbase connector... see: > https://issues.apache.org/jira/projects/FLINK/issues/FLINK-14359 > Looked around the end-to-end test module, seems almost all of the tests are > written in bash scripts, it's hard to maintain & debug. In future, we will > support e2e test for more and more external system, so it will be better to > desgin a generic e2e java framework.. > Will try to provide the java framework, and moving the current bash e2e test > to the new framework step by step, and before all the work finish, we better > to not allow more e2e test get in. > Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14446) Design a Java end to end test framework for supporting more external system
Zheng Hu created FLINK-14446: Summary: Design a Java end to end test framework for supporting more external system Key: FLINK-14446 URL: https://issues.apache.org/jira/browse/FLINK-14446 Project: Flink Issue Type: Improvement Reporter: Zheng Hu I'm trying to design an end-to-end tests for hbase connector... see: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-14359 Looked around the end-to-end test module, seems almost all of the tests are written in bash scripts, it's hard to maintain & debug. In future, we will support e2e test for more and more external system, so it will be better to desgin a generic e2e java framework.. Will try to provide the java framework, and moving the current bash e2e test to the new framework step by step, and before all the work finish, we better to not allow more e2e test get in. Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535430586 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6d323bf2657f8f82eac28d2f529fa1dbe36fcde6 (Fri Oct 18 06:28:09 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. URL: https://github.com/apache/flink/pull/9910#issuecomment-542644991 ## CI report: * 8d1696b97872fdd9fd3aca725c8b58ad8227f6f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132137194) * bd54289054f8b4d144cdb7cdffb7f3d0dee274b4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325988) * 7df654afc7b6f8da2fdc6eee5a516b9bd789d0da : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/132481274) * 90c4d2d3c6859de464622c8934b92f007c71734a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132482718) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. URL: https://github.com/apache/flink/pull/9910#issuecomment-542638190 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d2493bf0f15605fd0ec1aee8afa2ba47dc4267a6 (Fri Oct 18 06:26:07 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime]
Let fullRestart metric count both full restarts and fine grained restarts
URL: https://github.com/apache/flink/pull/9778#discussion_r336335410
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphRestartTest.java
##
@@ -595,9 +594,6 @@ public void
slotPoolExecutionGraph_ConcurrentSchedulingAndAllocationFailure_Shou
@Test
public void testRestartWithEagerSchedulingAndSlotSharing() throws
Exception {
- // this test is inconclusive if not used with a proper
multi-threaded executor
Review comment:
The executor(futureExecutor) is not used for scheduling/failover anymore
since '[FLINK-11417] Make access to ExecutionGraph single threaded from
JobMaster main thread' is merged.
I think there is no need to specify the executor now and the tests does not
rely on that executor to do any thing.
The idea of testRestartWithEagerSchedulingAndSlotSharing is easy that it
tests a eager scheduling job can recover from a task failure.
The idea of testRestartWithSlotSharingAndNotEnoughResources is also easy
that the job will keep failing due to not enough resource until the restart
strategy suppresses the restarts.
So I think we can just remove the assumption and the executor specification.
Like in 6d323bf2657f8f82eac28d2f529fa1dbe36fcde6.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535430586 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6d323bf2657f8f82eac28d2f529fa1dbe36fcde6 (Fri Oct 18 06:18:59 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime]
Let fullRestart metric count both full restarts and fine grained restarts
URL: https://github.com/apache/flink/pull/9778#discussion_r336333505
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionGraph.java
##
@@ -734,15 +737,13 @@ public ErrorInfo getFailureInfo() {
}
/**
-* Gets the number of full restarts that the execution graph went
through.
-* If a full restart recovery is currently pending, this recovery is
included in the
-* count.
+* Gets the number of restarts, including full restarts and fine
grained restarts.
+* If a recovery is currently pending, this recovery is included in the
count.
*
-* @return The number of full restarts so far
+* @return The number restarts so far
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime]
Let fullRestart metric count both full restarts and fine grained restarts
URL: https://github.com/apache/flink/pull/9778#discussion_r336333479
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphRestartTest.java
##
@@ -690,7 +683,7 @@ public void
testRestartWithSlotSharingAndNotEnoughResources() throws Exception {
eg.scheduleForExecution();
// wait until no more changes happen
- while (eg.getNumberOfFullRestarts() < numRestarts) {
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535430586 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6d323bf2657f8f82eac28d2f529fa1dbe36fcde6 (Fri Oct 18 06:16:56 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535430586 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cfe6acdc0038aca92ff72cd201076eb07e78cd0d (Fri Oct 18 06:14:55 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime]
Let fullRestart metric count both full restarts and fine grained restarts
URL: https://github.com/apache/flink/pull/9778#discussion_r336332841
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphRestartTest.java
##
@@ -690,7 +683,7 @@ public void
testRestartWithSlotSharingAndNotEnoughResources() throws Exception {
eg.scheduleForExecution();
// wait until no more changes happen
- while (eg.getNumberOfFullRestarts() < numRestarts) {
Review comment:
You are right. This wait loop is not needed any more since the scheduling
and failover happens in the main thread.
Will replace it with an assertion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. URL: https://github.com/apache/flink/pull/9910#issuecomment-542644991 ## CI report: * 8d1696b97872fdd9fd3aca725c8b58ad8227f6f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132137194) * bd54289054f8b4d144cdb7cdffb7f3d0dee274b4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325988) * 7df654afc7b6f8da2fdc6eee5a516b9bd789d0da : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132481274) * 90c4d2d3c6859de464622c8934b92f007c71734a : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. URL: https://github.com/apache/flink/pull/9910#issuecomment-542644991 ## CI report: * 8d1696b97872fdd9fd3aca725c8b58ad8227f6f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132137194) * bd54289054f8b4d144cdb7cdffb7f3d0dee274b4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325988) * 7df654afc7b6f8da2fdc6eee5a516b9bd789d0da : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. URL: https://github.com/apache/flink/pull/9910#issuecomment-542638190 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 90c4d2d3c6859de464622c8934b92f007c71734a (Fri Oct 18 05:33:09 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test
bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add
hive e2e connector test
URL: https://github.com/apache/flink/pull/9919#discussion_r336322272
##
File path: flink-end-to-end-tests/flink-hive-test/pom.xml
##
@@ -0,0 +1,183 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+
+flink-end-to-end-tests
+org.apache.flink
+1.10-SNAPSHOT
+
+4.0.0
+
+flink-hive-test
+flink-hive-test
+jar
+
+
+
+org.apache.flink
+flink-table-common
+${project.version}
+provided
+
+
+org.apache.flink
+flink-table-api-java
+${project.version}
+provided
+
+
+org.apache.flink
+
flink-connector-hive_${scala.binary.version}
+${project.version}
+provided
+
+
+org.apache.flink
+
flink-table-planner-blink_${scala.binary.version}
+${project.version}
+provided
+
+
+
+
+
+org.apache.flink
+
flink-hadoop-compatibility_${scala.binary.version}
+${project.version}
+provided
+
+
+
+
+
+org.apache.flink
+flink-shaded-hadoop-2-uber
+2.7.5-8.0
+provided
+
+
+
+
+org.apache.hive
+hive-exec
+2.3.4
+
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+
+
+package
+
+shade
+
+
+SqlToolbox
+
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-dependency-plugin
+
+
+copy
+package
+
+copy
+
+
+
${project.build.directory}/sql-jars
+
+
+
+org.apache.flink
+flink-avro
+${project.version}
+sql-jar
+jar
+
+
+org.apache.flink
+flink-json
+${project.version}
+sql-jar
+jar
+
+
+org.apache.flink
+flink-csv
+${project.version}
+sql-jar
+jar
+
+
[GitHub] [flink] bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test
bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test URL: https://github.com/apache/flink/pull/9919#discussion_r336322138 ## File path: flink-end-to-end-tests/flink-hive-test/pom.xml ## @@ -0,0 +1,183 @@ + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + +flink-end-to-end-tests +org.apache.flink +1.10-SNAPSHOT + +4.0.0 + +flink-hive-test Review comment: shall we rename the module to `flink-connector-hive-test`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test
bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add
hive e2e connector test
URL: https://github.com/apache/flink/pull/9919#discussion_r336322467
##
File path:
flink-end-to-end-tests/flink-hive-test/src/main/java/org/apache/flink/connectors/hive/tests/HiveReadWriteDataExample.java
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive.tests;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.SqlDialect;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.catalog.hive.HiveCatalog;
+
+/**
+ * End to end test for HiveConnector.
+ */
+public class HiveReadWriteDataExample {
Review comment:
Is this an `Example` or `Test`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test
bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test URL: https://github.com/apache/flink/pull/9919#discussion_r336325463 ## File path: flink-end-to-end-tests/test-scripts/docker-hive-hadoop-cluster/config/mapred-site.xml ## @@ -0,0 +1,39 @@ + + + + + +mapreduce.framework.name Review comment: why do we need mapred? I thought we just need yarn and hdfs, right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test
bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add
hive e2e connector test
URL: https://github.com/apache/flink/pull/9919#discussion_r336325001
##
File path:
flink-end-to-end-tests/flink-hive-test/src/main/java/org/apache/flink/connectors/hive/tests/HiveReadWriteDataExample.java
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive.tests;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.SqlDialect;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.catalog.hive.HiveCatalog;
+
+/**
+ * End to end test for HiveConnector.
+ */
+public class HiveReadWriteDataExample {
+ public static void main(String[] args) throws Exception {
+ EnvironmentSettings settings =
EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build();
+ TableEnvironment tableEnv = TableEnvironment.create(settings);
+
tableEnv.getConfig().getConfiguration().setInteger(ExecutionConfigOptions.TABLE_EXEC_RESOURCE_DEFAULT_PARALLELISM.key(),
1);
+ tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
+
+ String catalogName = "hive";
+ String defaultDatabase = "default";
+ String hiveConfDir = "/usr/local/hive/conf";
+ String version = "2.3.4";
+ HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
defaultDatabase, hiveConfDir, version);
+
+ tableEnv.registerCatalog("hive", hiveCatalog);
+ tableEnv.sqlUpdate("insert into
hive.default.dest_non_partition_table " +
Review comment:
does `default` need back quotes as keyword?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add hive e2e connector test
bowenli86 commented on a change in pull request #9919: [FLINK-13303][hive]add
hive e2e connector test
URL: https://github.com/apache/flink/pull/9919#discussion_r336322335
##
File path: flink-end-to-end-tests/flink-hive-test/pom.xml
##
@@ -0,0 +1,183 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+
+flink-end-to-end-tests
+org.apache.flink
+1.10-SNAPSHOT
+
+4.0.0
+
+flink-hive-test
+flink-hive-test
+jar
+
+
+
+org.apache.flink
+flink-table-common
+${project.version}
+provided
+
+
+org.apache.flink
+flink-table-api-java
+${project.version}
+provided
+
+
+org.apache.flink
+
flink-connector-hive_${scala.binary.version}
+${project.version}
+provided
+
+
+org.apache.flink
+
flink-table-planner-blink_${scala.binary.version}
+${project.version}
+provided
+
+
+
+
+
+org.apache.flink
+
flink-hadoop-compatibility_${scala.binary.version}
+${project.version}
+provided
+
+
+
+
+
+org.apache.flink
+flink-shaded-hadoop-2-uber
+2.7.5-8.0
+provided
+
+
+
+
+org.apache.hive
+hive-exec
+2.3.4
+
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+
+
+package
+
+shade
+
+
+SqlToolbox
+
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-dependency-plugin
+
+
+copy
+package
+
+copy
+
+
+
${project.build.directory}/sql-jars
+
+
+
+org.apache.flink
+flink-avro
+${project.version}
+sql-jar
+jar
+
+
+org.apache.flink
+flink-json
+${project.version}
+sql-jar
+jar
+
+
+org.apache.flink
+flink-csv
+${project.version}
+sql-jar
+jar
+
+
+
+org.apache.flink
+
flink-sql-connector-kafka-0.10_${scala.binary.version}
Review comment:
why would we need all these kafka and elasticsearch jars?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535435111 ## CI report: * 1aad21713df6970025319183b3774aec674fefde : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129246538) * 7c3fd57d89d6dc77c3629758a95c4cbf94414d34 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132084942) * cfe6acdc0038aca92ff72cd201076eb07e78cd0d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132477620) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
flinkbot edited a comment on issue #9910: [FLINK-14405][runtime] Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. URL: https://github.com/apache/flink/pull/9910#issuecomment-542638190 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7df654afc7b6f8da2fdc6eee5a516b9bd789d0da (Fri Oct 18 05:13:45 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-14445) Python module build failed when making sdist
[
https://issues.apache.org/jira/browse/FLINK-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954253#comment-16954253
]
Dian Fu edited comment on FLINK-14445 at 10/18/19 5:08 AM:
---
[~yunta] Thanks for reporting this issue. It seems that it is caused by the
latest license change in FLINK-14008.
[~zhongwei] Could you help to take a look at this issue?
was (Author: dian.fu):
[~yunta] Thanks for reporting this issue. It seems that it is caused by the
latest license change in FLINK-14008.
[~zhongwei] Could you help to fix this issue?
> Python module build failed when making sdist
>
>
> Key: FLINK-14445
> URL: https://issues.apache.org/jira/browse/FLINK-14445
> Project: Flink
> Issue Type: Bug
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> From the description of error-log from building python module in travis, it
> seems invocation failed for {{sdist-make}} and then the phase of building
> python module exited.
> The instance log: https://api.travis-ci.com/v3/job/246710918/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9927: [FLINK-14397][hive] Failed to run Hive UDTF with array arguments
flinkbot edited a comment on issue #9927: [FLINK-14397][hive] Failed to run Hive UDTF with array arguments URL: https://github.com/apache/flink/pull/9927#issuecomment-543172555 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 14b59100e851fa7494838f1b5c68b4dc732da590 (Fri Oct 18 05:08:38 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535848177 ## CI report: * 5d9e3113e10bbf0220b4f8e080ecd1cddd9c9059 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129411044) * 55c766190d205798534f6d09170e66463c832082 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129544930) * 8c7b264e3d17dfa2eb2019ecb470cbe4ef9fa339 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/13191) * dd55f2f66aec3cbcee7211e2769496d0a5c873e2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131961829) * 69dbf0fa215fa7cb7aaadd596fcaadd3747697d2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132096181) * 16077687afe140b71bc91a99315c50219ac8c732 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132279941) * 0332dd139f1b2857efe5de3852cf4bb4573575a3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325960) * 7542ca7c45594de8ac6f7112e43d0a7bcd8615b6 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132476281) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535435111 ## CI report: * 1aad21713df6970025319183b3774aec674fefde : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129246538) * 7c3fd57d89d6dc77c3629758a95c4cbf94414d34 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132084942) * cfe6acdc0038aca92ff72cd201076eb07e78cd0d : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132477620) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9576: [FLINK-13915][ml] Add several base classes of summarizer.
flinkbot edited a comment on issue #9576: [FLINK-13915][ml] Add several base classes of summarizer. URL: https://github.com/apache/flink/pull/9576#issuecomment-526645449 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e04e263ec3c7819ed4fa3049e3584a6f1ec81c54 (Fri Oct 18 04:39:10 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13915).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add several base classes of summarizer.
walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add
several base classes of summarizer.
URL: https://github.com/apache/flink/pull/9576#discussion_r336317258
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/statistics/basicstatistic/BaseSummarizer.java
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.statistics.basicstatistic;
+
+import org.apache.flink.ml.common.linalg.DenseMatrix;
+
+import java.io.Serializable;
+
+/**
+ * Summarizer is the base class to calculate summary and store , and Summary
is the result of Summarizer.
+ * It will compute sum, squareSum = sum(x_i*x_i), min, max, normL1.
+ * All statistics can use summary to calculate from these statistics.
+ */
+public abstract class BaseSummarizer implements Serializable {
Review comment:
Also higher level question: Given this class is very basic, I was wondering
why we need this. If this is for interface sake. then better rework this as
interface and explain what are the APIs to be supproted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add several base classes of summarizer.
walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add several base classes of summarizer. URL: https://github.com/apache/flink/pull/9576#discussion_r336317216 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/statistics/basicstatistic/BaseSummarizer.java ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.statistics.basicstatistic; + +import org.apache.flink.ml.common.linalg.DenseMatrix; + +import java.io.Serializable; + +/** + * Summarizer is the base class to calculate summary and store , and Summary is the result of Summarizer. + * It will compute sum, squareSum = sum(x_i*x_i), min, max, normL1. Review comment: better to explain what the base class do. because this base class does not provide any of these functionalities. ``` Base Summarizer class holds basic stats (element count, outerproduct, etc) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add several base classes of summarizer.
walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add several base classes of summarizer. URL: https://github.com/apache/flink/pull/9576#discussion_r336317015 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/statistics/basicstatistic/BaseSummarizer.java ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.statistics.basicstatistic; + +import org.apache.flink.ml.common.linalg.DenseMatrix; + +import java.io.Serializable; + +/** + * Summarizer is the base class to calculate summary and store , and Summary is the result of Summarizer. Review comment: This needs more explanation. ``` Summarizer provides summary statistics about a given object (Matrix/Vector) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add several base classes of summarizer.
walterddr commented on a change in pull request #9576: [FLINK-13915][ml] Add several base classes of summarizer. URL: https://github.com/apache/flink/pull/9576#discussion_r336317421 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/statistics/basicstatistic/BaseSummarizer.java ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.statistics.basicstatistic; + +import org.apache.flink.ml.common.linalg.DenseMatrix; + +import java.io.Serializable; + +/** + * Summarizer is the base class to calculate summary and store , and Summary is the result of Summarizer. + * It will compute sum, squareSum = sum(x_i*x_i), min, max, normL1. Review comment: also. I don't see `normL1` anywhere in this diff. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14445) Python module build failed when making sdist
[
https://issues.apache.org/jira/browse/FLINK-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954253#comment-16954253
]
Dian Fu commented on FLINK-14445:
-
[~yunta] Thanks for reporting this issue. It seems that it is caused by the
latest license change in FLINK-14008.
[~zhongwei] Could you help to fix this issue?
> Python module build failed when making sdist
>
>
> Key: FLINK-14445
> URL: https://issues.apache.org/jira/browse/FLINK-14445
> Project: Flink
> Issue Type: Bug
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> From the description of error-log from building python module in travis, it
> seems invocation failed for {{sdist-make}} and then the phase of building
> python module exited.
> The instance log: https://api.travis-ci.com/v3/job/246710918/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.
flinkbot edited a comment on issue #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication. URL: https://github.com/apache/flink/pull/9732#issuecomment-533540408 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6b34d9ea1145c7677af262e738590e26de3d5222 (Fri Oct 18 04:26:55 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14153).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.
walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to
BLAS a method that performs DenseMatrix and SparseVector multiplication.
URL: https://github.com/apache/flink/pull/9732#discussion_r336316035
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/linalg/BLAS.java
##
@@ -146,4 +148,38 @@ public static void gemv(double alpha, DenseMatrix matA,
boolean transA,
final String ta = transA ? "T" : "N";
NATIVE_BLAS.dgemv(ta, m, n, alpha, matA.getData(), lda,
x.getData(), 1, beta, y.getData(), 1);
}
+
+ /**
+* y := alpha * A * x + beta * y .
+*/
+ public static void gemv(double alpha, DenseMatrix matA, boolean transA,
+ SparseVector x, double
beta, DenseVector y) {
+ if (transA) {
+ Preconditions.checkArgument(matA.numCols() == y.size()
&& matA.numRows() == x.size(),
+ "Matrix and vector size mismatched.");
+ } else {
+ Preconditions.checkArgument(matA.numRows() == y.size()
&& matA.numCols() == x.size(),
+ "Matrix and vector size mismatched.");
+ }
+ final int m = matA.numRows();
+ final int n = matA.numCols();
+ if (transA) {
+ int start = 0;
+ for (int i = 0; i < n; i++) {
+ double s = 0.;
+ for (int j = 0; j < x.indices.length; j++) {
+ s += x.values[j] * matA.data[start +
x.indices[j]];
+ }
+ y.data[i] = beta * y.data[i] + alpha * s;
+ start += m;
+ }
+ } else {
+ scal(beta, y);
+ for (int i = 0; i < x.indices.length; i++) {
+ int index = x.indices[i];
+ double value = alpha * x.values[i];
+ F2J_BLAS.daxpy(m, value, matA.data, index * m,
1, y.data, 0, 1);
Review comment:
I am not familiar with BLAS internal performance. is this faster than
directly coding it up ?
the reason why I am asking is that: there's two step involved. (`scal` and
`daxpy`). may have duplicate mem access ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.
flinkbot edited a comment on issue #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication. URL: https://github.com/apache/flink/pull/9732#issuecomment-533540408 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6b34d9ea1145c7677af262e738590e26de3d5222 (Fri Oct 18 04:22:50 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14153).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.
walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to
BLAS a method that performs DenseMatrix and SparseVector multiplication.
URL: https://github.com/apache/flink/pull/9732#discussion_r336315170
##
File path:
flink-ml-parent/flink-ml-lib/src/test/java/org/apache/flink/ml/common/linalg/BLASTest.java
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.linalg;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * The test cases for {@link BLAS}.
+ */
+public class BLASTest {
+ private static final double TOL = 1.0e-8;
+ private DenseMatrix mat = new DenseMatrix(2, 3, new double[]{1, 4, 2,
5, 3, 6});
+ private DenseVector dv1 = new DenseVector(new double[]{1, 2});
+ private DenseVector dv2 = new DenseVector(new double[]{1, 2, 3});
+ private SparseVector spv1 = new SparseVector(2, new int[]{0, 1}, new
double[]{1, 2});
+ private SparseVector spv2 = new SparseVector(3, new int[]{0, 2}, new
double[]{1, 3});
+
+ @Test
+ public void testGemvDense() throws Exception {
Review comment:
missing validator exception test cases:
1. invalid dimension
2. invalid dimension after transpose
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.
walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to
BLAS a method that performs DenseMatrix and SparseVector multiplication.
URL: https://github.com/apache/flink/pull/9732#discussion_r336314745
##
File path:
flink-ml-parent/flink-ml-lib/src/test/java/org/apache/flink/ml/common/linalg/BLASTest.java
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.linalg;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * The test cases for {@link BLAS}.
+ */
+public class BLASTest {
+ private static final double TOL = 1.0e-8;
+ private DenseMatrix mat = new DenseMatrix(2, 3, new double[]{1, 4, 2,
5, 3, 6});
+ private DenseVector dv1 = new DenseVector(new double[]{1, 2});
+ private DenseVector dv2 = new DenseVector(new double[]{1, 2, 3});
+ private SparseVector spv1 = new SparseVector(2, new int[]{0, 1}, new
double[]{1, 2});
+ private SparseVector spv2 = new SparseVector(3, new int[]{0, 2}, new
double[]{1, 3});
+
+ @Test
+ public void testGemvDense() throws Exception {
Review comment:
why is the dense version not tested until now?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.
walterddr commented on a change in pull request #9732: [FLINK-14153][ml] Add to
BLAS a method that performs DenseMatrix and SparseVector multiplication.
URL: https://github.com/apache/flink/pull/9732#discussion_r336313420
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/linalg/BLAS.java
##
@@ -146,4 +148,38 @@ public static void gemv(double alpha, DenseMatrix matA,
boolean transA,
final String ta = transA ? "T" : "N";
NATIVE_BLAS.dgemv(ta, m, n, alpha, matA.getData(), lda,
x.getData(), 1, beta, y.getData(), 1);
}
+
+ /**
+* y := alpha * A * x + beta * y .
+*/
+ public static void gemv(double alpha, DenseMatrix matA, boolean transA,
+ SparseVector x, double
beta, DenseVector y) {
+ if (transA) {
Review comment:
Can we create a `transposePreconditionChecker`? feel like this would be use
multiple place/times with duplicate code in such as `gemv`, `gemm`, etc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535848177 ## CI report: * 5d9e3113e10bbf0220b4f8e080ecd1cddd9c9059 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129411044) * 55c766190d205798534f6d09170e66463c832082 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129544930) * 8c7b264e3d17dfa2eb2019ecb470cbe4ef9fa339 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/13191) * dd55f2f66aec3cbcee7211e2769496d0a5c873e2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131961829) * 69dbf0fa215fa7cb7aaadd596fcaadd3747697d2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132096181) * 16077687afe140b71bc91a99315c50219ac8c732 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132279941) * 0332dd139f1b2857efe5de3852cf4bb4573575a3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325960) * 7542ca7c45594de8ac6f7112e43d0a7bcd8615b6 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132476281) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535435111 ## CI report: * 1aad21713df6970025319183b3774aec674fefde : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129246538) * 7c3fd57d89d6dc77c3629758a95c4cbf94414d34 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132084942) * cfe6acdc0038aca92ff72cd201076eb07e78cd0d : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535430586 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cfe6acdc0038aca92ff72cd201076eb07e78cd0d (Fri Oct 18 04:05:30 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
zhuzhurk commented on a change in pull request #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#discussion_r336312977 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionGraph.java ## @@ -264,6 +266,9 @@ @Nullable private InternalTaskFailuresListener internalTaskFailuresListener; + /** Counts all restarts. Used by other Gauges/Meters and does not register to metric group. */ + private final Counter numberOfRestartsCounter = new SimpleCounter(); Review comment: This is mainly for future changes to add a meter metric "numberOfRestarts" with the ctor `MeterView(Counter counter, int timeSpanInSeconds)`. So that we can create that metric differently for legacy and ng schedulers. Here's a POC for that 43095f17ac770a3d7308e8ba5a4f3d6256fd4298. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts
flinkbot edited a comment on issue #9778: [FLINK-14206][runtime] Let fullRestart metric count both full restarts and fine grained restarts URL: https://github.com/apache/flink/pull/9778#issuecomment-535430586 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cfe6acdc0038aca92ff72cd201076eb07e78cd0d (Fri Oct 18 04:00:25 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF URL: https://github.com/apache/flink/pull/9890#issuecomment-541418126 ## CI report: * ec7f1a70fd420d25326e49ac1f4de878d4aae481 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131691044) * 47706d03f178962d0e0fabc4b4a51d0810be5e91 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/131693072) * 9546adc32393cf7a777a963133a134ea2f5f4fd6 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132096206) * ffac224585e0d5001dae23ae1778ecbffc61a7b6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132271229) * baa2faf79c63f5bbfc5b9648779cc0c006045b25 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132329883) * 6d89c775e67489dc56d41184c717655c63f5b417 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132471768) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering
flinkbot edited a comment on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering URL: https://github.com/apache/flink/pull/9853#issuecomment-539472188 ## CI report: * b10f5603fc55999adb8d5334a3d142d20322de5b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130941094) * bd7553b9d922e246f7b9b7bafe0d8428a9be6a99 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/131514700) * 272536f7c0a89d283e76c187ca29cca2b0951318 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132093266) * 9c32b0c5e3c5b4c55cbfce6f8ecb3135d2bd1cb5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132284738) * 5183203896305067056e68e8dddc3b413e66788b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132312087) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF URL: https://github.com/apache/flink/pull/9890#issuecomment-541416833 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6d89c775e67489dc56d41184c717655c63f5b417 (Fri Oct 18 03:56:21 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF URL: https://github.com/apache/flink/pull/9890#discussion_r336296251 ## File path: flink-python/pyflink/table/table_environment.py ## @@ -47,10 +47,17 @@ class TableEnvironment(object): __metaclass__ = ABCMeta -def __init__(self, j_tenv, serializer=PickleSerializer()): +def __init__(self, j_tenv, need_judge_planner=False, serializer=PickleSerializer()): Review comment: This is confusing. Why not passing whether it's blink planner directly? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF URL: https://github.com/apache/flink/pull/9890#discussion_r336297346 ## File path: flink-python/pyflink/table/udf.py ## @@ -167,13 +166,25 @@ def _create_judf(self): j_input_types = utils.to_jarray(gateway.jvm.TypeInformation, [_to_java_type(i) for i in self._input_types]) j_result_type = _to_java_type(self._result_type) -return gateway.jvm.org.apache.flink.table.util.python.PythonTableUtils \ -.createPythonScalarFunction(self._name, -bytearray(serialized_func), -j_input_types, -j_result_type, -self._deterministic, -_get_python_env()) +if is_blink_planner: Review comment: What about change it as following? ``` if is_blink_planner: PythonTableUtils = gateway.jvm.org.apache.flink.table.planner.utils.python.PythonTableUtils else: PythonTableUtils= gateway.jvm.PythonTableUtils j_scalar_function = gateway.jvm.PythonTableUtils \ .createPythonScalarFunction(self._name, bytearray(serialized_func), j_input_types, j_result_type, self._deterministic, _get_python_env()) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
dianfu commented on a change in pull request #9890:
[FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
URL: https://github.com/apache/flink/pull/9890#discussion_r336298507
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/PythonFunctionCodeGenerator.scala
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.planner.codegen
+
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.table.functions.python.{PythonEnv, PythonFunction}
+import org.apache.flink.table.functions.{FunctionLanguage, ScalarFunction,
UserDefinedFunction}
+import org.apache.flink.table.planner.codegen.CodeGenUtils.{newName,
primitiveDefaultValue, primitiveTypeTermForType}
+import org.apache.flink.table.planner.codegen.Indenter.toISC
+import org.apache.flink.table.runtime.generated.GeneratedFunction
+import org.apache.flink.table.runtime.types.TypeInfoLogicalTypeConverter
+import org.apache.flink.table.utils.EncodingUtils
+
+/**
+ * A code generator for generating Python [[UserDefinedFunction]]s.
+ */
+object PythonFunctionCodeGenerator {
+
+ private val PYTHON_SCALAR_FUNCTION_NAME = "PythonScalarFunction"
+
+ /**
+* Generates a [[ScalarFunction]] for the specified Python user-defined
function.
+*
+* @param name class name of the user-defined function. Must be a valid
Java class identifier
Review comment:
Change the comment of name to the following as it's not used for the class
name of generated ScalarFunction?
```
@param name name of the user-defined function
```
Should also update the comment in PythonFunctionCodeGenerator in the legacy
planner.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF URL: https://github.com/apache/flink/pull/9890#discussion_r336297753 ## File path: flink-python/pyflink/testing/test_case_utils.py ## @@ -149,6 +149,32 @@ def collect(self, table): return string_result +class PyFlinkBlinkStreamTableTestCase(PyFlinkTestCase): +""" +Base class for stream unit tests of blink planner. Review comment: What about remove the word `unit` as it could also be used in IT cases? Should also update the comments in other places such as PyFlinkStreamTableTestCase, PyFlinkBatchTableTestCase, PyFlinkBlinkBatchTableTestCase, etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
dianfu commented on a change in pull request #9890:
[FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
URL: https://github.com/apache/flink/pull/9890#discussion_r336300424
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/PythonFunctionCodeGenerator.scala
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.planner.codegen
+
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.table.functions.python.{PythonEnv, PythonFunction}
+import org.apache.flink.table.functions.{FunctionLanguage, ScalarFunction,
UserDefinedFunction}
+import org.apache.flink.table.planner.codegen.CodeGenUtils.{newName,
primitiveDefaultValue, primitiveTypeTermForType}
+import org.apache.flink.table.planner.codegen.Indenter.toISC
+import org.apache.flink.table.runtime.generated.GeneratedFunction
+import org.apache.flink.table.runtime.types.TypeInfoLogicalTypeConverter
+import org.apache.flink.table.utils.EncodingUtils
+
+/**
+ * A code generator for generating Python [[UserDefinedFunction]]s.
+ */
+object PythonFunctionCodeGenerator {
+
+ private val PYTHON_SCALAR_FUNCTION_NAME = "PythonScalarFunction"
+
+ /**
+* Generates a [[ScalarFunction]] for the specified Python user-defined
function.
+*
+* @param name class name of the user-defined function. Must be a valid
Java class identifier
+* @param serializedScalarFunction serialized Python scalar function
+* @param inputTypes input data types
+* @param resultType expected result type
+* @param deterministic the determinism of the function's results
+* @param pythonEnv the Python execution environment
+* @return instance of generated ScalarFunction
+*/
+ def generateScalarFunction(
+ name: String,
+ serializedScalarFunction: Array[Byte],
+ inputTypes: Array[TypeInformation[_]],
+ resultType: TypeInformation[_],
+ deterministic: Boolean,
+ pythonEnv: PythonEnv): ScalarFunction = {
+val funcName = newName(PYTHON_SCALAR_FUNCTION_NAME)
+val resultLogicType =
TypeInfoLogicalTypeConverter.fromTypeInfoToLogicalType(resultType)
+val resultTypeTerm = primitiveTypeTermForType(resultLogicType)
+val defaultResultValue = primitiveDefaultValue(resultLogicType)
+val inputParamCode = inputTypes.zipWithIndex.map { case (inputType, index)
=>
+ s"${primitiveTypeTermForType(
+TypeInfoLogicalTypeConverter.fromTypeInfoToLogicalType(inputType))}
in$index"
+}.mkString(", ")
+
+val encodingUtilsTypeTerm = classOf[EncodingUtils].getCanonicalName
+val typeInfoTypeTerm = classOf[TypeInformation[_]].getCanonicalName
+val inputTypesCode =
inputTypes.map(EncodingUtils.encodeObjectToString).map { inputType =>
+ s"""
+ |($typeInfoTypeTerm) $encodingUtilsTypeTerm.decodeStringToObject(
+ | "$inputType", $typeInfoTypeTerm.class)
+ |""".stripMargin
+}.mkString(", ")
+
+val encodedResultType = EncodingUtils.encodeObjectToString(resultType)
+val encodedScalarFunction =
EncodingUtils.encodeBytesToBase64(serializedScalarFunction)
+val encodedPythonEnv = EncodingUtils.encodeObjectToString(pythonEnv)
+val pythonEnvTypeTerm = classOf[PythonEnv].getCanonicalName
+
+val funcCode = j"""
+ |public class $funcName extends
${classOf[ScalarFunction].getCanonicalName}
+ | implements ${classOf[PythonFunction].getCanonicalName} {
+ |
+ | private static final long serialVersionUID = 1L;
+ |
+ | public $resultTypeTerm eval($inputParamCode) {
+ |return $defaultResultValue;
+ | }
+ |
+ | @Override
+ | public $typeInfoTypeTerm[] getParameterTypes(Class[] signature) {
+ |return new $typeInfoTypeTerm[]{$inputTypesCode};
+ | }
+ |
+ | @Override
+ | public $typeInfoTypeTerm getResultType(Class[] signature) {
+ |return ($typeInfoTypeTerm)
$encodingUtilsTypeTerm.decodeStringToObject(
+ | "$encodedResultType", $typeInfoTypeTerm.class);
+ | }
+ |
+ | @Override
+ | public ${classOf[Fun
[GitHub] [flink] dianfu commented on a change in pull request #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
dianfu commented on a change in pull request #9890:
[FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
URL: https://github.com/apache/flink/pull/9890#discussion_r336311539
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/rules/logical/PythonScalarFunctionSplitRuleTest.scala
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.logical
+
+import org.apache.calcite.plan.hep.HepMatchOrder
+import org.apache.flink.api.scala._
+import org.apache.flink.table.api.scala._
+import org.apache.flink.table.planner.plan.nodes.FlinkConventions
+import org.apache.flink.table.planner.plan.optimize.program._
+import org.apache.flink.table.planner.plan.rules.{FlinkBatchRuleSets,
FlinkStreamRuleSets}
+import
org.apache.flink.table.planner.runtime.utils.JavaUserDefinedScalarFunctions.{BooleanPythonScalarFunction,
PythonScalarFunction}
+import org.apache.flink.table.planner.utils.TableTestBase
+import org.junit.{Before, Test}
+
+/**
+ * Test for [[PythonScalarFunctionSplitRule]].
+ */
+class PythonScalarFunctionSplitRuleTest extends TableTestBase {
+
+ private val util = batchTestUtil()
+
+ @Before
+ def setup(): Unit = {
+val programs = new FlinkChainedProgram[BatchOptimizeContext]()
+programs.addLast(
+ "table_ref",
Review comment:
remove this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535848177 ## CI report: * 5d9e3113e10bbf0220b4f8e080ecd1cddd9c9059 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129411044) * 55c766190d205798534f6d09170e66463c832082 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129544930) * 8c7b264e3d17dfa2eb2019ecb470cbe4ef9fa339 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/13191) * dd55f2f66aec3cbcee7211e2769496d0a5c873e2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131961829) * 69dbf0fa215fa7cb7aaadd596fcaadd3747697d2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132096181) * 16077687afe140b71bc91a99315c50219ac8c732 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132279941) * 0332dd139f1b2857efe5de3852cf4bb4573575a3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132325960) * 7542ca7c45594de8ac6f7112e43d0a7bcd8615b6 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14308) Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954238#comment-16954238
]
Yun Tang commented on FLINK-14308:
--
Another instance: https://api.travis-ci.com/v3/job/246710922/log.txt
> Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator
> fails on Travis
> -
>
> Key: FLINK-14308
> URL: https://issues.apache.org/jira/browse/FLINK-14308
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The
> {{Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator}}
> fails on Travis with
> {code}
> Test
> testOneToOneAtLeastOnceCustomOperator(org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase)
> failed with:
> java.lang.AssertionError: Expected to contain all of: <[0]>, but was: <[]>
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.assertAtLeastOnceForTopic(KafkaTestBase.java:235)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnce(KafkaProducerTestBase.java:289)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator(KafkaProducerTestBase.java:214)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> https://api.travis-ci.com/v3/job/240747188/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage.
flinkbot edited a comment on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage. URL: https://github.com/apache/flink/pull/9894#issuecomment-541555428 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9c05be8a14dbf6b31fd24f6114be36e1adec3b3a (Fri Oct 18 03:51:16 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong edited a comment on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage.
wuchong edited a comment on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage. URL: https://github.com/apache/flink/pull/9894#issuecomment-543482276 Hi @hequn8128 , the part of blink looks good to me. I only left a comment about the naming of `containsFunctionOf` which a is minor problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage.
wuchong commented on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage. URL: https://github.com/apache/flink/pull/9894#issuecomment-543482276 Hi @hequn8128 , the part of blink looks good to me. I only left a comment about the naming of `containsFunctionOf` which is minor problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage.
wuchong commented on a change in pull request #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage. URL: https://github.com/apache/flink/pull/9894#discussion_r336310564 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/ExpressionReducer.scala ## @@ -68,7 +68,7 @@ class ExpressionReducer( // Skip expressions that contain python functions because it's quite expensive to // call Python UDFs during optimization phase. They will be optimized during the runtime. - case (_, e) if PythonUtil.containsFunctionOf(e, FunctionLanguage.PYTHON) => + case (_, e) if PythonUtil.containsFunctionOf(e, findPythonFunction = true) => Review comment: I prefer `containsPythonFunction()` and `containsNonPythonFunction`, because it's more clear than `findPythonFunction = false` flag. I think "contains" already means any node matched in the tree. Meanwhile, I think `recursive` is also not a very clear flag. So I would suggest following utils: ``` boolean containsPythonCall(RexNode) boolean containsNonPythonCall(RexNode) boolean isPythonCall(RexNode) boolean isNonPythonCall(RexNode) ``` This also follows `RexUtil.isConstant(RexNode)` and `RexUtil.containsFieldAccess(RexNode)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage.
wuchong commented on a change in pull request #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage. URL: https://github.com/apache/flink/pull/9894#discussion_r336310564 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/ExpressionReducer.scala ## @@ -68,7 +68,7 @@ class ExpressionReducer( // Skip expressions that contain python functions because it's quite expensive to // call Python UDFs during optimization phase. They will be optimized during the runtime. - case (_, e) if PythonUtil.containsFunctionOf(e, FunctionLanguage.PYTHON) => + case (_, e) if PythonUtil.containsFunctionOf(e, findPythonFunction = true) => Review comment: I prefer `containsPythonFunction()` and `containsNonPythonFunction`, because it's more clear than `findPythonFunction = false` flag. Meanwhile, I think `recursive` is also not a very clear flag. I think "contains" already means any node matched in the tree. So I would suggest following utils: ``` boolean containsPythonCall(RexNode) boolean containsNonPythonCall(RexNode) boolean isPythonCall(RexNode) boolean isNonPythonCall(RexNode) ``` This also follows `RexUtil.isConstant(RexNode)` and `RexUtil.containsFieldAccess(RexNode)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage.
flinkbot edited a comment on issue #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage. URL: https://github.com/apache/flink/pull/9894#issuecomment-541555428 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9c05be8a14dbf6b31fd24f6114be36e1adec3b3a (Fri Oct 18 03:48:14 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage.
wuchong commented on a change in pull request #9894: [FLINK-14342][table][python] Remove method FunctionDefinition#getLanguage. URL: https://github.com/apache/flink/pull/9894#discussion_r336310564 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/ExpressionReducer.scala ## @@ -68,7 +68,7 @@ class ExpressionReducer( // Skip expressions that contain python functions because it's quite expensive to // call Python UDFs during optimization phase. They will be optimized during the runtime. - case (_, e) if PythonUtil.containsFunctionOf(e, FunctionLanguage.PYTHON) => + case (_, e) if PythonUtil.containsFunctionOf(e, findPythonFunction = true) => Review comment: I prefer `containsPythonFunction()` and `containsNonPythonFunction`, because it's more clear than `findPythonFunction = false` flag. Meanwhile, I think `recursive` is also not a very clear flag. So I would suggest following utils: ``` boolean containsPythonCall(RexNode) boolean containsNonPythonCall(RexNode) boolean isPythonCall(RexNode) boolean isNonPythonCall(RexNode) ``` This also follows `RexUtil.isConstant(RexNode)` and `RexUtil.containsFieldAccess(RexNode)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-14445) Python module build failed when making sdist
[
https://issues.apache.org/jira/browse/FLINK-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954232#comment-16954232
]
Yun Tang edited comment on FLINK-14445 at 10/18/19 3:45 AM:
CC [~dian.fu]
was (Author: yunta):
CC [~dianfu]
> Python module build failed when making sdist
>
>
> Key: FLINK-14445
> URL: https://issues.apache.org/jira/browse/FLINK-14445
> Project: Flink
> Issue Type: Bug
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> From the description of error-log from building python module in travis, it
> seems invocation failed for {{sdist-make}} and then the phase of building
> python module exited.
> The instance log: https://api.travis-ci.com/v3/job/246710918/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14445) Python module build failed when making sdist
[
https://issues.apache.org/jira/browse/FLINK-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954232#comment-16954232
]
Yun Tang commented on FLINK-14445:
--
CC [~dianfu]
> Python module build failed when making sdist
>
>
> Key: FLINK-14445
> URL: https://issues.apache.org/jira/browse/FLINK-14445
> Project: Flink
> Issue Type: Bug
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> From the description of error-log from building python module in travis, it
> seems invocation failed for {{sdist-make}} and then the phase of building
> python module exited.
> The instance log: https://api.travis-ci.com/v3/job/246710918/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14237) No need to rename shipped Flink jar
[
https://issues.apache.org/jira/browse/FLINK-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-14237.
-
Resolution: Fixed
master via b022a26b34d7ada92fa459e62f7ec84ce85cfe6f
> No need to rename shipped Flink jar
> ---
>
> Key: FLINK-14237
> URL: https://issues.apache.org/jira/browse/FLINK-14237
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, when we ship Flink jar configured by -yj, we always rename it as
> {{flink.jar}}. It seems a redundant operation since we can always use the
> exact name of the real jar. It also causes some confusion to our users who
> should not be required to know about Flink internal implementation that they
> configure a specific Flink jar(said {{flink-private-version-suffix.jar}}) but
> cannot find it on YARN container, because it is now {{flink.jar}}.
> CC [~trohrmann]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar
flinkbot edited a comment on issue #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar URL: https://github.com/apache/flink/pull/9861#issuecomment-539888511 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 99adbf906d40ff2a8808f395def14a01084f24c5 (Fri Oct 18 03:39:05 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar
flinkbot edited a comment on issue #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar URL: https://github.com/apache/flink/pull/9861#issuecomment-539888511 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 99adbf906d40ff2a8808f395def14a01084f24c5 (Fri Oct 18 03:37:02 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun closed pull request #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar
TisonKun closed pull request #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar URL: https://github.com/apache/flink/pull/9861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on issue #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar
TisonKun commented on issue #9861: [FLINK-14237][yarn] No need to rename shipped Flink jar URL: https://github.com/apache/flink/pull/9861#issuecomment-543477490 Thanks for your reminder Till. I've verified it locally with `-yj /path/to/flink-dist_2.11-1.10-SNAPSHOT.jar` and check TM log with > INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Classpath: :lib/flink-table-blink_2.11-1.10-SNAPSHOT.jar:lib/flink-table_2.11-1.10-SNAPSHOT.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:plugins:flink-dist_2.11-1.10-SNAPSHOT.jar:flink-conf.yaml::/data/tdbank/hadoop/hadoop-2.7.4/etc/hadoop:/data/tdbank/hadoop/hadoop-2.7.4/share/hadoop/common/hadoop-common-2.7.4-tests.jar:/data/tdbank/hadoop/hadoop-2.7.4/share/hadoop/common/hadoop-common-2.7.4.jar:/data/tdbank/hadoop/hadoop-2.7.4/share/hadoop/common/hadoop-nfs-2.7.4.jar:... Merging... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-14445) Python module build failed when making sdist
Yun Tang created FLINK-14445:
Summary: Python module build failed when making sdist
Key: FLINK-14445
URL: https://issues.apache.org/jira/browse/FLINK-14445
Project: Flink
Issue Type: Bug
Reporter: Yun Tang
Fix For: 1.10.0
>From the description of error-log from building python module in travis, it
>seems invocation failed for {{sdist-make}} and then the phase of building
>python module exited.
The instance log: https://api.travis-ci.com/v3/job/246710918/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535844880 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7542ca7c45594de8ac6f7112e43d0a7bcd8615b6 (Fri Oct 18 03:26:50 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF URL: https://github.com/apache/flink/pull/9890#issuecomment-541418126 ## CI report: * ec7f1a70fd420d25326e49ac1f4de878d4aae481 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131691044) * 47706d03f178962d0e0fabc4b4a51d0810be5e91 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/131693072) * 9546adc32393cf7a777a963133a134ea2f5f4fd6 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132096206) * ffac224585e0d5001dae23ae1778ecbffc61a7b6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132271229) * baa2faf79c63f5bbfc5b9648779cc0c006045b25 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132329883) * 6d89c775e67489dc56d41184c717655c63f5b417 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132471768) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF
flinkbot edited a comment on issue #9890: [FLINK-14272][python][table-planner-blink] Support Blink planner for Python UDF URL: https://github.com/apache/flink/pull/9890#issuecomment-541418126 ## CI report: * ec7f1a70fd420d25326e49ac1f4de878d4aae481 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/131691044) * 47706d03f178962d0e0fabc4b4a51d0810be5e91 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/131693072) * 9546adc32393cf7a777a963133a134ea2f5f4fd6 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132096206) * ffac224585e0d5001dae23ae1778ecbffc61a7b6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132271229) * baa2faf79c63f5bbfc5b9648779cc0c006045b25 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132329883) * 6d89c775e67489dc56d41184c717655c63f5b417 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering
flinkbot edited a comment on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering URL: https://github.com/apache/flink/pull/9853#issuecomment-539472188 ## CI report: * b10f5603fc55999adb8d5334a3d142d20322de5b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130941094) * bd7553b9d922e246f7b9b7bafe0d8428a9be6a99 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/131514700) * 272536f7c0a89d283e76c187ca29cca2b0951318 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132093266) * 9c32b0c5e3c5b4c55cbfce6f8ecb3135d2bd1cb5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132284738) * 5183203896305067056e68e8dddc3b413e66788b : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/132312087) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9930: [FLINK-12848][table] Fix invalid row type caching
flinkbot edited a comment on issue #9930: [FLINK-12848][table] Fix invalid row type caching URL: https://github.com/apache/flink/pull/9930#issuecomment-543227145 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3198ddb653a8a16fd6d79619bf37a11580d570c5 (Fri Oct 18 02:41:01 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954207#comment-16954207 ] lichong commented on FLINK-12574: - [~yitz589] I think it's all right for the flink design about this problem, but I want to know how do you resolve this when you resume your application or reprocess old data from MQ such as kafka. In this case we need to consume the data in kafka from a give offset instead of the offset in the checkpoint or savepoint. Any reply will be appreciated. Thx. > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #8737: [FLINK-12848][core] Consider fieldNames in RowTypeInfo#equals()
flinkbot edited a comment on issue #8737: [FLINK-12848][core] Consider fieldNames in RowTypeInfo#equals() URL: https://github.com/apache/flink/pull/8737#issuecomment-502016582 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 82e3196dd4b8d42023647006caacb88f30e294fa (Fri Oct 18 02:34:54 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #8737: [FLINK-12848][core] Consider fieldNames in RowTypeInfo#equals()
wuchong commented on issue #8737: [FLINK-12848][core] Consider fieldNames in RowTypeInfo#equals() URL: https://github.com/apache/flink/pull/8737#issuecomment-543458523 We will quick fix in `FlinkTypeFactory`, see #9930 We will fix `RowTypeInfo.equals` in FLINK-14438 in the future. Thereby close this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong closed pull request #8737: [FLINK-12848][core] Consider fieldNames in RowTypeInfo#equals()
wuchong closed pull request #8737: [FLINK-12848][core] Consider fieldNames in RowTypeInfo#equals() URL: https://github.com/apache/flink/pull/8737 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-9953) Active Kubernetes integration
[
https://issues.apache.org/jira/browse/FLINK-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954206#comment-16954206
]
Yang Wang edited comment on FLINK-9953 at 10/18/19 2:32 AM:
I have made an new complete implementation based on the [ML
discussion|[http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html]]
and [Implementation
doc|[https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing].]
Some key changes are as follows.
* Fabric8 flink kubernetes client implementation
* Use deployment for flink master
* Dynamically allocation for TaskManager pod
* Support submission from cli by FlinkKubernetesCustomCli
* Use configmap to save log4j and flink-conf.yaml
* Support cluster id, nodeport and load banlancer
* Support both perjob and session mode
As more and more users are interested in this feature and want to have a taste.
I hope to take over this JIRA and push the integration faster. [~trohrmann],
could you please re-assign this ticket to me, including the 1-11 subtasks? I
will create another umbrella JIRA for phase2 implementation. It is mainly
focused on the production optimization, including k8s native high-availability,
storage, network, log collector and etc.
was (Author: fly_in_gis):
I have made an new complete implementation based on the [ML
discussion|[http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html]]
and [Implementation
doc|[https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing].]
Some key changes are as follows.
* Fabric8 flink kubernetes client implementation
* Use deployment for flink master
* Dynamically allocation for TaskManager pod
* Support submission from cli by FlinkKubernetesCustomCli
* Use configmap to save log4j and flink-conf.yaml
* Support cluster id, nodeport and load banlancer
* Support both perjob and session mode
As more and more users are interested in this feature and want to have a taste.
I hope to take over this JIRA and push the integration faster. [~trohrmann],
could be re-assign this ticket to me, including the 1-11 subtasks? I will
create another umbrella JIRA for phase2 implementation. It is mainly focused on
the production optimization, including k8s native high-availability, storage,
network, log collector and etc.
> Active Kubernetes integration
> -
>
> Key: FLINK-9953
> URL: https://issues.apache.org/jira/browse/FLINK-9953
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Coordination
>Reporter: Till Rohrmann
>Assignee: Chunhui Shi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is the umbrella issue tracking Flink's active Kubernetes integration.
> Active means in this context that the {{ResourceManager}} can talk to
> Kubernetes to launch new pods similar to Flink's Yarn and Mesos integration.
>
> Document can be found here:
> [https://docs.google.com/document/d/1Zmhui_29VASPcBOEqyMWnF3L6WEWZ4kedrCqya0WaAk/edit?usp=sharing]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-9953) Active Kubernetes integration
[
https://issues.apache.org/jira/browse/FLINK-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954206#comment-16954206
]
Yang Wang commented on FLINK-9953:
--
I have made an new complete implementation based on the [ML
discussion|[http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html]]
and [Implementation
doc|[https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing].]
Some key changes are as follows.
* Fabric8 flink kubernetes client implementation
* Use deployment for flink master
* Dynamically allocation for TaskManager pod
* Support submission from cli by FlinkKubernetesCustomCli
* Use configmap to save log4j and flink-conf.yaml
* Support cluster id, nodeport and load banlancer
* Support both perjob and session mode
As more and more users are interested in this feature and want to have a taste.
I hope to take over this JIRA and push the integration faster. [~trohrmann],
could be re-assign this ticket to me, including the 1-11 subtasks? I will
create another umbrella JIRA for phase2 implementation. It is mainly focused on
the production optimization, including k8s native high-availability, storage,
network, log collector and etc.
> Active Kubernetes integration
> -
>
> Key: FLINK-9953
> URL: https://issues.apache.org/jira/browse/FLINK-9953
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Coordination
>Reporter: Till Rohrmann
>Assignee: Chunhui Shi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is the umbrella issue tracking Flink's active Kubernetes integration.
> Active means in this context that the {{ResourceManager}} can talk to
> Kubernetes to launch new pods similar to Flink's Yarn and Mesos integration.
>
> Document can be found here:
> [https://docs.google.com/document/d/1Zmhui_29VASPcBOEqyMWnF3L6WEWZ4kedrCqya0WaAk/edit?usp=sharing]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14428) Non-consistency key access in KeyedProcessFunction when use keyed state api in both processElement and onTimer method
[
https://issues.apache.org/jira/browse/FLINK-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954205#comment-16954205
]
vinoyang commented on FLINK-14428:
--
I read the source code again, you are right. It's not a problem.
> Non-consistency key access in KeyedProcessFunction when use keyed state api
> in both processElement and onTimer method
> -
>
> Key: FLINK-14428
> URL: https://issues.apache.org/jira/browse/FLINK-14428
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: vinoyang
>Priority: Major
>
> Scenario:
> In {{KeyedProcessFunction}}, uses keyed state API in both {{processElement}}
> and {{onTimer}} method may cause non-consistency key access.
> Analysis:
> For timer, in {{InternalTimerServiceImpl}}, the key context is set to the key
> which comes from timer when calling registerXXXTimeTimer:
> {code:java}
> public void onProcessingTime(long time) throws Exception {
> // null out the timer in case the Triggerable calls
> registerProcessingTimeTimer()
> // inside the callback.
> nextTimer = null;
> InternalTimer timer;
> while ((timer = processingTimeTimersQueue.peek()) != null &&
> timer.getTimestamp() <= time) {
> processingTimeTimersQueue.poll();
> keyContext.setCurrentKey(timer.getKey());//here
> triggerTarget.onProcessingTime(timer);
> }
> if (timer != null && nextTimer == null) {
> nextTimer =
> processingTimeService.registerTimer(timer.getTimestamp(), this);
> }
> }
> {code}
> For processElement method, in {{OneInputStreamTask}} it is called after
> seting key context:
> {code:java}
> @Override
> public void emitRecord(StreamRecord record) throws
> Exception {
> synchronized (lock) {
> numRecordsIn.inc();
> operator.setKeyContextElement1(record);
> //here
> operator.processElement(record);
> }
> }
> {code}
> The setCurrentKey method in the first code snippet and the
> setKeyContextElement1 method in the second code snippet are point to the same
> {{AbstractStreamOperator#setCurrentKey}} method. However, they are in the
> different thread and there is only one keyed State Backend instance. And
> {{AbstractStreamOperator#setCurrentKey}} will change the current key of keyed
> state backend.
> So if we access keyed state API in both {{processElement}} and {{onTimer}},
> we may get error state value, because one of these methods may change the key
> and cause non-consistency problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-14428) Non-consistency key access in KeyedProcessFunction when use keyed state api in both processElement and onTimer method
[
https://issues.apache.org/jira/browse/FLINK-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vinoyang updated FLINK-14428:
-
Comment: was deleted
(was: [~aljoscha] It seems after 1.9+ we introduced {{MailboxProcessor}} it is
not a problem. Before Flink 1.9(I am watching 1.8.2), {{processInput}} is
invoked in task thread and {{onProcessingTime}} is invoked in time service
thread, right?)
> Non-consistency key access in KeyedProcessFunction when use keyed state api
> in both processElement and onTimer method
> -
>
> Key: FLINK-14428
> URL: https://issues.apache.org/jira/browse/FLINK-14428
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: vinoyang
>Priority: Major
>
> Scenario:
> In {{KeyedProcessFunction}}, uses keyed state API in both {{processElement}}
> and {{onTimer}} method may cause non-consistency key access.
> Analysis:
> For timer, in {{InternalTimerServiceImpl}}, the key context is set to the key
> which comes from timer when calling registerXXXTimeTimer:
> {code:java}
> public void onProcessingTime(long time) throws Exception {
> // null out the timer in case the Triggerable calls
> registerProcessingTimeTimer()
> // inside the callback.
> nextTimer = null;
> InternalTimer timer;
> while ((timer = processingTimeTimersQueue.peek()) != null &&
> timer.getTimestamp() <= time) {
> processingTimeTimersQueue.poll();
> keyContext.setCurrentKey(timer.getKey());//here
> triggerTarget.onProcessingTime(timer);
> }
> if (timer != null && nextTimer == null) {
> nextTimer =
> processingTimeService.registerTimer(timer.getTimestamp(), this);
> }
> }
> {code}
> For processElement method, in {{OneInputStreamTask}} it is called after
> seting key context:
> {code:java}
> @Override
> public void emitRecord(StreamRecord record) throws
> Exception {
> synchronized (lock) {
> numRecordsIn.inc();
> operator.setKeyContextElement1(record);
> //here
> operator.processElement(record);
> }
> }
> {code}
> The setCurrentKey method in the first code snippet and the
> setKeyContextElement1 method in the second code snippet are point to the same
> {{AbstractStreamOperator#setCurrentKey}} method. However, they are in the
> different thread and there is only one keyed State Backend instance. And
> {{AbstractStreamOperator#setCurrentKey}} will change the current key of keyed
> state backend.
> So if we access keyed state API in both {{processElement}} and {{onTimer}},
> we may get error state value, because one of these methods may change the key
> and cause non-consistency problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9931: [FLINK-14441][Table-Common] Fix ValueLiteralExpression#getValueAs when valueClass is Period.class
flinkbot edited a comment on issue #9931: [FLINK-14441][Table-Common] Fix ValueLiteralExpression#getValueAs when valueClass is Period.class URL: https://github.com/apache/flink/pull/9931#issuecomment-543240888 ## CI report: * a91235b01f24114bd0634f15dfaedc04a280b250 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132381590) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9916: [FLINK-14408][Table-Planner]UDF's open method is invoked to initialize when sql is optimized in oldPlanner
flinkbot edited a comment on issue #9916: [FLINK-14408][Table-Planner]UDF's open method is invoked to initialize when sql is optimized in oldPlanner URL: https://github.com/apache/flink/pull/9916#issuecomment-542764294 ## CI report: * cd122b399a8dc12c3c604d7db6dea012ee7c0c7f : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132188836) * 926952d0d3159aa788269aa3e51ce5e0a33088d2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/132275809) * 339ccf7b1e6193336bf67ea896270eda033edb76 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132339131) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering
flinkbot edited a comment on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering URL: https://github.com/apache/flink/pull/9853#issuecomment-539467530 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5183203896305067056e68e8dddc3b413e66788b (Fri Oct 18 02:16:35 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❗ 3. Needs [attention] from. - Needs attention by @pnowojski [committer] * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] ifndef-SleePy commented on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering
ifndef-SleePy commented on issue #9853: [FLINK-13904][checkpointing] Avoid competition of checkpoint triggering URL: https://github.com/apache/flink/pull/9853#issuecomment-543453584 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
