[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 7da11c60c656bfab79cf3ae76bc56cc729ae24a6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29857) * 154b36442660ab9898943a3c3d34a2591023f312 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18407: [FLINK-25669][runtime] Support register operator coordinators for newly initialized ExecutionJobVertex
flinkbot edited a comment on pull request #18407: URL: https://github.com/apache/flink/pull/18407#issuecomment-1017107067 ## CI report: * 1716622b272f2141ee4c29cf978265d8de00fa1b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29752) * c2f62c24ad91bd123e6e5797ea117220c28d5499 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29996) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * 36e5148ffb71691d7c9798b51fb52e88f1927748 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29514) * a4acd8e2b6d14724243eddfdd22caa8bf11890e5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18126: [FLINK-25036][runtime] Introduce vertex wise scheduling strategy
flinkbot edited a comment on pull request #18126: URL: https://github.com/apache/flink/pull/18126#issuecomment-995489268 ## CI report: * 8a780946fc7b1f8657f46dde23f23f642776efa2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29975) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25649) Scheduling jobs fails with org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
[
https://issues.apache.org/jira/browse/FLINK-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480881#comment-17480881
]
Gil De Grove edited comment on FLINK-25649 at 1/24/22, 7:54 AM:
Hello [~trohrmann], [~zhuzh]
Thanks for the reply, due to the answer, I decided to went for a full
anonymization, we removed all data that felt confidential, and replaced then
with a tag, if you feel that those logs contains important information, we can
get the, from the raw source.
To note;
- we removed the apache.wire, apache.http.headers and aws.requests and aws
S3client that wre quite vorbose from the logs to try to keep only the one
interesting for the issue at hand.
- The ID of the jobs are still present in the logs.
- We are using json formated logs, I kep them in that format in the logs.
[^flink_scheduler_deadlock.json.zip]
was (Author: JIRAUSER283553):
Hello [~trohrmann], [~zhuzh]
Thanks for the reply, due to the answer, I decided to went for a full
anonymization, we removed all data that felt confidential, and replaced then
with a tag, if you feel that those logs contains important information, we can
get the, from the raw source.
To note;
- we removed the apache.wire, apache.http.headers and aws.requests and aws
S3client that wre quite vorbose from the logs to try to keep only the one
interesting for the issue at hand.
- The ID of the jobs are still present in the logs.
- We are using json formated logs, I kep them in that format in the logs.
> Scheduling jobs fails with
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
> -
>
> Key: FLINK-25649
> URL: https://issues.apache.org/jira/browse/FLINK-25649
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.1
>Reporter: Gil De Grove
>Priority: Major
> Attachments: flink_scheduler_deadlock.json.zip
>
>
> Following comment from Till on this [SO
> question|https://stackoverflow.com/questions/70683048/scheduling-jobs-fails-with-org-apache-flink-runtime-jobmanager-scheduler-noresou?noredirect=1#comment124980546_70683048]
> h2. *Summary*
> We are currently experiencing a scheduling issue with our flink cluster.
> The symptoms are that some/most/all (it depend, the symptoms are not always
> the same) of our tasks are showed as _SCHEDULED_ but fail after a timeout.
> The jobs are them showed a _RUNNING_
> The failing exception is the following one:
> {{Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout}}
> After analysis, we assume (we cannot prove it, as there are not that much
> logs for that part of the code) that the failure is due to a deadlock/race
> condition that is happening when several jobs are being submitted at the same
> time to the flink cluster, even though we have enough slots available in the
> cluster.
> We actually have the error with 52 available task slots, and have 12 jobs
> that are not scheduled.
> h2. Additional information
> * Flink version: 1.13.1 commit a7f3192
> * Flink cluster in session mode
> * 2 Job managers using k8s HA mode (resource requests: 2 CPU, 4Gb Ram,
> limits sets on memory to 4Gb)
> * 50 task managers with 2 slots each (resource requests: 2 CPUs, 2GB Ram. No
> limits set).
> * Our Flink cluster is shut down every night, and restarted every morning.
> The error seems to occur when a lot of jobs needs to be scheduled. The jobs
> are configured to restore their state, and we do not see any issues for jobs
> that are being scheduled and run correctly, it seems to really be related to
> a scheduling issue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25649) Scheduling jobs fails with org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
[
https://issues.apache.org/jira/browse/FLINK-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480881#comment-17480881
]
Gil De Grove commented on FLINK-25649:
--
Hello [~trohrmann], [~zhuzh]
Thanks for the reply, due to the answer, I decided to went for a full
anonymization, we removed all data that felt confidential, and replaced then
with a tag, if you feel that those logs contains important information, we can
get the, from the raw source.
To note;
- we removed the apache.wire, apache.http.headers and aws.requests and aws
S3client that wre quite vorbose from the logs to try to keep only the one
interesting for the issue at hand.
- The ID of the jobs are still present in the logs.
- We are using json formated logs, I kep them in that format in the logs.
> Scheduling jobs fails with
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
> -
>
> Key: FLINK-25649
> URL: https://issues.apache.org/jira/browse/FLINK-25649
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.1
>Reporter: Gil De Grove
>Priority: Major
> Attachments: flink_scheduler_deadlock.json.zip
>
>
> Following comment from Till on this [SO
> question|https://stackoverflow.com/questions/70683048/scheduling-jobs-fails-with-org-apache-flink-runtime-jobmanager-scheduler-noresou?noredirect=1#comment124980546_70683048]
> h2. *Summary*
> We are currently experiencing a scheduling issue with our flink cluster.
> The symptoms are that some/most/all (it depend, the symptoms are not always
> the same) of our tasks are showed as _SCHEDULED_ but fail after a timeout.
> The jobs are them showed a _RUNNING_
> The failing exception is the following one:
> {{Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout}}
> After analysis, we assume (we cannot prove it, as there are not that much
> logs for that part of the code) that the failure is due to a deadlock/race
> condition that is happening when several jobs are being submitted at the same
> time to the flink cluster, even though we have enough slots available in the
> cluster.
> We actually have the error with 52 available task slots, and have 12 jobs
> that are not scheduled.
> h2. Additional information
> * Flink version: 1.13.1 commit a7f3192
> * Flink cluster in session mode
> * 2 Job managers using k8s HA mode (resource requests: 2 CPU, 4Gb Ram,
> limits sets on memory to 4Gb)
> * 50 task managers with 2 slots each (resource requests: 2 CPUs, 2GB Ram. No
> limits set).
> * Our Flink cluster is shut down every night, and restarted every morning.
> The error seems to occur when a lot of jobs needs to be scheduled. The jobs
> are configured to restore their state, and we do not see any issues for jobs
> that are being scheduled and run correctly, it seems to really be related to
> a scheduling issue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25649) Scheduling jobs fails with org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
[
https://issues.apache.org/jira/browse/FLINK-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gil De Grove updated FLINK-25649:
-
Attachment: flink_scheduler_deadlock.json.zip
> Scheduling jobs fails with
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
> -
>
> Key: FLINK-25649
> URL: https://issues.apache.org/jira/browse/FLINK-25649
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.1
>Reporter: Gil De Grove
>Priority: Major
> Attachments: flink_scheduler_deadlock.json.zip
>
>
> Following comment from Till on this [SO
> question|https://stackoverflow.com/questions/70683048/scheduling-jobs-fails-with-org-apache-flink-runtime-jobmanager-scheduler-noresou?noredirect=1#comment124980546_70683048]
> h2. *Summary*
> We are currently experiencing a scheduling issue with our flink cluster.
> The symptoms are that some/most/all (it depend, the symptoms are not always
> the same) of our tasks are showed as _SCHEDULED_ but fail after a timeout.
> The jobs are them showed a _RUNNING_
> The failing exception is the following one:
> {{Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout}}
> After analysis, we assume (we cannot prove it, as there are not that much
> logs for that part of the code) that the failure is due to a deadlock/race
> condition that is happening when several jobs are being submitted at the same
> time to the flink cluster, even though we have enough slots available in the
> cluster.
> We actually have the error with 52 available task slots, and have 12 jobs
> that are not scheduled.
> h2. Additional information
> * Flink version: 1.13.1 commit a7f3192
> * Flink cluster in session mode
> * 2 Job managers using k8s HA mode (resource requests: 2 CPU, 4Gb Ram,
> limits sets on memory to 4Gb)
> * 50 task managers with 2 slots each (resource requests: 2 CPUs, 2GB Ram. No
> limits set).
> * Our Flink cluster is shut down every night, and restarted every morning.
> The error seems to occur when a lot of jobs needs to be scheduled. The jobs
> are configured to restore their state, and we do not see any issues for jobs
> that are being scheduled and run correctly, it seems to really be related to
> a scheduling issue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zjureel commented on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
zjureel commented on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1019809836 Thanks @KarmaGYZ I have updated the codes and add a config option `jobstore.flush-to-disk` in `FileExecutionGraphInfoStore` instead of a memory store -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-22578) deployment configuration page miss content navigate menu
[ https://issues.apache.org/jira/browse/FLINK-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang closed FLINK-22578. Fix Version/s: (was: 1.15.0) (was: 1.13.6) Resolution: Implemented > deployment configuration page miss content navigate menu > > > Key: FLINK-22578 > URL: https://issues.apache.org/jira/browse/FLINK-22578 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.13.0 >Reporter: 谢波 >Priority: Minor > Labels: auto-deprioritized-major > > [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/] > The page miss content navigate menu cuase Inconvenient to use. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (FLINK-22578) deployment configuration page miss content navigate menu
[ https://issues.apache.org/jira/browse/FLINK-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang reopened FLINK-22578: -- > deployment configuration page miss content navigate menu > > > Key: FLINK-22578 > URL: https://issues.apache.org/jira/browse/FLINK-22578 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.13.0 >Reporter: 谢波 >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.13.6 > > > [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/] > The page miss content navigate menu cuase Inconvenient to use. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18407: [FLINK-25669][runtime] Support register operator coordinators for newly initialized ExecutionJobVertex
flinkbot edited a comment on pull request #18407: URL: https://github.com/apache/flink/pull/18407#issuecomment-1017107067 ## CI report: * 1716622b272f2141ee4c29cf978265d8de00fa1b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29752) * c2f62c24ad91bd123e6e5797ea117220c28d5499 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * 36e5148ffb71691d7c9798b51fb52e88f1927748 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29514) * 93ceba0641baa96ad792e7c10240e83373aff3d4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18459: fix syntax errors in joins.md
flinkbot edited a comment on pull request #18459: URL: https://github.com/apache/flink/pull/18459#issuecomment-1019802598 ## CI report: * 0c3df235f0c927490dd0d4917d39092f175e7fd0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29994) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
flinkbot edited a comment on pull request #18324: URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905 ## CI report: * b05c9a39758867364fb4d00a709696785a3f054f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25769) Add internal functions and basic function versioning
Timo Walther created FLINK-25769:
Summary: Add internal functions and basic function versioning
Key: FLINK-25769
URL: https://issues.apache.org/jira/browse/FLINK-25769
Project: Flink
Issue Type: Sub-task
Components: Table SQL / API, Table SQL / Planner
Reporter: Timo Walther
Assignee: Timo Walther
As mentioned in FLIP-190:
We keep the old function definition around under an internal name. The physical
plan after restore might look similar to:
{code}
== Optimized Physical Plan ==
Calc(select=[$TO_TIMESTAMP_LTZ$1(f0)])
{code}
{{$TO_TIMESTAMP_LTZ$1}} has been returned by the ModuleManager.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-18356) Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480871#comment-17480871
]
Yun Gao edited comment on FLINK-18356 at 1/24/22, 7:40 AM:
---
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=10064
|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=10064]
It's also after FLINK-25681 (8c8e658636de1874b2abcf28827f34043bc49aff)
was (Author: gaoyunhaii):
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=10064
> Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18459: fix syntax errors in joins.md
flinkbot commented on pull request #18459: URL: https://github.com/apache/flink/pull/18459#issuecomment-1019802598 ## CI report: * 0c3df235f0c927490dd0d4917d39092f175e7fd0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18459: fix syntax errors in joins.md
flinkbot commented on pull request #18459: URL: https://github.com/apache/flink/pull/18459#issuecomment-1019801901 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0c3df235f0c927490dd0d4917d39092f175e7fd0 (Mon Jan 24 07:39:18 UTC 2022) **Warnings:** * Documentation files were touched, but no `docs/content.zh/` files: Update Chinese documentation or file Jira ticket. * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18356) Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480871#comment-17480871
]
Yun Gao commented on FLINK-18356:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=10064
> Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25758) GCS Filesystem implementation fails on Java 11 tests due to licensing issues
[
https://issues.apache.org/jira/browse/FLINK-25758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480870#comment-17480870
]
Yun Gao commented on FLINK-25758:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=946871de-358d-5815-3994-8175615bc253=e0240c62-4570-5d1c-51af-dd63d2093da1=26699]
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=e9d3d34f-3d15-59f4-0e3e-35067d100dfe=a7382ec4-87d2-5a9d-7c53-a2f93e317458=27949]
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=6e8542d7-de38-5a33-4aca-458d6c87066d=dffc2faa-5b48-5b4e-0797-dec1b1f74872=27949]
> GCS Filesystem implementation fails on Java 11 tests due to licensing issues
>
>
> Key: FLINK-25758
> URL: https://issues.apache.org/jira/browse/FLINK-25758
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
>Affects Versions: 1.15.0
>Reporter: Martijn Visser
>Assignee: Galen Warren
>Priority: Blocker
> Labels: pull-request-available
>
> {code}
> 00:33:45,410 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker
> [] - Dependency io.netty:netty-common:4.1.51.Final is mentioned in NOTICE
> file /__w/2/s/flink-python/src/main/resources/META-INF/NOTICE, but was not
> mentioned by the build output as a bundled dependency
> 00:33:45,411 ERROR org.apache.flink.tools.ci.licensecheck.NoticeFileChecker
> [] - Could not find dependency javax.annotation:javax.annotation-api:1.3.2
> in NOTICE file
> /__w/2/s/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
> 00:33:45,536 INFO org.apache.flink.tools.ci.licensecheck.JarFileChecker
> [] - Checking directory /tmp/flink-validation-deployment with a total of
> 197 jar files.
> 00:34:18,554 ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker
> [] - File '/javax/annotation/security/package.html' in jar
> '/tmp/flink-validation-deployment/org/apache/flink/flink-gs-fs-hadoop/1.15-SNAPSHOT/flink-gs-fs-hadoop-1.15-20220122.001944-1.jar'
> contains match with forbidden regex 'gnu ?\R?[\s/#]*general
> ?\R?[\s/#]*public ?\R?[\s/#]*license'.
> 00:34:18,555 ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker
> [] - File '/javax/annotation/package.html' in jar
> '/tmp/flink-validation-deployment/org/apache/flink/flink-gs-fs-hadoop/1.15-SNAPSHOT/flink-gs-fs-hadoop-1.15-20220122.001944-1.jar'
> contains match with forbidden regex 'gnu ?\R?[\s/#]*general
> ?\R?[\s/#]*public ?\R?[\s/#]*license'.
> 00:35:46,612 WARN org.apache.flink.tools.ci.licensecheck.LicenseChecker
> [] - Found a total of 3 severe license issues
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29932=logs=946871de-358d-5815-3994-8175615bc253=e0240c62-4570-5d1c-51af-dd63d2093da1
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] JacobAP opened a new pull request #18459: fix syntax errors in joins.md
JacobAP opened a new pull request #18459: URL: https://github.com/apache/flink/pull/18459 flink/docs/content/docs/dev/table/sql/queries/joins.md " update_time TIMESTAMP(3) METADATA FROM `values.source.timestamp` VIRTUAL " change to " update_time TIMESTAMP(3) METADATA FROM `value.source.timestamp` VIRTUAL " ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 6a4a6c064380db3219fb92805e9a87712bb85fa2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29971) * 237027aabd1c103dbdd722e67f1f2a8475149c48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29978) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * a7afbb3176d79338140da86934d9b8cafdb5c1c7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29972) * 6441c9641faa7398a9c1d77f5f3482b397a79690 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29982) * 7dcb2f8df322c2fb454f0e8aa0ef95d7bb696120 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29988) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18454: [hotfix][connector/pulsar] Fix typo in JavaDocs example.
flinkbot edited a comment on pull request #18454: URL: https://github.com/apache/flink/pull/18454#issuecomment-1019661681 ## CI report: * dc8bef8341a40b7c13a8c72a9fd9f70dfd1899a1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29970) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25768) Python test BlinkBatchTableEnvironmentTests.test_explain_with_multi_sinks failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25768:
Component/s: Table SQL / Planner
> Python test BlinkBatchTableEnvironmentTests.test_explain_with_multi_sinks
> failed on azure
> -
>
> Key: FLINK-25768
> URL: https://issues.apache.org/jira/browse/FLINK-25768
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Table SQL / Planner
>Affects Versions: 1.13.5
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> 2022-01-22T01:51:13.0242386Z Jan 22 01:51:13 answer = 'xro24635'
> 2022-01-22T01:51:13.0242938Z Jan 22 01:51:13 gateway_client =
>
> 2022-01-22T01:51:13.0243909Z Jan 22 01:51:13 target_id = 'o24634', name =
> 'addInsertSql'
> 2022-01-22T01:51:13.0244309Z Jan 22 01:51:13
> 2022-01-22T01:51:13.0244761Z Jan 22 01:51:13 def get_return_value(answer,
> gateway_client, target_id=None, name=None):
> 2022-01-22T01:51:13.0245397Z Jan 22 01:51:13 """Converts an answer
> received from the Java gateway into a Python object.
> 2022-01-22T01:51:13.0245923Z Jan 22 01:51:13
> 2022-01-22T01:51:13.0246348Z Jan 22 01:51:13 For example, string
> representation of integers are converted to Python
> 2022-01-22T01:51:13.0246963Z Jan 22 01:51:13 integer, string
> representation of objects are converted to JavaObject
> 2022-01-22T01:51:13.0247486Z Jan 22 01:51:13 instances, etc.
> 2022-01-22T01:51:13.0247820Z Jan 22 01:51:13
> 2022-01-22T01:51:13.0248220Z Jan 22 01:51:13 :param answer: the
> string returned by the Java gateway
> 2022-01-22T01:51:13.0248846Z Jan 22 01:51:13 :param gateway_client:
> the gateway client used to communicate with the Java
> 2022-01-22T01:51:13.0249505Z Jan 22 01:51:13 Gateway. Only
> necessary if the answer is a reference (e.g., object,
> 2022-01-22T01:51:13.0249945Z Jan 22 01:51:13 list, map)
> 2022-01-22T01:51:13.0250470Z Jan 22 01:51:13 :param target_id: the
> name of the object from which the answer comes from
> 2022-01-22T01:51:13.0251084Z Jan 22 01:51:13 (e.g., *object1* in
> `object1.hello()`). Optional.
> 2022-01-22T01:51:13.0251607Z Jan 22 01:51:13 :param name: the name of
> the member from which the answer comes from
> 2022-01-22T01:51:13.0252199Z Jan 22 01:51:13 (e.g., *hello* in
> `object1.hello()`). Optional.
> 2022-01-22T01:51:13.0252646Z Jan 22 01:51:13 """
> 2022-01-22T01:51:13.0253198Z Jan 22 01:51:13 if is_error(answer)[0]:
> 2022-01-22T01:51:13.0253684Z Jan 22 01:51:13 if len(answer) > 1:
> 2022-01-22T01:51:13.0254169Z Jan 22 01:51:13 type = answer[1]
> 2022-01-22T01:51:13.0254757Z Jan 22 01:51:13 value =
> OUTPUT_CONVERTER[type](answer[2:], gateway_client)
> 2022-01-22T01:51:13.0255450Z Jan 22 01:51:13 if answer[1] ==
> REFERENCE_TYPE:
> 2022-01-22T01:51:13.0256085Z Jan 22 01:51:13 > raise
> Py4JJavaError(
> 2022-01-22T01:51:13.0256768Z Jan 22 01:51:13 "An
> error occurred while calling {0}{1}{2}.\n".
> 2022-01-22T01:51:13.0257432Z Jan 22 01:51:13
> format(target_id, ".", name), value)
> 2022-01-22T01:51:13.0258250Z Jan 22 01:51:13 E
> py4j.protocol.Py4JJavaError: An error occurred while calling
> o24634.addInsertSql.
> 2022-01-22T01:51:13.0259174Z Jan 22 01:51:13 E :
> java.lang.NullPointerException
> 2022-01-22T01:51:13.0259824Z Jan 22 01:51:13 Eat
> java.util.Objects.requireNonNull(Objects.java:203)
> 2022-01-22T01:51:13.0260748Z Jan 22 01:51:13 Eat
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:144)
> 2022-01-22T01:51:13.0261604Z Jan 22 01:51:13 Eat
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:108)
> 2022-01-22T01:51:13.0262653Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.(FlinkRelMetadataQuery.java:78)
> 2022-01-22T01:51:13.0263927Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.instance(FlinkRelMetadataQuery.java:59)
> 2022-01-22T01:51:13.0264864Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:39)
> 2022-01-22T01:51:13.0266075Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:38)
> 2022-01-22T01:51:13.0267017Z Jan
[jira] [Created] (FLINK-25768) Python test BlinkBatchTableEnvironmentTests.test_explain_with_multi_sinks failed on azure
Yun Gao created FLINK-25768:
---
Summary: Python test
BlinkBatchTableEnvironmentTests.test_explain_with_multi_sinks failed on azure
Key: FLINK-25768
URL: https://issues.apache.org/jira/browse/FLINK-25768
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.13.5
Reporter: Yun Gao
{code:java}
2022-01-22T01:51:13.0242386Z Jan 22 01:51:13 answer = 'xro24635'
2022-01-22T01:51:13.0242938Z Jan 22 01:51:13 gateway_client =
2022-01-22T01:51:13.0243909Z Jan 22 01:51:13 target_id = 'o24634', name =
'addInsertSql'
2022-01-22T01:51:13.0244309Z Jan 22 01:51:13
2022-01-22T01:51:13.0244761Z Jan 22 01:51:13 def get_return_value(answer,
gateway_client, target_id=None, name=None):
2022-01-22T01:51:13.0245397Z Jan 22 01:51:13 """Converts an answer
received from the Java gateway into a Python object.
2022-01-22T01:51:13.0245923Z Jan 22 01:51:13
2022-01-22T01:51:13.0246348Z Jan 22 01:51:13 For example, string
representation of integers are converted to Python
2022-01-22T01:51:13.0246963Z Jan 22 01:51:13 integer, string
representation of objects are converted to JavaObject
2022-01-22T01:51:13.0247486Z Jan 22 01:51:13 instances, etc.
2022-01-22T01:51:13.0247820Z Jan 22 01:51:13
2022-01-22T01:51:13.0248220Z Jan 22 01:51:13 :param answer: the string
returned by the Java gateway
2022-01-22T01:51:13.0248846Z Jan 22 01:51:13 :param gateway_client: the
gateway client used to communicate with the Java
2022-01-22T01:51:13.0249505Z Jan 22 01:51:13 Gateway. Only
necessary if the answer is a reference (e.g., object,
2022-01-22T01:51:13.0249945Z Jan 22 01:51:13 list, map)
2022-01-22T01:51:13.0250470Z Jan 22 01:51:13 :param target_id: the name
of the object from which the answer comes from
2022-01-22T01:51:13.0251084Z Jan 22 01:51:13 (e.g., *object1* in
`object1.hello()`). Optional.
2022-01-22T01:51:13.0251607Z Jan 22 01:51:13 :param name: the name of
the member from which the answer comes from
2022-01-22T01:51:13.0252199Z Jan 22 01:51:13 (e.g., *hello* in
`object1.hello()`). Optional.
2022-01-22T01:51:13.0252646Z Jan 22 01:51:13 """
2022-01-22T01:51:13.0253198Z Jan 22 01:51:13 if is_error(answer)[0]:
2022-01-22T01:51:13.0253684Z Jan 22 01:51:13 if len(answer) > 1:
2022-01-22T01:51:13.0254169Z Jan 22 01:51:13 type = answer[1]
2022-01-22T01:51:13.0254757Z Jan 22 01:51:13 value =
OUTPUT_CONVERTER[type](answer[2:], gateway_client)
2022-01-22T01:51:13.0255450Z Jan 22 01:51:13 if answer[1] ==
REFERENCE_TYPE:
2022-01-22T01:51:13.0256085Z Jan 22 01:51:13 > raise
Py4JJavaError(
2022-01-22T01:51:13.0256768Z Jan 22 01:51:13 "An error
occurred while calling {0}{1}{2}.\n".
2022-01-22T01:51:13.0257432Z Jan 22 01:51:13
format(target_id, ".", name), value)
2022-01-22T01:51:13.0258250Z Jan 22 01:51:13 E
py4j.protocol.Py4JJavaError: An error occurred while calling
o24634.addInsertSql.
2022-01-22T01:51:13.0259174Z Jan 22 01:51:13 E :
java.lang.NullPointerException
2022-01-22T01:51:13.0259824Z Jan 22 01:51:13 E at
java.util.Objects.requireNonNull(Objects.java:203)
2022-01-22T01:51:13.0260748Z Jan 22 01:51:13 E at
org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:144)
2022-01-22T01:51:13.0261604Z Jan 22 01:51:13 E at
org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:108)
2022-01-22T01:51:13.0262653Z Jan 22 01:51:13 E at
org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.(FlinkRelMetadataQuery.java:78)
2022-01-22T01:51:13.0263927Z Jan 22 01:51:13 E at
org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.instance(FlinkRelMetadataQuery.java:59)
2022-01-22T01:51:13.0264864Z Jan 22 01:51:13 E at
org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:39)
2022-01-22T01:51:13.0266075Z Jan 22 01:51:13 E at
org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:38)
2022-01-22T01:51:13.0267017Z Jan 22 01:51:13 E at
org.apache.calcite.plan.RelOptCluster.getMetadataQuery(RelOptCluster.java:178)
2022-01-22T01:51:13.0267991Z Jan 22 01:51:13 E at
org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:108)
2022-01-22T01:51:13.0268849Z Jan 22 01:51:13 E at
org.apache.calcite.rel.core.RelFactories$FilterFactoryImpl.createFilter(RelFactories.java:344)
[GitHub] [flink] flinkbot edited a comment on pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
flinkbot edited a comment on pull request #18376: URL: https://github.com/apache/flink/pull/18376#issuecomment-1014266348 ## CI report: * a415c9901fe5addb7e042b2ea760b2de95c736ca Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29529) * 369e3ea1432e9b033e5660642ab0f86b03ab4d79 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29992) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25768) Python test BlinkBatchTableEnvironmentTests.test_explain_with_multi_sinks failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480861#comment-17480861
]
Yun Gao commented on FLINK-25768:
-
cc [~hxbks2ks] ~
> Python test BlinkBatchTableEnvironmentTests.test_explain_with_multi_sinks
> failed on azure
> -
>
> Key: FLINK-25768
> URL: https://issues.apache.org/jira/browse/FLINK-25768
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.5
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> 2022-01-22T01:51:13.0242386Z Jan 22 01:51:13 answer = 'xro24635'
> 2022-01-22T01:51:13.0242938Z Jan 22 01:51:13 gateway_client =
>
> 2022-01-22T01:51:13.0243909Z Jan 22 01:51:13 target_id = 'o24634', name =
> 'addInsertSql'
> 2022-01-22T01:51:13.0244309Z Jan 22 01:51:13
> 2022-01-22T01:51:13.0244761Z Jan 22 01:51:13 def get_return_value(answer,
> gateway_client, target_id=None, name=None):
> 2022-01-22T01:51:13.0245397Z Jan 22 01:51:13 """Converts an answer
> received from the Java gateway into a Python object.
> 2022-01-22T01:51:13.0245923Z Jan 22 01:51:13
> 2022-01-22T01:51:13.0246348Z Jan 22 01:51:13 For example, string
> representation of integers are converted to Python
> 2022-01-22T01:51:13.0246963Z Jan 22 01:51:13 integer, string
> representation of objects are converted to JavaObject
> 2022-01-22T01:51:13.0247486Z Jan 22 01:51:13 instances, etc.
> 2022-01-22T01:51:13.0247820Z Jan 22 01:51:13
> 2022-01-22T01:51:13.0248220Z Jan 22 01:51:13 :param answer: the
> string returned by the Java gateway
> 2022-01-22T01:51:13.0248846Z Jan 22 01:51:13 :param gateway_client:
> the gateway client used to communicate with the Java
> 2022-01-22T01:51:13.0249505Z Jan 22 01:51:13 Gateway. Only
> necessary if the answer is a reference (e.g., object,
> 2022-01-22T01:51:13.0249945Z Jan 22 01:51:13 list, map)
> 2022-01-22T01:51:13.0250470Z Jan 22 01:51:13 :param target_id: the
> name of the object from which the answer comes from
> 2022-01-22T01:51:13.0251084Z Jan 22 01:51:13 (e.g., *object1* in
> `object1.hello()`). Optional.
> 2022-01-22T01:51:13.0251607Z Jan 22 01:51:13 :param name: the name of
> the member from which the answer comes from
> 2022-01-22T01:51:13.0252199Z Jan 22 01:51:13 (e.g., *hello* in
> `object1.hello()`). Optional.
> 2022-01-22T01:51:13.0252646Z Jan 22 01:51:13 """
> 2022-01-22T01:51:13.0253198Z Jan 22 01:51:13 if is_error(answer)[0]:
> 2022-01-22T01:51:13.0253684Z Jan 22 01:51:13 if len(answer) > 1:
> 2022-01-22T01:51:13.0254169Z Jan 22 01:51:13 type = answer[1]
> 2022-01-22T01:51:13.0254757Z Jan 22 01:51:13 value =
> OUTPUT_CONVERTER[type](answer[2:], gateway_client)
> 2022-01-22T01:51:13.0255450Z Jan 22 01:51:13 if answer[1] ==
> REFERENCE_TYPE:
> 2022-01-22T01:51:13.0256085Z Jan 22 01:51:13 > raise
> Py4JJavaError(
> 2022-01-22T01:51:13.0256768Z Jan 22 01:51:13 "An
> error occurred while calling {0}{1}{2}.\n".
> 2022-01-22T01:51:13.0257432Z Jan 22 01:51:13
> format(target_id, ".", name), value)
> 2022-01-22T01:51:13.0258250Z Jan 22 01:51:13 E
> py4j.protocol.Py4JJavaError: An error occurred while calling
> o24634.addInsertSql.
> 2022-01-22T01:51:13.0259174Z Jan 22 01:51:13 E :
> java.lang.NullPointerException
> 2022-01-22T01:51:13.0259824Z Jan 22 01:51:13 Eat
> java.util.Objects.requireNonNull(Objects.java:203)
> 2022-01-22T01:51:13.0260748Z Jan 22 01:51:13 Eat
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:144)
> 2022-01-22T01:51:13.0261604Z Jan 22 01:51:13 Eat
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:108)
> 2022-01-22T01:51:13.0262653Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.(FlinkRelMetadataQuery.java:78)
> 2022-01-22T01:51:13.0263927Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.instance(FlinkRelMetadataQuery.java:59)
> 2022-01-22T01:51:13.0264864Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:39)
> 2022-01-22T01:51:13.0266075Z Jan 22 01:51:13 Eat
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:38)
> 2022-01-22T01:51:13.0267017Z Jan
[GitHub] [flink] flinkbot edited a comment on pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
flinkbot edited a comment on pull request #18376: URL: https://github.com/apache/flink/pull/18376#issuecomment-1014266348 ## CI report: * a415c9901fe5addb7e042b2ea760b2de95c736ca Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29529) * 369e3ea1432e9b033e5660642ab0f86b03ab4d79 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18409: [hotfix][dist] Fix that flink-gs-fs-hadoop not included in flink-dist
flinkbot edited a comment on pull request #18409: URL: https://github.com/apache/flink/pull/18409#issuecomment-1017132626 ## CI report: * d5d44917306061cd0bb6db9a20f2d1a4f95b0f94 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29771) * a44383b3a2c7a6dc5a9c96ad225ad8baaf03fee4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29986) * 74c02f4b21ad4488b0466a748114e26f15495c4e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29991) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18409: [hotfix][dist] Fix that flink-gs-fs-hadoop not included in flink-dist
flinkbot edited a comment on pull request #18409: URL: https://github.com/apache/flink/pull/18409#issuecomment-1017132626 ## CI report: * d5d44917306061cd0bb6db9a20f2d1a4f95b0f94 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29771) * a44383b3a2c7a6dc5a9c96ad225ad8baaf03fee4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29986) * 74c02f4b21ad4488b0466a748114e26f15495c4e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25728) Protential memory leaks in StreamMultipleInputProcessor
[ https://issues.apache.org/jira/browse/FLINK-25728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480850#comment-17480850 ] Yun Gao commented on FLINK-25728: - Thanks [~wpc009] for reporting the issue! I'll have a look~ > Protential memory leaks in StreamMultipleInputProcessor > --- > > Key: FLINK-25728 > URL: https://issues.apache.org/jira/browse/FLINK-25728 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.12.5, 1.15.0, 1.13.5, 1.14.2 >Reporter: pc wang >Priority: Blocker > Labels: pull-request-available > Attachments: flink-completablefuture-issue.tar.xz, > image-2022-01-20-18-43-32-816.png > > > We have an application that contains a broadcast process stage. The > none-broadcast input has roughly 10 million messages per second, and the > broadcast side is some kind of control stream, rarely has message follow > through. > After several hours of running, the TaskManager will run out of heap memory > and restart. We reviewed the application code without finding any relevant > issues. > We found that the running to crash time was roughly the same. Then we make a > heap dump before the crash and found mass `CompletableFuture$UniRun` > instances. > These `CompletableFuture$UniRun` instances consume several gigabytes memories. > > The following pic is from the heap dump we get from a mock testing stream > with the same scenario. > !image-2022-01-20-18-43-32-816.png|width=1161,height=471! > > After some source code research. We found that it might be caused by the > *StreamMultipleInputProcessor.getAvailableFuture()*. > *StreamMultipleInputProcessor* has multiple *inputProcessors* , it's > *availableFuture* got completed when any of it's input's *availableFuture* is > complete. > The current implementation create a new *CompletableFuture* and a new > *CompletableFuture$UniRun* append to delegate inputProcessor's > *avaiableFuture*. > The issue is caused by the stacking of *CompletableFuture$UniRun* on the slow > inputProcessor's *avaiableFuture*. > See the source code below. > [StreamMultipleInputProcessor.java#L65|https://github.com/wpc009/flink/blob/d33c39d974f08a5ac520f220219ecb0796c9448c/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessor.java#L65] > Because the *UniRun* holds the reference of outside > *StreamMultipleInputProcessor*'s avaiableFuture, that cause mass > *CompletableFuture* instance which can not be recycled. > We made some modifications to the > *StreamMultipleInputProcessor*.*getAvaiableFuture* function, and verify that > the issue is gone on our modified version. > We are willing to make a PR for this fix. > Heap Dump File [^flink-completablefuture-issue.tar.xz] > PS: This is a YourKit heap dump. may be not compatible HPROF files. > [Sample Code to reproduce the > issue|https://github.com/wpc009/flink/blob/FLINK-25728/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/MultipleInputStreamMemoryIssueTest.java] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18412: [FLINK-25696][datastream] Introduce MetadataPublisher interface to SinkWriter
flinkbot edited a comment on pull request #18412: URL: https://github.com/apache/flink/pull/18412#issuecomment-1017147201 ## CI report: * 256bff679ff13cc486f2f6543745924d23920176 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29969) * b42032867fb8a31ac9dd068ec1ae0f35f477952c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29980) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18409: [hotfix][dist] Fix that flink-gs-fs-hadoop not included in flink-dist
flinkbot edited a comment on pull request #18409: URL: https://github.com/apache/flink/pull/18409#issuecomment-1017132626 ## CI report: * d5d44917306061cd0bb6db9a20f2d1a4f95b0f94 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29771) * a44383b3a2c7a6dc5a9c96ad225ad8baaf03fee4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29986) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25728) Protential memory leaks in StreamMultipleInputProcessor
[ https://issues.apache.org/jira/browse/FLINK-25728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480848#comment-17480848 ] pc wang commented on FLINK-25728: - [~gaoyunhaii] hello, could you please take a look at this issue? Thanks. > Protential memory leaks in StreamMultipleInputProcessor > --- > > Key: FLINK-25728 > URL: https://issues.apache.org/jira/browse/FLINK-25728 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.12.5, 1.15.0, 1.13.5, 1.14.2 >Reporter: pc wang >Priority: Blocker > Labels: pull-request-available > Attachments: flink-completablefuture-issue.tar.xz, > image-2022-01-20-18-43-32-816.png > > > We have an application that contains a broadcast process stage. The > none-broadcast input has roughly 10 million messages per second, and the > broadcast side is some kind of control stream, rarely has message follow > through. > After several hours of running, the TaskManager will run out of heap memory > and restart. We reviewed the application code without finding any relevant > issues. > We found that the running to crash time was roughly the same. Then we make a > heap dump before the crash and found mass `CompletableFuture$UniRun` > instances. > These `CompletableFuture$UniRun` instances consume several gigabytes memories. > > The following pic is from the heap dump we get from a mock testing stream > with the same scenario. > !image-2022-01-20-18-43-32-816.png|width=1161,height=471! > > After some source code research. We found that it might be caused by the > *StreamMultipleInputProcessor.getAvailableFuture()*. > *StreamMultipleInputProcessor* has multiple *inputProcessors* , it's > *availableFuture* got completed when any of it's input's *availableFuture* is > complete. > The current implementation create a new *CompletableFuture* and a new > *CompletableFuture$UniRun* append to delegate inputProcessor's > *avaiableFuture*. > The issue is caused by the stacking of *CompletableFuture$UniRun* on the slow > inputProcessor's *avaiableFuture*. > See the source code below. > [StreamMultipleInputProcessor.java#L65|https://github.com/wpc009/flink/blob/d33c39d974f08a5ac520f220219ecb0796c9448c/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessor.java#L65] > Because the *UniRun* holds the reference of outside > *StreamMultipleInputProcessor*'s avaiableFuture, that cause mass > *CompletableFuture* instance which can not be recycled. > We made some modifications to the > *StreamMultipleInputProcessor*.*getAvaiableFuture* function, and verify that > the issue is gone on our modified version. > We are willing to make a PR for this fix. > Heap Dump File [^flink-completablefuture-issue.tar.xz] > PS: This is a YourKit heap dump. may be not compatible HPROF files. > [Sample Code to reproduce the > issue|https://github.com/wpc009/flink/blob/FLINK-25728/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/MultipleInputStreamMemoryIssueTest.java] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18409: [hotfix][dist] Fix that flink-gs-fs-hadoop not included in flink-dist
flinkbot edited a comment on pull request #18409: URL: https://github.com/apache/flink/pull/18409#issuecomment-1017132626 ## CI report: * d5d44917306061cd0bb6db9a20f2d1a4f95b0f94 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29771) * a44383b3a2c7a6dc5a9c96ad225ad8baaf03fee4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29986) * 74c02f4b21ad4488b0466a748114e26f15495c4e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18458: [FLINK-21789][network] Make FileChannelManagerImpl#getNextPathNum select data directories fairly
flinkbot edited a comment on pull request #18458: URL: https://github.com/apache/flink/pull/18458#issuecomment-1019767867 ## CI report: * 572d143d54fb5f9c87c5e251e3c2e1816a0249ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29990) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18458: [FLINK-21789][network] Make FileChannelManagerImpl#getNextPathNum select data directories fairly
flinkbot commented on pull request #18458: URL: https://github.com/apache/flink/pull/18458#issuecomment-1019767867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18455: [FLINK-25529][hive] add orc-core dependency specification in hive con…
flinkbot edited a comment on pull request #18455: URL: https://github.com/apache/flink/pull/18455#issuecomment-1019685703 ## CI report: * a739fb9e7bf9856f58be079986d6d19c4c1a11d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29973) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25760) Support custom SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
melin updated FLINK-25760:
--
Description:
Supports custom SQL syntax, similar to Spark Extensions Feature. Custom SQL
syntax can be implemented. example:
{code:java}
datax reader('数据类型名称') options(键值对参数)
[filter('filter action name') options(键值对参数)]*
writer('数据类型名称') options(键值对参数)
datax reader("jdbc") options(
username="dataworks",
password="dataworks2021",
type="mysql",
url="jdbc:mysql://10.5.20.20:3306",
databaseName='dataworks', tableName='dc_datax_datasource', column=["*"])
writer("hive") options(databaseName="bigdata",
tableName='hive_datax_datasource', writeMode='overwrite', column=["*"]); {code}
https://github.com/melin/datatunnel
was:Supports extended SQL syntax, similar to Spark Extensions Feature. Custom
SQL syntax can be implemented
> Support custom SQL syntax
> -
>
> Key: FLINK-25760
> URL: https://issues.apache.org/jira/browse/FLINK-25760
> Project: Flink
> Issue Type: Improvement
>Reporter: melin
>Priority: Major
>
> Supports custom SQL syntax, similar to Spark Extensions Feature. Custom SQL
> syntax can be implemented. example:
> {code:java}
> datax reader('数据类型名称') options(键值对参数)
> [filter('filter action name') options(键值对参数)]*
> writer('数据类型名称') options(键值对参数)
> datax reader("jdbc") options(
> username="dataworks",
> password="dataworks2021",
> type="mysql",
> url="jdbc:mysql://10.5.20.20:3306",
> databaseName='dataworks', tableName='dc_datax_datasource', column=["*"])
> writer("hive") options(databaseName="bigdata",
> tableName='hive_datax_datasource', writeMode='overwrite', column=["*"]);
> {code}
> https://github.com/melin/datatunnel
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] TanYuxin-tyx opened a new pull request #18458: [FLINK-21789][network] Make FileChannelManagerImpl#getNextPathNum select data directories fairly
TanYuxin-tyx opened a new pull request #18458: URL: https://github.com/apache/flink/pull/18458 ## What is the purpose of the change `FileChannelManagerImpl#getNextPathNum` is not atomic which may cause unfairness of data directory selection. This patch makes the selection of data directories fair which is good for load balance and performance especially when there are multiple disks. ## Brief change log - *Make FileChannelManagerImpl#getNextPathNum select data directories fairly.* ## Verifying this change This change added tests `FileChannelManagerImplTest#testFairness`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25760) Support custom SQL syntax
[ https://issues.apache.org/jira/browse/FLINK-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated FLINK-25760: -- Summary: Support custom SQL syntax (was: Support for extended SQL syntax) > Support custom SQL syntax > - > > Key: FLINK-25760 > URL: https://issues.apache.org/jira/browse/FLINK-25760 > Project: Flink > Issue Type: Improvement >Reporter: melin >Priority: Major > > Supports extended SQL syntax, similar to Spark Extensions Feature. Custom SQL > syntax can be implemented -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25728) Protential memory leaks in StreamMultipleInputProcessor
[ https://issues.apache.org/jira/browse/FLINK-25728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pc wang updated FLINK-25728: Summary: Protential memory leaks in StreamMultipleInputProcessor (was: Protential memory leeks in StreamMultipleInputProcessor) > Protential memory leaks in StreamMultipleInputProcessor > --- > > Key: FLINK-25728 > URL: https://issues.apache.org/jira/browse/FLINK-25728 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.12.5, 1.15.0, 1.13.5, 1.14.2 >Reporter: pc wang >Priority: Blocker > Labels: pull-request-available > Attachments: flink-completablefuture-issue.tar.xz, > image-2022-01-20-18-43-32-816.png > > > We have an application that contains a broadcast process stage. The > none-broadcast input has roughly 10 million messages per second, and the > broadcast side is some kind of control stream, rarely has message follow > through. > After several hours of running, the TaskManager will run out of heap memory > and restart. We reviewed the application code without finding any relevant > issues. > We found that the running to crash time was roughly the same. Then we make a > heap dump before the crash and found mass `CompletableFuture$UniRun` > instances. > These `CompletableFuture$UniRun` instances consume several gigabytes memories. > > The following pic is from the heap dump we get from a mock testing stream > with the same scenario. > !image-2022-01-20-18-43-32-816.png|width=1161,height=471! > > After some source code research. We found that it might be caused by the > *StreamMultipleInputProcessor.getAvailableFuture()*. > *StreamMultipleInputProcessor* has multiple *inputProcessors* , it's > *availableFuture* got completed when any of it's input's *availableFuture* is > complete. > The current implementation create a new *CompletableFuture* and a new > *CompletableFuture$UniRun* append to delegate inputProcessor's > *avaiableFuture*. > The issue is caused by the stacking of *CompletableFuture$UniRun* on the slow > inputProcessor's *avaiableFuture*. > See the source code below. > [StreamMultipleInputProcessor.java#L65|https://github.com/wpc009/flink/blob/d33c39d974f08a5ac520f220219ecb0796c9448c/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessor.java#L65] > Because the *UniRun* holds the reference of outside > *StreamMultipleInputProcessor*'s avaiableFuture, that cause mass > *CompletableFuture* instance which can not be recycled. > We made some modifications to the > *StreamMultipleInputProcessor*.*getAvaiableFuture* function, and verify that > the issue is gone on our modified version. > We are willing to make a PR for this fix. > Heap Dump File [^flink-completablefuture-issue.tar.xz] > PS: This is a YourKit heap dump. may be not compatible HPROF files. > [Sample Code to reproduce the > issue|https://github.com/wpc009/flink/blob/FLINK-25728/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/MultipleInputStreamMemoryIssueTest.java] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29983) * 11799600e83d55b8150cdb93ad90d1a55fd2651f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29989) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29983) * 11799600e83d55b8150cdb93ad90d1a55fd2651f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * bdbbf550bb2fa01fdfad3d1e64ee08cff4a2fb0e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29949) * a7afbb3176d79338140da86934d9b8cafdb5c1c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29972) * 6441c9641faa7398a9c1d77f5f3482b397a79690 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29982) * 7dcb2f8df322c2fb454f0e8aa0ef95d7bb696120 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29988) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * bdbbf550bb2fa01fdfad3d1e64ee08cff4a2fb0e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29949) * a7afbb3176d79338140da86934d9b8cafdb5c1c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29972) * 6441c9641faa7398a9c1d77f5f3482b397a79690 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29982) * 7dcb2f8df322c2fb454f0e8aa0ef95d7bb696120 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #18431: [FLINK-25024][docs] Add Changelog backend docs
curcur commented on a change in pull request #18431:
URL: https://github.com/apache/flink/pull/18431#discussion_r790412520
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
Review comment:
Usually, if we do not have a Chinese corresponding version, we will copy
the English version to the Chinese Version and open a ticket there?
The ticket can be grabbed by anyone.
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
+3. Snapshot upload time (asynchronous phase)
+
+The latter (upload time) can be decreased by [Incremental checkpoints]({{< ref
"#incremental-checkpoints" >}}). However,
+even with Incremental checkpoints, large deployments tend to have at least one
task in every checkpoint that uploads a
+lot of data (e.g. after compaction).
Review comment:
1. Add some context here:
Current Incremental Checkpoints depend on the implementation of different
types of state backends. For example, for rocksdb, compaction happens when ...
2. Explain a bit why compaction is bad
Compaction may cause more data to be uploaded, and more time to upload
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
+3. Snapshot upload time (asynchronous phase)
+
+The latter (upload time) can be decreased by [Incremental checkpoints]({{< ref
"#incremental-checkpoints" >}}). However,
+even with Incremental checkpoints, large deployments tend to have at least one
task in every checkpoint that uploads a
+lot of data (e.g. after compaction).
+
+With Changelog enabled, Flink uploads state changes continuously, forming a
changelog. On checkpoint, only the relevant
+part of this changelog needs to be uploaded. Independently, configured state
backend is checkpointed in the
+background periodically. Upon successful upload, changelog is truncated.
+
+As a result, asynchronous phase is reduced, as well as synchronous phase (in
particular, long-tail).
+
+On the flip side, resource usage is higher:
+
+- more files are created on DFS
+- more IO bandwidth is used to upload
Review comment:
upload state change/changelogs
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative
[jira] [Commented] (FLINK-25767) Translation of page 'Working with State' is incomplete
[ https://issues.apache.org/jira/browse/FLINK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480830#comment-17480830 ] Yun Tang commented on FLINK-25767: -- [~paul8263] already assigned to you, please go ahead. > Translation of page 'Working with State' is incomplete > -- > > Key: FLINK-25767 > URL: https://issues.apache.org/jira/browse/FLINK-25767 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.14.3 >Reporter: Yao Zhang >Assignee: Yao Zhang >Priority: Major > Fix For: 1.15.0 > > > The translation of page [Working with State | Apache > Flink|https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/datastream/fault-tolerance/state/] > is incomplete. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25767) Translation of page 'Working with State' is incomplete
[ https://issues.apache.org/jira/browse/FLINK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang reassigned FLINK-25767: Assignee: Yao Zhang > Translation of page 'Working with State' is incomplete > -- > > Key: FLINK-25767 > URL: https://issues.apache.org/jira/browse/FLINK-25767 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.14.3 >Reporter: Yao Zhang >Assignee: Yao Zhang >Priority: Major > Fix For: 1.15.0 > > > The translation of page [Working with State | Apache > Flink|https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/datastream/fault-tolerance/state/] > is incomplete. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17452: [FLINK-20732][connector/pulsar] Introduction of Pulsar Sink
flinkbot edited a comment on pull request #17452: URL: https://github.com/apache/flink/pull/17452#issuecomment-940136217 ## CI report: * c8820e335d23677ffb548b0d1fa8771fa7c1bce1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29965) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25761) Translate Avro format page into Chinese.
[ https://issues.apache.org/jira/browse/FLINK-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480460#comment-17480460 ] Zhiwu Wang edited comment on FLINK-25761 at 1/24/22, 6:12 AM: -- [~RocMarshal] i‘d like to help with this task. and will complete within few days. was (Author: JIRAUSER283917): [~RocMarshal] i‘d like to help with this task. and will complete within maybe 1-2 days. > Translate Avro format page into Chinese. > > > Key: FLINK-25761 > URL: https://issues.apache.org/jira/browse/FLINK-25761 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: RocMarshal >Priority: Minor > Labels: chinese-translation > > file location: > flink/docs/content.zh/docs/connectors/datastream/formats/avro.md -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18457: [FLINK-14902][code] Supports jdbc async lookup join
flinkbot edited a comment on pull request #18457: URL: https://github.com/apache/flink/pull/18457#issuecomment-1019749225 ## CI report: * 1ff6b7fdc37a68de37d05a96a922ae4fb56055b0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29987) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18457: [FLINK-14902][code] Supports jdbc async lookup join
flinkbot commented on pull request #18457: URL: https://github.com/apache/flink/pull/18457#issuecomment-1019749225 ## CI report: * 1ff6b7fdc37a68de37d05a96a922ae4fb56055b0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18409: [hotfix][dist] Fix that flink-gs-fs-hadoop not included in flink-dist
flinkbot edited a comment on pull request #18409: URL: https://github.com/apache/flink/pull/18409#issuecomment-1017132626 ## CI report: * d5d44917306061cd0bb6db9a20f2d1a4f95b0f94 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29771) * a44383b3a2c7a6dc5a9c96ad225ad8baaf03fee4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29986) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18457: [FLINK-14902][code] Supports jdbc async lookup join
flinkbot commented on pull request #18457: URL: https://github.com/apache/flink/pull/18457#issuecomment-1019748804 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1ff6b7fdc37a68de37d05a96a922ae4fb56055b0 (Mon Jan 24 06:02:49 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18409: [hotfix][dist] Fix that flink-gs-fs-hadoop not included in flink-dist
flinkbot edited a comment on pull request #18409: URL: https://github.com/apache/flink/pull/18409#issuecomment-1017132626 ## CI report: * d5d44917306061cd0bb6db9a20f2d1a4f95b0f94 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29771) * a44383b3a2c7a6dc5a9c96ad225ad8baaf03fee4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ranqiqiang opened a new pull request #18457: [FLINK-14902][code] Supports jdbc async lookup join
ranqiqiang opened a new pull request #18457: URL: https://github.com/apache/flink/pull/18457 ## **What is the purpose of the change** JdbcTableSource supports jdbc async look up join ## **Brief change log** 1.add class JdbcRowDataAsyncLookupFunction 2.add class JdbcConnectionPoolManager,manager connection entry 3.add class JdbcConnectionEntry 4.modify JdbcConnectorOptions, add async/async.parallelism options 5.modify JdbcDynamicTableFactory, add async options 6.modify JdbcDynamicTableSource, add asyncFunction provider ## **Verifying this change** This change added tests and can be verified as follows: 1.JdbcDynamicTableFactoryTest#testJdbcAsyncLookupProvider ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (don't know) - The runtime per-record code paths (performance sensitive): (don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no ) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? ( docs / JavaDocs ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ranqiqiang commented on pull request #18268: [FLINK-14902][connector] Supports jdbc async lookup join
ranqiqiang commented on pull request #18268: URL: https://github.com/apache/flink/pull/18268#issuecomment-1019745078 Found a bug ,resubmit the pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ranqiqiang closed pull request #18268: [FLINK-14902][connector] Supports jdbc async lookup join
ranqiqiang closed pull request #18268: URL: https://github.com/apache/flink/pull/18268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18414: [hotfix][docs]fix flink sql Cascading Window TVF Aggregation exception
flinkbot edited a comment on pull request #18414: URL: https://github.com/apache/flink/pull/18414#issuecomment-1017182067 ## CI report: * 2d7b1b7425be2b9cfb9020d92cf08cc6d5596ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29864) * e59d3cd9b63b0fef051a3b02094eb05bd1c58d12 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29985) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18414: [hotfix][docs]fix flink sql Cascading Window TVF Aggregation exception
flinkbot edited a comment on pull request #18414: URL: https://github.com/apache/flink/pull/18414#issuecomment-1017182067 ## CI report: * 2d7b1b7425be2b9cfb9020d92cf08cc6d5596ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29864) * e59d3cd9b63b0fef051a3b02094eb05bd1c58d12 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] simenliuxing commented on a change in pull request #18414: [hotfix][docs]fix flink sql Cascading Window TVF Aggregation exception
simenliuxing commented on a change in pull request #18414: URL: https://github.com/apache/flink/pull/18414#discussion_r790426803 ## File path: docs/content.zh/docs/dev/table/sql/queries/window-agg.md ## @@ -199,7 +199,8 @@ The following shows a cascading window aggregation where the first window aggreg ```sql -- tumbling 5 minutes for each supplier_id CREATE VIEW window1 AS -SELECT window_start, window_end, window_time as rowtime, SUM(price) as partial_price +-- under the Cascading Window Aggregation to avoid field ambiguity window_start and window_end need to be renamed Review comment: Looks good to me, I have revised -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] deadwind4 commented on a change in pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
deadwind4 commented on a change in pull request #18388:
URL: https://github.com/apache/flink/pull/18388#discussion_r790425806
##
File path: flink-python/pyflink/datastream/tests/test_connectors.py
##
@@ -149,6 +150,80 @@ def tearDown(self):
get_gateway().jvm.Thread.currentThread().setContextClassLoader(self._cxt_clz_loader)
+class FlinkPulsarTest(PyFlinkTestCase):
+
+def setUp(self) -> None:
+self.env = StreamExecutionEnvironment.get_execution_environment()
+self.env.set_parallelism(2)
+# Cache current ContextClassLoader, we will replace it with a
temporary URLClassLoader to
+# load specific connector jars with given module path to do dependency
isolation. And We
+# will change the ClassLoader back to the cached ContextClassLoader
after the test case
+# finished.
+self._cxt_clz_loader =
get_gateway().jvm.Thread.currentThread().getContextClassLoader()
+
_load_specific_flink_module_jars('/flink-connectors/flink-sql-connector-pulsar')
Review comment:
extracting to a ConnectorTestBase class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
wanglijie95 commented on a change in pull request #18376:
URL: https://github.com/apache/flink/pull/18376#discussion_r790425616
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SsgNetworkMemoryCalculationUtils.java
##
@@ -148,6 +161,78 @@ private static TaskInputsOutputsDescriptor
buildTaskInputsOutputsDescriptor(
return ret;
}
+private static Map
getMaxInputChannelNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+for (IntermediateResult consumedResult : ejv.getInputs()) {
+ret.put(consumedResult.getId(),
getMaxInputChannelNumForResult(ejv, consumedResult));
+}
+
+return ret;
+}
+
+private static Map
getMaxSubpartitionNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+
+for (IntermediateResult intermediateResult :
ejv.getProducedDataSets()) {
+final int maxNum =
+Arrays.stream(intermediateResult.getPartitions())
+
.map(IntermediateResultPartition::getNumberOfSubpartitions)
+.reduce(0, Integer::max);
+ret.put(intermediateResult.getId(), maxNum);
+}
+
+return ret;
+}
+
+@VisibleForTesting
+static int getMaxInputChannelNumForResult(
+ExecutionJobVertex ejv, IntermediateResult consumedResult) {
+DistributionPattern distributionPattern =
consumedResult.getConsumingDistributionPattern();
+
+if (distributionPattern == DistributionPattern.ALL_TO_ALL) {
+int numChannelsToConsumePerPartition =
+
getMaxNumOfChannelsForConsuming(consumedResult.getPartitions()[0]);
+int numConsumedPartitions =
consumedResult.getNumberOfAssignedPartitions();
+return numChannelsToConsumePerPartition * numConsumedPartitions;
+
+} else if (distributionPattern == DistributionPattern.POINTWISE) {
+int numPartitions = consumedResult.getNumberOfAssignedPartitions();
+int numConsumers = ejv.getParallelism();
+// when using dynamic graph, all partitions have the same number
of subpartitions
Review comment:
> e.g. if later someone tried to optimized
`IntermediateResultPartition#computeNumberOfSubpartitions()` to avoid creating
extra subpartitions if the parallelism of the downstream vertex is known
already, this assumption may break. However, he/she may not be aware of the
changes here the assumption here may stay unchanged.
You are right. I will change it to use
`TaskDeploymentDescriptorFactory#computeConsumedSubpartitionRange` to compute
subpartition range for each consumer, and then calculate the max number of
channels according to the subpartition range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
wanglijie95 commented on a change in pull request #18376:
URL: https://github.com/apache/flink/pull/18376#discussion_r790425616
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SsgNetworkMemoryCalculationUtils.java
##
@@ -148,6 +161,78 @@ private static TaskInputsOutputsDescriptor
buildTaskInputsOutputsDescriptor(
return ret;
}
+private static Map
getMaxInputChannelNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+for (IntermediateResult consumedResult : ejv.getInputs()) {
+ret.put(consumedResult.getId(),
getMaxInputChannelNumForResult(ejv, consumedResult));
+}
+
+return ret;
+}
+
+private static Map
getMaxSubpartitionNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+
+for (IntermediateResult intermediateResult :
ejv.getProducedDataSets()) {
+final int maxNum =
+Arrays.stream(intermediateResult.getPartitions())
+
.map(IntermediateResultPartition::getNumberOfSubpartitions)
+.reduce(0, Integer::max);
+ret.put(intermediateResult.getId(), maxNum);
+}
+
+return ret;
+}
+
+@VisibleForTesting
+static int getMaxInputChannelNumForResult(
+ExecutionJobVertex ejv, IntermediateResult consumedResult) {
+DistributionPattern distributionPattern =
consumedResult.getConsumingDistributionPattern();
+
+if (distributionPattern == DistributionPattern.ALL_TO_ALL) {
+int numChannelsToConsumePerPartition =
+
getMaxNumOfChannelsForConsuming(consumedResult.getPartitions()[0]);
+int numConsumedPartitions =
consumedResult.getNumberOfAssignedPartitions();
+return numChannelsToConsumePerPartition * numConsumedPartitions;
+
+} else if (distributionPattern == DistributionPattern.POINTWISE) {
+int numPartitions = consumedResult.getNumberOfAssignedPartitions();
+int numConsumers = ejv.getParallelism();
+// when using dynamic graph, all partitions have the same number
of subpartitions
Review comment:
> e.g. if later someone tried to optimized
`IntermediateResultPartition#computeNumberOfSubpartitions()` to avoid creating
extra subpartitions if the parallelism of the downstream vertex is known
already, this assumption may break. However, he/she may not be aware of the
changes here the assumption here may stay unchanged.
You are right. I will change it to use
TaskDeploymentDescriptorFactory#computeConsumedSubpartitionRange to compute
subpartition range for each consumer, and then calculate the max number of
channels according to the subpartition range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
wanglijie95 commented on a change in pull request #18376:
URL: https://github.com/apache/flink/pull/18376#discussion_r790425357
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SsgNetworkMemoryCalculationUtils.java
##
@@ -148,6 +161,78 @@ private static TaskInputsOutputsDescriptor
buildTaskInputsOutputsDescriptor(
return ret;
}
+private static Map
getMaxInputChannelNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+for (IntermediateResult consumedResult : ejv.getInputs()) {
+ret.put(consumedResult.getId(),
getMaxInputChannelNumForResult(ejv, consumedResult));
+}
+
+return ret;
+}
+
+private static Map
getMaxSubpartitionNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+
+for (IntermediateResult intermediateResult :
ejv.getProducedDataSets()) {
+final int maxNum =
+Arrays.stream(intermediateResult.getPartitions())
+
.map(IntermediateResultPartition::getNumberOfSubpartitions)
+.reduce(0, Integer::max);
+ret.put(intermediateResult.getId(), maxNum);
+}
+
+return ret;
+}
+
+@VisibleForTesting
+static int getMaxInputChannelNumForResult(
+ExecutionJobVertex ejv, IntermediateResult consumedResult) {
+DistributionPattern distributionPattern =
consumedResult.getConsumingDistributionPattern();
+
+if (distributionPattern == DistributionPattern.ALL_TO_ALL) {
+int numChannelsToConsumePerPartition =
+
getMaxNumOfChannelsForConsuming(consumedResult.getPartitions()[0]);
+int numConsumedPartitions =
consumedResult.getNumberOfAssignedPartitions();
+return numChannelsToConsumePerPartition * numConsumedPartitions;
+
+} else if (distributionPattern == DistributionPattern.POINTWISE) {
+int numPartitions = consumedResult.getNumberOfAssignedPartitions();
+int numConsumers = ejv.getParallelism();
+// when using dynamic graph, all partitions have the same number
of subpartitions
+int numOfSubpartitionsPerPartition =
+
consumedResult.getPartitions()[0].getNumberOfSubpartitions();
+
+if (numPartitions >= numConsumers) {
+// multiple partitions to one consumer
+int maxConsumedPartitionsPerConsumer =
+(int) Math.ceil((double) numPartitions / numConsumers);
+return numOfSubpartitionsPerPartition *
maxConsumedPartitionsPerConsumer;
+} else {
+// one partition to multiple consumers
+int minConsumersPerPartition =
+(int) Math.floor((double) numConsumers /
numPartitions);
+return (int)
+Math.ceil(
+(double) numOfSubpartitionsPerPartition /
minConsumersPerPartition);
+}
Review comment:
This calculation logic here is no longer needed after modification as
described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
wanglijie95 commented on a change in pull request #18376:
URL: https://github.com/apache/flink/pull/18376#discussion_r790424936
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SsgNetworkMemoryCalculationUtils.java
##
@@ -148,6 +161,78 @@ private static TaskInputsOutputsDescriptor
buildTaskInputsOutputsDescriptor(
return ret;
}
+private static Map
getMaxInputChannelNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+for (IntermediateResult consumedResult : ejv.getInputs()) {
+ret.put(consumedResult.getId(),
getMaxInputChannelNumForResult(ejv, consumedResult));
+}
+
+return ret;
+}
+
+private static Map
getMaxSubpartitionNumsForDynamicGraph(
+ExecutionJobVertex ejv) {
+
+Map ret = new HashMap<>();
+
+for (IntermediateResult intermediateResult :
ejv.getProducedDataSets()) {
+final int maxNum =
+Arrays.stream(intermediateResult.getPartitions())
+
.map(IntermediateResultPartition::getNumberOfSubpartitions)
+.reduce(0, Integer::max);
+ret.put(intermediateResult.getId(), maxNum);
+}
+
+return ret;
+}
+
+@VisibleForTesting
+static int getMaxInputChannelNumForResult(
+ExecutionJobVertex ejv, IntermediateResult consumedResult) {
+DistributionPattern distributionPattern =
consumedResult.getConsumingDistributionPattern();
+
+if (distributionPattern == DistributionPattern.ALL_TO_ALL) {
+int numChannelsToConsumePerPartition =
+
getMaxNumOfChannelsForConsuming(consumedResult.getPartitions()[0]);
+int numConsumedPartitions =
consumedResult.getNumberOfAssignedPartitions();
+return numChannelsToConsumePerPartition * numConsumedPartitions;
+
+} else if (distributionPattern == DistributionPattern.POINTWISE) {
+int numPartitions = consumedResult.getNumberOfAssignedPartitions();
+int numConsumers = ejv.getParallelism();
+// when using dynamic graph, all partitions have the same number
of subpartitions
Review comment:
You are right. I will change it to use
`TaskDeploymentDescriptorFactory#computeConsumedSubpartitionRange` to compute
subpartition range for each consumer, and then calculate the max number of
channels according to the subpartition range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-13550) Support for CPU FlameGraphs in web UI
[ https://issues.apache.org/jira/browse/FLINK-13550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480821#comment-17480821 ] jackylau commented on FLINK-13550: -- hi [~afedulov] do you have any thoughts the comments i add > Support for CPU FlameGraphs in web UI > - > > Key: FLINK-13550 > URL: https://issues.apache.org/jira/browse/FLINK-13550 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: David Morávek >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-11-23-13-36-03-269.png > > > For a better insight into a running job, it would be useful to have ability > to render a CPU flame graph for a particular job vertex. > Flink already has a stack-trace sampling mechanism in-place, so it should be > straightforward to implement. > This should be done by implementing a new endpoint in REST API, which would > sample the stack-trace the same way as current BackPressureTracker does, only > with a different sampling rate and length of sampling. > [Here|https://www.youtube.com/watch?v=GUNDehj9z9o] is a little demo of the > feature. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-13550) Support for CPU FlameGraphs in web UI
[ https://issues.apache.org/jira/browse/FLINK-13550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472570#comment-17472570 ] jackylau edited comment on FLINK-13550 at 1/24/22, 5:29 AM: hi [~dmvk] , [~afedulov] i found the flame graph webui is hang when the job is parallesim is big such as 500+. i support it using async profiler by add script ability in task manager. and that is taskmanger level instead of Jobvertex level. will it be able to add it open source flink? was (Author: jackylau): hi [~dmvk] , i found the flame graph webui is hang when the job is parallesim is big such as 500+. i support it using async profiler by add script ability in task manager. and that is taskmanger level instead of Jobvertex level. will it be able to add it open source flink? > Support for CPU FlameGraphs in web UI > - > > Key: FLINK-13550 > URL: https://issues.apache.org/jira/browse/FLINK-13550 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: David Morávek >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-11-23-13-36-03-269.png > > > For a better insight into a running job, it would be useful to have ability > to render a CPU flame graph for a particular job vertex. > Flink already has a stack-trace sampling mechanism in-place, so it should be > straightforward to implement. > This should be done by implementing a new endpoint in REST API, which would > sample the stack-trace the same way as current BackPressureTracker does, only > with a different sampling rate and length of sampling. > [Here|https://www.youtube.com/watch?v=GUNDehj9z9o] is a little demo of the > feature. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18453: [FLINK-25562][hive] add detail explanation about bundled hive jar to …
flinkbot edited a comment on pull request #18453: URL: https://github.com/apache/flink/pull/18453#issuecomment-1019659584 ## CI report: * 653b4f8de5758180898ee04fde4d2870770f1a9e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29966) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29983) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * bdbbf550bb2fa01fdfad3d1e64ee08cff4a2fb0e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29949) * a7afbb3176d79338140da86934d9b8cafdb5c1c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29972) * 6441c9641faa7398a9c1d77f5f3482b397a79690 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29982) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * bdbbf550bb2fa01fdfad3d1e64ee08cff4a2fb0e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29949) * a7afbb3176d79338140da86934d9b8cafdb5c1c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29972) * 6441c9641faa7398a9c1d77f5f3482b397a79690 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zuston commented on a change in pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
zuston commented on a change in pull request #18157:
URL: https://github.com/apache/flink/pull/18157#discussion_r790413135
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointMetadataOutputStream.java
##
@@ -162,4 +161,20 @@ public FsCompletedCheckpointStorageLocation
closeAndFinalizeCheckpoint() throws
}
}
}
+
+static MetadataOutputStreamWrapper getOutputStreamWrapper(
+final FileSystem fileSystem, final Path metadataFilePath) throws
IOException {
+try {
+RecoverableWriter recoverableWriter =
fileSystem.createRecoverableWriter();
+if (fileSystem.exists(metadataFilePath)) {
+throw new IOException("Target file " + metadataFilePath + " is
already exists.");
Review comment:
Updated. @Myasuka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zuston commented on a change in pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
zuston commented on a change in pull request #18157:
URL: https://github.com/apache/flink/pull/18157#discussion_r790413069
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointMetadataOutputStream.java
##
@@ -162,4 +161,20 @@ public FsCompletedCheckpointStorageLocation
closeAndFinalizeCheckpoint() throws
}
}
}
+
+static MetadataOutputStreamWrapper getOutputStreamWrapper(
+final FileSystem fileSystem, final Path metadataFilePath) throws
IOException {
+try {
+RecoverableWriter recoverableWriter =
fileSystem.createRecoverableWriter();
+if (fileSystem.exists(metadataFilePath)) {
+throw new IOException("Target file " + metadataFilePath + " is
already exists.");
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] deadwind4 commented on a change in pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
deadwind4 commented on a change in pull request #18388: URL: https://github.com/apache/flink/pull/18388#discussion_r790406471 ## File path: flink-python/pyflink/datastream/connectors.py ## @@ -1079,6 +1085,320 @@ def for_row_format(base_path: str, encoder: Encoder) -> 'FileSink.RowFormatBuild JFileSink.forRowFormat(JPath(base_path), encoder._j_encoder)) +class PulsarDeserializationSchema(object): +""" +A schema bridge for deserializing the pulsar's Message into a flink managed instance. We +support both the pulsar's self managed schema and flink managed schema. +""" + +def __init__(self, _j_pulsar_deserialization_schema): +self._j_pulsar_deserialization_schema = _j_pulsar_deserialization_schema + +@staticmethod +def flink_schema(deserialization_schema: DeserializationSchema) \ +-> 'PulsarDeserializationSchema': +""" +Create a PulsarDeserializationSchema by using the flink's DeserializationSchema. It would +consume the pulsar message as byte array and decode the message by using flink's logic. +""" +JPulsarDeserializationSchema = get_gateway().jvm.org.apache.flink \ + .connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema +_j_pulsar_deserialization_schema = JPulsarDeserializationSchema.flinkSchema( +deserialization_schema._j_deserialization_schema) +return PulsarDeserializationSchema(_j_pulsar_deserialization_schema) + +@staticmethod +def flink_type_info(type_information: TypeInformation, execution_config: ExecutionConfig) \ +-> 'PulsarDeserializationSchema': +""" +Create a PulsarDeserializationSchema by using the given TypeInformation. This method is +only used for treating message that was written into pulsar by TypeInformation. +""" +JPulsarDeserializationSchema = get_gateway().jvm.org.apache.flink \ + .connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema +_j_execution_config = execution_config._j_execution_config \ +if execution_config is not None else None +_j_pulsar_deserialization_schema = JPulsarDeserializationSchema.flinkTypeInfo( +type_information.get_java_type_info(), _j_execution_config) +return PulsarDeserializationSchema(_j_pulsar_deserialization_schema) + + +class SubscriptionType(Enum): +""" +Types of subscription supported by Pulsar. + +:data: `Exclusive`: + +There can be only 1 consumer on the same topic with the same subscription name. + +:data: `Shared`: + +Multiple consumer will be able to use the same subscription name and the messages will be +dispatched according to a round-robin rotation between the connected consumers. In this mode, +the consumption order is not guaranteed. + +:data: `Failover`: + +Multiple consumer will be able to use the same subscription name but only 1 consumer will +receive the messages. If that consumer disconnects, one of the other connected consumers will +start receiving messages. In failover mode, the consumption ordering is guaranteed. In case of +partitioned topics, the ordering is guaranteed on a per-partition basis. The partitions +assignments will be split across the available consumers. On each partition, at most one +consumer will be active at a given point in time. + +:data: `Key_Shared`: + +Multiple consumer will be able to use the same subscription and all messages with the same key +will be dispatched to only one consumer. Use ordering_key to overwrite the message key for +message ordering. +""" + +Exclusive = 0, +Shared = 1, +Failover = 2, +Key_Shared = 3 + +def _to_j_subscription_type(self): +JSubscriptionType = get_gateway().jvm.org.apache.pulsar.client.api.SubscriptionType +return getattr(JSubscriptionType, self.name) + + +class StartCursor(object): +""" +A interface for users to specify the start position of a pulsar subscription. +Since it would be serialized into split. +The implementation for this interface should be well considered. +I don't recommend adding extra internal state for this implementation. + +This class would be used only for SubscriptionType.Exclusive and SubscriptionType.Failover. +""" + +def __init__(self, _j_start_cursor): +self._j_start_cursor = _j_start_cursor + +@staticmethod +def default_start_cursor() -> 'StartCursor': +return StartCursor.earliest() + +@staticmethod +def earliest() -> 'StartCursor': +JStartCursor = get_gateway().jvm \ + .org.apache.flink.connector.pulsar.source.enumerator.cursor.StartCursor +return StartCursor(JStartCursor.earliest()) + +@staticmethod +def latest() -> 'StartCursor': +JStartCursor = get_gateway().jvm \ +
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25577) Update GCS documentation
[ https://issues.apache.org/jira/browse/FLINK-25577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480813#comment-17480813 ] Xintong Song commented on FLINK-25577: -- [~galenwarren], The release notes can only be added when closing the ticket. You may comment the draft of the release notes, and I or whoever else closing the ticket can add them. bq. Add the flink-gs-fs-hadoop FileSystem plugin for Google Cloud Storage, with RecoverableWriter support. I think the release notes are usually a bit more descriptive. Unlike the JIRA summary or commit message tells what has been done by the developers, the release notes are meant for the users to learn what they can do with the new feature. Check `docs/content/release-notes/` for examples. > Update GCS documentation > > > Key: FLINK-25577 > URL: https://issues.apache.org/jira/browse/FLINK-25577 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, Documentation >Reporter: David Morávek >Assignee: Galen Warren >Priority: Blocker > Labels: pull-request-available > Fix For: 1.15.0 > > > Update GCS documentation with respect to: > - flink-shaded-hadoop artifacts and hadoop version 2.8.3 that are no longer > supported as of 1.15 > - The recoverable writer new feature > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/gcs/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18361: [FLINK-25631][table] Support enhanced `show tables` syntax
flinkbot edited a comment on pull request #18361: URL: https://github.com/apache/flink/pull/18361#issuecomment-1012985152 ## CI report: * a99d1e2258fda0f16b582210a3ab7bd950506aed Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29889) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18412: [FLINK-25696][datastream] Introduce MetadataPublisher interface to SinkWriter
flinkbot edited a comment on pull request #18412: URL: https://github.com/apache/flink/pull/18412#issuecomment-1017147201 ## CI report: * 91a7a1406e7d09b6914bff1caeaf4ec92a798d8d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29760) * 256bff679ff13cc486f2f6543745924d23920176 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29969) * b42032867fb8a31ac9dd068ec1ae0f35f477952c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29980) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18268: [FLINK-14902][connector] Supports jdbc async lookup join
flinkbot edited a comment on pull request #18268: URL: https://github.com/apache/flink/pull/18268#issuecomment-1005479356 ## CI report: * f64a2fe9f7c1ca9a041641256dc09d00253ce837 UNKNOWN * 259aeca4d91fc0adfaef93fdba9aa872a866a3c8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29884) * e9b9f77af909871ca49edf1d3ff6169827c0407d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29979) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18412: [FLINK-25696][datastream] Introduce MetadataPublisher interface to SinkWriter
flinkbot edited a comment on pull request #18412: URL: https://github.com/apache/flink/pull/18412#issuecomment-1017147201 ## CI report: * 91a7a1406e7d09b6914bff1caeaf4ec92a798d8d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29760) * 256bff679ff13cc486f2f6543745924d23920176 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29969) * b42032867fb8a31ac9dd068ec1ae0f35f477952c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #15599: [FLINK-11838][flink-gs-fs-hadoop] Create Google Storage file system with recoverable writer support
xintongsong commented on pull request #15599: URL: https://github.com/apache/flink/pull/15599#issuecomment-1019700632 @galenwarren, sorry for the late response. I was OoO the past a few days. > The change I'd propose to help @sap1ens (or anyone else who used the pre-released code) would be to change 0 to 1 in the following lines in GSCommitRecoverableSerializer and GSResumeRecoverableSerializer We don't usually provide compatibility guarantees for pre-released codes. However, in this specific case, I see barely any cost in making lives of the pre-released code users easier. So +1 for changing the serializer version. I've already opened a hotfix PR #18409 for including `flink-gs-fs-hadoop` into `flink-dist`, and I can change the serializer version there. Concerning the authentication, do you have an idea how much effort the 2nd approach may require? - I think it would be ideal that we can fallback to `core-site.xml`, if it contains the credentials AND `GOOGLE_APPLICATION_CREDENTIALS` is not specified. This would be my top preference, if feasible. - If the above approach requires too much effort / time (as we are approaching to the feature freeze date), I'd also be fine with documenting this as a known limitation (that `core-site.xml` does not work with the `RecoverableWriter` part), and address this as a follow-up issue in the next release cycle. - I would suggest not to exclude the `core-site.xml` approach from the documentation, unless we decide to deprecate / remove this approach for good. Otherwise, it might confuse users whether the approach is still available or not. And as far as I can see, there's not yet strong reasons for making such an incompatible change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18268: [FLINK-14902][connector] Supports jdbc async lookup join
flinkbot edited a comment on pull request #18268: URL: https://github.com/apache/flink/pull/18268#issuecomment-1005479356 ## CI report: * f64a2fe9f7c1ca9a041641256dc09d00253ce837 UNKNOWN * 259aeca4d91fc0adfaef93fdba9aa872a866a3c8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29884) * e9b9f77af909871ca49edf1d3ff6169827c0407d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * c15632cf1ee4b38d0060a87c3bedb5cb4d545264 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29851) * 6a4a6c064380db3219fb92805e9a87712bb85fa2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29971) * 237027aabd1c103dbdd722e67f1f2a8475149c48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29978) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18268: [FLINK-14902][connector] Supports jdbc async lookup join
flinkbot edited a comment on pull request #18268: URL: https://github.com/apache/flink/pull/18268#issuecomment-1005479356 ## CI report: * f64a2fe9f7c1ca9a041641256dc09d00253ce837 UNKNOWN * 259aeca4d91fc0adfaef93fdba9aa872a866a3c8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29884) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * c15632cf1ee4b38d0060a87c3bedb5cb4d545264 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29851) * 6a4a6c064380db3219fb92805e9a87712bb85fa2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29971) * 237027aabd1c103dbdd722e67f1f2a8475149c48 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
Myasuka commented on a change in pull request #18157:
URL: https://github.com/apache/flink/pull/18157#discussion_r790401088
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointMetadataOutputStream.java
##
@@ -162,4 +161,20 @@ public FsCompletedCheckpointStorageLocation
closeAndFinalizeCheckpoint() throws
}
}
}
+
+static MetadataOutputStreamWrapper getOutputStreamWrapper(
+final FileSystem fileSystem, final Path metadataFilePath) throws
IOException {
+try {
+RecoverableWriter recoverableWriter =
fileSystem.createRecoverableWriter();
+if (fileSystem.exists(metadataFilePath)) {
+throw new IOException("Target file " + metadataFilePath + " is
already exists.");
Review comment:
I think we should try to verify the file whether existed first.
Moreover, the statement `"Target file " + metadataFilePath + " already
exists."` looks better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18456: [FLINK-21790][network] Shuffle data directories to make directory selection of different TaskManagers fairer
flinkbot edited a comment on pull request #18456: URL: https://github.com/apache/flink/pull/18456#issuecomment-1019691654 ## CI report: * cd74a88227edf6a00098a9d651634e009ac3fe9f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29977) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 837b39cfdaf47aeb2f3a1b824bcffc22aff529ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29767) * 68b840fe2df44305be126b638e965bf8b3433b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29976) * d7c57c901a1d061766c39d1572885b6df3c6fc1e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
