[GitHub] [flink] godfreyhe commented on pull request #18785: [FLINK-26167][table-planner] Explicitly set the partitioner for the sql operators whose shuffle and sort are removed
godfreyhe commented on pull request #18785: URL: https://github.com/apache/flink/pull/18785#issuecomment-1041345782 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18726: [FLINK-25983] Add API for configuring maximal watermark drift
flinkbot edited a comment on pull request #18726: URL: https://github.com/apache/flink/pull/18726#issuecomment-1036101042 ## CI report: * 318d5650f14a51a9f1aee5fe5f97a14d5d54bef6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31630) * 0ee9c6a291b7d2a9bf83707eee0f8d8f06f05afb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31640) * 255530e344dce93e506b9cfc1c3d2c15894c51b0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
flinkbot edited a comment on pull request #18680: URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635 ## CI report: * 2ffd741d316346745bd8fe754bb02a73cbf0d0ba Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614) * 48412828133309fc4f4cddecfedcdfcbbbee8ebf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26148) Change the format of adaptive batch scheduler config option to "jobmanager.adaptive-batch-scheduler.XXX"
[
https://issues.apache.org/jira/browse/FLINK-26148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493156#comment-17493156
]
Lijie Wang commented on FLINK-26148:
Yes, I will update it soon.
> Change the format of adaptive batch scheduler config option to
> "jobmanager.adaptive-batch-scheduler.XXX"
>
>
> Key: FLINK-26148
> URL: https://issues.apache.org/jira/browse/FLINK-26148
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Lijie Wang
>Assignee: Lijie Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Change the format of adaptive batch scheduler config option to
> {{jobmanager.adaptive-batch-scheduler.XXX}} to align the format with the
> existing scheduler option (For example,
> {{{}jobmanager.adaptive-scheduler.min-parallelism-increase{}}}).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26148) Change the format of adaptive batch scheduler config option to "jobmanager.adaptive-batch-scheduler.XXX"
[
https://issues.apache.org/jira/browse/FLINK-26148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493154#comment-17493154
]
Zhu Zhu commented on FLINK-26148:
-
[~wanglijie95] would you update FLIP-187 for the configuration name change?
> Change the format of adaptive batch scheduler config option to
> "jobmanager.adaptive-batch-scheduler.XXX"
>
>
> Key: FLINK-26148
> URL: https://issues.apache.org/jira/browse/FLINK-26148
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Lijie Wang
>Assignee: Lijie Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Change the format of adaptive batch scheduler config option to
> {{jobmanager.adaptive-batch-scheduler.XXX}} to align the format with the
> existing scheduler option (For example,
> {{{}jobmanager.adaptive-scheduler.min-parallelism-increase{}}}).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26121) ZooKeeperLeaderRetrievalConnectionHandlingTest.testNewLeaderAfterReconnectTriggersListenerNotification failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493153#comment-17493153
]
Matthias Pohl commented on FLINK-26121:
---
There seems to be a gap of 10s in processing time while waiting for the
leadership information. It's hard to tell here whether that's a AzureCI worker
issue where the instance was just not operating for some time or whether there
was no output due to the busy waiting.
> ZooKeeperLeaderRetrievalConnectionHandlingTest.testNewLeaderAfterReconnectTriggersListenerNotification
> failed on azure
> --
>
> Key: FLINK-26121
> URL: https://issues.apache.org/jira/browse/FLINK-26121
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Assignee: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> {code:java}
> 2022-02-11T21:43:35.4936452Z Feb 11 21:43:35 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2022-02-11T21:43:35.4940444Z Feb 11 21:43:35 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:81)
> 2022-02-11T21:43:35.4941937Z Feb 11 21:43:35 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:36)
> 2022-02-11T21:43:35.4943249Z Feb 11 21:43:35 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:60)
> 2022-02-11T21:43:35.4944745Z Feb 11 21:43:35 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-02-11T21:43:35.4945682Z Feb 11 21:43:35 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-02-11T21:43:35.4946655Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-02-11T21:43:35.4947847Z Feb 11 21:43:35 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-02-11T21:43:35.4948876Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-02-11T21:43:35.4949842Z Feb 11 21:43:35 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-02-11T21:43:35.4951142Z Feb 11 21:43:35 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-02-11T21:43:35.4952153Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-02-11T21:43:35.4953115Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-02-11T21:43:35.4954068Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-02-11T21:43:35.4955003Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-02-11T21:43:35.4955981Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-02-11T21:43:35.4956930Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-02-11T21:43:35.4958008Z Feb 11 21:43:35 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2022-02-11T21:43:35.4958899Z Feb 11 21:43:35 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2022-02-11T21:43:35.4959774Z Feb 11 21:43:35 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2022-02-11T21:43:35.4960911Z Feb 11 21:43:35 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> 2022-02-11T21:43:35.4962095Z Feb 11 21:43:35 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> 2022-02-11T21:43:35.4963136Z Feb 11 21:43:35 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> 2022-02-11T21:43:35.4964275Z Feb 11 21:43:35 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
> 2022-02-11T21:43:35.4965527Z Feb 11 21:43:35 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
> 2022-02-11T21:43:35.4966787Z Feb 11 21:43:35 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
> 2022-02-11T21:43:35.4968228Z Feb 11 21:43:35 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
> 2022-02-11T21:43:35.4969485Z Feb 11 21:43:35 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
> 2022-02-11T21:43:35.4970753Z Feb 11 21:43:35 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #18786: [hotfix][test] add support for field assignability
flinkbot edited a comment on pull request #18786: URL: https://github.com/apache/flink/pull/18786#issuecomment-1040539589 ## CI report: * 199d94fca6df52f120633c8202b40409e1d7e794 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31615) * 84d45868a291322310ba3d321b580b8513b35031 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31642) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
flinkbot edited a comment on pull request #18680: URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635 ## CI report: * Unknown: [CANCELED](TBD) * 2ffd741d316346745bd8fe754bb02a73cbf0d0ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614) * 48412828133309fc4f4cddecfedcdfcbbbee8ebf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18786: [hotfix][test] add support for field assignability
flinkbot edited a comment on pull request #18786: URL: https://github.com/apache/flink/pull/18786#issuecomment-1040539589 ## CI report: * 6fa8c290984d253ef7408abb6d23d49370c0f4b8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31580) * 199d94fca6df52f120633c8202b40409e1d7e794 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31615) * 84d45868a291322310ba3d321b580b8513b35031 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18799: [FLINK-26145][docs] fix a kubernetes image that does not exist in the…
flinkbot edited a comment on pull request #18799: URL: https://github.com/apache/flink/pull/18799#issuecomment-1041334465 ## CI report: * 73640cb14e6a68cacc27b281f8d5508a5ffa21ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31641) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18726: [FLINK-25983] Add API for configuring maximal watermark drift
flinkbot edited a comment on pull request #18726: URL: https://github.com/apache/flink/pull/18726#issuecomment-1036101042 ## CI report: * 318d5650f14a51a9f1aee5fe5f97a14d5d54bef6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31630) * 0ee9c6a291b7d2a9bf83707eee0f8d8f06f05afb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31640) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25000) Scala 2.12.7 doesn't compile on Java 17
[ https://issues.apache.org/jira/browse/FLINK-25000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-25000: - Description: Fails with "fails with /packages cannot be represented as URI" during compilation. 2.12.15 was working fine. was: Fails with "fails with /packages cannot be represented as URI" during compilation. 2.12.5 was working fine. > Scala 2.12.7 doesn't compile on Java 17 > --- > > Key: FLINK-25000 > URL: https://issues.apache.org/jira/browse/FLINK-25000 > Project: Flink > Issue Type: Sub-task > Components: API / Scala >Reporter: Chesnay Schepler >Priority: Major > > Fails with "fails with /packages cannot be represented as URI" during > compilation. > 2.12.15 was working fine. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wanglijie95 commented on a change in pull request #18757: [FLINK-25226][doc] Add documentation about the AdaptiveBatchScheduler
wanglijie95 commented on a change in pull request #18757:
URL: https://github.com/apache/flink/pull/18757#discussion_r807768428
##
File path: docs/content/docs/deployment/adaptive_batch_scheduler.md
##
@@ -0,0 +1,63 @@
+---
+title: Adaptive Batch Scheduler
+weight: 5
+type: docs
+
+---
+
+
+## Adaptive Batch Scheduler
+
+The Adaptive Batch Scheduler can automatically decide parallelisms of job
vertices for batch jobs. If a job vertex is not set with a parallelism, the
scheduler will decide parallelism for the job vertex according to the size of
its consumed datasets. This can bring many benefits:
+- Batch job users can be relieved from parallelism tuning
+- Automatically tuned parallelisms can be vertex level and can better fit
consumed datasets which have a varying volume size every day
+- Vertices from SQL batch jobs can be assigned with different parallelisms
which are automatically tuned
+
+### Usage
+
+To automatically decide parallelisms for job vertices through Adaptive Batch
Scheduler, you need to:
+- Configure to use Adaptive Batch Scheduler.
+- Set the parallelism of job vertices to `-1`.
+
+ Configure to use Adaptive Batch Scheduler
+To use Adaptive Batch Scheduler, you need to set the
[`jobmanager.scheduler`]({{< ref "docs/deployment/config"
>}}#jobmanager-scheduler) to `AdpaptiveBatch`. In addition, there are several
optional config options that might need adjustment when using Adaptive Batch
Scheduler:
+- [`jobmanager.scheduler.adaptive-batch.min-parallelism`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-min-parallelism): The lower bound of
allowed parallelism to set adaptively
+- [`jobmanager.scheduler.adaptive-batch.max-parallelism`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-max-parallelism): The upper bound of
allowed parallelism to set adaptively
+- [`jobmanager.scheduler.adaptive-batch.data-volume-per-task`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-data-volume-per-task): The size of data
volume to expect each task instance to process
+- [`jobmanager.scheduler.adaptive-batch.source-parallelism.default`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-source-parallelism-default): The
default parallelism of source vertices
+
+ Set the parallelism of job vertices to `-1`
+Adaptive Batch Scheduler will only decide parallelism for job vertices whose
parallelism is not specified by users (parallelism is `-1`). So if you want the
parallelism of vertices can be decided automatically, you should configure as
follows:
+- Set `paralleims.default` to `-1`
+- Set `table.exec.resource.default-parallelism` to -1 in SQL jobs.
+- Don't call `setParallelism()` for operators in datastream jobs.
+
+### Performance tuning
+
+1. It's recommended to use `Sort Shuffle` and set
[`taskmanager.network.memory.buffers-per-channel`]({{< ref
"docs/deployment/config" >}}#taskmanager-network-memory-buffers-per-channel) to
`0`. This can decouple the network memory consumption from parallelism, so for
large scale jobs, the possibility of "Insufficient number of network buffers"
error can be decreased.
+2. It's not recommended to configure an excessive value for
[`jobmanager.scheduler.adaptive-batch.max-parallelism`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-max-parallelism), otherwise it will
affect the performance. Because this option can affect the number of
subpartitions produced by upstream tasks, excessive number of subpartitions may
degrade the performance of hash shuffle and the performance of network
transmission due to small packets.
+
+### Limitations
+
+- **ALL-EDGES-BLOCKING batch jobs only**: The first version of Adaptive Batch
Scheduler only supports ALL-EDGES-BLOCKING batch jobs only.
Review comment:
+1 for @zhuzhurk 's comment. Just tell user adaptive batch scheduler
only support the case where `execution.batch-shuffle-mode` is
`ALL-EXCHANGES-BLOCKING`, and link to the config pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18799: [FLINK-26145][docs] fix a kubernetes image that does not exist in the…
flinkbot commented on pull request #18799: URL: https://github.com/apache/flink/pull/18799#issuecomment-1041334465 ## CI report: * 73640cb14e6a68cacc27b281f8d5508a5ffa21ea UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18798: [FLINK-26177][tests] Disable PulsarSourceITCase rescaling tests temporarily
flinkbot edited a comment on pull request #18798: URL: https://github.com/apache/flink/pull/18798#issuecomment-1041265347 ## CI report: * 7b3b53570a9ac3f4b7159e7b1bf24eb7d29b2649 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31627) * d87a36d6c2290af48b79a325beeeb917fe3501ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31638) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26185) E2E Elasticsearch Tests should use the new Sink interface
Alexander Preuss created FLINK-26185: Summary: E2E Elasticsearch Tests should use the new Sink interface Key: FLINK-26185 URL: https://issues.apache.org/jira/browse/FLINK-26185 Project: Flink Issue Type: Bug Components: Connectors / ElasticSearch, Tests Affects Versions: 1.15.0 Reporter: Alexander Preuss Currently the E2E tests for Elasticsearch (test_streaming_elasticsearch.sh) is testing the old Sink interface implementation. As we are now moving to the new Sink interface, we should update the tests to test the new implementation -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18726: [FLINK-25983] Add API for configuring maximal watermark drift
flinkbot edited a comment on pull request #18726: URL: https://github.com/apache/flink/pull/18726#issuecomment-1036101042 ## CI report: * 318d5650f14a51a9f1aee5fe5f97a14d5d54bef6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31630) * 0ee9c6a291b7d2a9bf83707eee0f8d8f06f05afb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18756: [FLINK-26075][table-api][table-planner] Persist node configuration
flinkbot edited a comment on pull request #18756: URL: https://github.com/apache/flink/pull/18756#issuecomment-1039112190 ## CI report: * c7c123b2541e931272beb18f619bf8d8a418e471 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31572) * c018f4c7da0b607e45293323b568ca55d414cea7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31637) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18799: [FLINK-26145][docs] fix a kubernetes image that does not exist in the…
flinkbot commented on pull request #18799: URL: https://github.com/apache/flink/pull/18799#issuecomment-1041332368 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 73640cb14e6a68cacc27b281f8d5508a5ffa21ea (Wed Feb 16 10:20:48 UTC 2022) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18798: [FLINK-26177][tests] Disable PulsarSourceITCase rescaling tests temporarily
flinkbot edited a comment on pull request #18798: URL: https://github.com/apache/flink/pull/18798#issuecomment-1041265347 ## CI report: * 7b3b53570a9ac3f4b7159e7b1bf24eb7d29b2649 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31627) * d87a36d6c2290af48b79a325beeeb917fe3501ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-25532) Provide Flink SQL CLI as Docker image
[ https://issues.apache.org/jira/browse/FLINK-25532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Knauf resolved FLINK-25532. -- Resolution: Fixed Fixed via Documentation. > Provide Flink SQL CLI as Docker image > - > > Key: FLINK-25532 > URL: https://issues.apache.org/jira/browse/FLINK-25532 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Martijn Visser >Assignee: Konstantin Knauf >Priority: Major > Labels: pull-request-available > > Flink is currently available via as Docker images. However, the Flink SQL CLI > isn't available as a Docker image. We should also provide this. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25532) Provide Flink SQL CLI as Docker image
[ https://issues.apache.org/jira/browse/FLINK-25532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Knauf updated FLINK-25532: - Fix Version/s: 1.15.0 > Provide Flink SQL CLI as Docker image > - > > Key: FLINK-25532 > URL: https://issues.apache.org/jira/browse/FLINK-25532 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Martijn Visser >Assignee: Konstantin Knauf >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Flink is currently available via as Docker images. However, the Flink SQL CLI > isn't available as a Docker image. We should also provide this. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25532) Provide Flink SQL CLI as Docker image
[ https://issues.apache.org/jira/browse/FLINK-25532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Knauf reassigned FLINK-25532: Assignee: Konstantin Knauf > Provide Flink SQL CLI as Docker image > - > > Key: FLINK-25532 > URL: https://issues.apache.org/jira/browse/FLINK-25532 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Martijn Visser >Assignee: Konstantin Knauf >Priority: Major > Labels: pull-request-available > > Flink is currently available via as Docker images. However, the Flink SQL CLI > isn't available as a Docker image. We should also provide this. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18756: [FLINK-26075][table-api][table-planner] Persist node configuration
flinkbot edited a comment on pull request #18756: URL: https://github.com/apache/flink/pull/18756#issuecomment-1039112190 ## CI report: * c7c123b2541e931272beb18f619bf8d8a418e471 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31572) * c018f4c7da0b607e45293323b568ca55d414cea7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26145) k8s docs jobmanager-pod-template artifacts-fetcher:latest image is not exist, we can use busybox to replace it
[ https://issues.apache.org/jira/browse/FLINK-26145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26145: --- Labels: pull-request-available (was: ) > k8s docs jobmanager-pod-template artifacts-fetcher:latest image is not > exist, we can use busybox to replace it > > > Key: FLINK-26145 > URL: https://issues.apache.org/jira/browse/FLINK-26145 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.15.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > Attachments: image-2022-02-15-16-55-43-593.png > > > !image-2022-02-15-16-54-42-861.png! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] knaufk merged pull request #18725: [FLINK-25532] Improve Documentation of Flink Docker-Compose
knaufk merged pull request #18725: URL: https://github.com/apache/flink/pull/18725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18726: [FLINK-25983] Add API for configuring maximal watermark drift
flinkbot edited a comment on pull request #18726: URL: https://github.com/apache/flink/pull/18726#issuecomment-1036101042 ## CI report: * 318d5650f14a51a9f1aee5fe5f97a14d5d54bef6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31630) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuyongvs opened a new pull request #18799: [FLINK-26145][docs] fix a kubernetes image that does not exist in the…
liuyongvs opened a new pull request #18799: URL: https://github.com/apache/flink/pull/18799 ## What is the purpose of the change * fix docs* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18611: [FLINK-24385][table] Introduce TRY_CAST
flinkbot edited a comment on pull request #18611: URL: https://github.com/apache/flink/pull/18611#issuecomment-1028155639 ## CI report: * f369013d63f0d367eab9744112e56bd7a8987177 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31566) * 5296f1f1e20d9e966cb207bb9dda3c3f7dafea38 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31635) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-25980) remove unnecessary condition in IntervalJoinOperator
[
https://issues.apache.org/jira/browse/FLINK-25980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul resolved FLINK-25980.
-
Fix Version/s: 1.15.0
Resolution: Fixed
> remove unnecessary condition in IntervalJoinOperator
>
>
> Key: FLINK-25980
> URL: https://issues.apache.org/jira/browse/FLINK-25980
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.7, 1.13.5, 1.14.2, 1.14.3
>Reporter: hongshu
>Assignee: hongshu
>Priority: Major
> Labels: pull-request-available, pull_request_available
> Fix For: 1.15.0
>
>
> Condition 'currentWatermark != Long.MIN_VALUE' covered by subsequent
> condition 'timestamp < currentWatermark'
> org.apache.flink.streaming.api.operators.co.IntervalJoinOperator#isLate
> {code:java}
> private boolean isLate(long timestamp) {
> long currentWatermark = internalTimerService.currentWatermark();
> return currentWatermark != Long.MIN_VALUE && timestamp < currentWatermark;
> } {code}
> if currentWatermark == Long.MIN_VALUE, timestamp < currentWatermark it's also
> return false, so condition currentWatermark != Long.MIN_VALUE is unnecessary
> We can use the following code directly
> {code:java}
> private boolean isLate(long timestamp) {
> long currentWatermark = internalTimerService.currentWatermark();
> return timestamp < currentWatermark;
> } {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18611: [FLINK-24385][table] Introduce TRY_CAST
flinkbot edited a comment on pull request #18611: URL: https://github.com/apache/flink/pull/18611#issuecomment-1028155639 ## CI report: * f369013d63f0d367eab9744112e56bd7a8987177 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31566) * 5296f1f1e20d9e966cb207bb9dda3c3f7dafea38 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25980) remove unnecessary condition in IntervalJoinOperator
[
https://issues.apache.org/jira/browse/FLINK-25980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493139#comment-17493139
]
Fabian Paul commented on FLINK-25980:
-
Merged in master: ebfc7c8d599d561a7d18a9ebea23a3c8297d2e0c
> remove unnecessary condition in IntervalJoinOperator
>
>
> Key: FLINK-25980
> URL: https://issues.apache.org/jira/browse/FLINK-25980
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.7, 1.13.5, 1.14.2, 1.14.3
>Reporter: hongshu
>Assignee: hongshu
>Priority: Major
> Labels: pull-request-available, pull_request_available
>
> Condition 'currentWatermark != Long.MIN_VALUE' covered by subsequent
> condition 'timestamp < currentWatermark'
> org.apache.flink.streaming.api.operators.co.IntervalJoinOperator#isLate
> {code:java}
> private boolean isLate(long timestamp) {
> long currentWatermark = internalTimerService.currentWatermark();
> return currentWatermark != Long.MIN_VALUE && timestamp < currentWatermark;
> } {code}
> if currentWatermark == Long.MIN_VALUE, timestamp < currentWatermark it's also
> return false, so condition currentWatermark != Long.MIN_VALUE is unnecessary
> We can use the following code directly
> {code:java}
> private boolean isLate(long timestamp) {
> long currentWatermark = internalTimerService.currentWatermark();
> return timestamp < currentWatermark;
> } {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] fapaul merged pull request #18647: [FLINK-25980][datastream] remove unnecessary condition
fapaul merged pull request #18647: URL: https://github.com/apache/flink/pull/18647 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #18611: [FLINK-24385][table] Introduce TRY_CAST
slinkydeveloper commented on a change in pull request #18611:
URL: https://github.com/apache/flink/pull/18611#discussion_r807755945
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/CastRuleProvider.java
##
@@ -107,6 +108,16 @@ public static boolean exists(LogicalType inputType,
LogicalType targetType) {
return resolve(inputType, targetType) != null;
}
+/**
+ * Resolves the rule and returns the result of {@link
CastRule#canFail(LogicalType,
+ * LogicalType)}. Fails with {@link NullPointerException} if the rule
cannot be resolved.
+ */
+public static boolean canFail(LogicalType inputType, LogicalType
targetType) {
+return Preconditions.checkNotNull(
+resolve(inputType, targetType), "Cast rule cannot be
resolved")
+.canFail(inputType, targetType);
Review comment:
I pushed in the last commit a consistent change to fix this, in order to
merge the "can fail" algorithm with the rule matching algorithm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-25946) table-planner-loader jar NOTICE should list Scala
[ https://issues.apache.org/jira/browse/FLINK-25946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-25946. Resolution: Fixed > table-planner-loader jar NOTICE should list Scala > - > > Key: FLINK-25946 > URL: https://issues.apache.org/jira/browse/FLINK-25946 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Table SQL / Planner >Affects Versions: 1.15.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18798: [FLINK-26177][tests] Disable PulsarSourceITCase rescaling tests temporarily
flinkbot edited a comment on pull request #18798: URL: https://github.com/apache/flink/pull/18798#issuecomment-1041265347 ## CI report: * 7b3b53570a9ac3f4b7159e7b1bf24eb7d29b2649 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31627) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18726: [FLINK-25983] Add API for configuring maximal watermark drift
flinkbot edited a comment on pull request #18726: URL: https://github.com/apache/flink/pull/18726#issuecomment-1036101042 ## CI report: * 9b9e4dc96990436cb5684646ed801b84f2357848 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31619) * 318d5650f14a51a9f1aee5fe5f97a14d5d54bef6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31630) * 0ee9c6a291b7d2a9bf83707eee0f8d8f06f05afb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25958) OOME Checkpoints & Savepoints were shown as COMPLETE in Flink UI
[
https://issues.apache.org/jira/browse/FLINK-25958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-25958:
--
Assignee: Anton Kalashnikov
> OOME Checkpoints & Savepoints were shown as COMPLETE in Flink UI
>
>
> Key: FLINK-25958
> URL: https://issues.apache.org/jira/browse/FLINK-25958
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0, 1.12.7, 1.13.5, 1.14.3
> Environment: Ververica Platform 2.6.2
> Flink 1.13.5
>Reporter: Victor Xu
>Assignee: Anton Kalashnikov
>Priority: Major
> Attachments: JIRA-1.jpg
>
>
> Flink job was running but the checkpoints & savepoints were failing all the
> time due to OOM Exception. However, the Flink UI showed COMPLETE for those
> checkpoints & savepoints.
> For example (checkpoint 39 & 40):
> {noformat}
> 2022-01-27 02:41:39,969 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 39 (type=CHECKPOINT) @ 1643251299952 for job
> ab2217e5ce144087bbddf6bd6c3
> 668eb.
> 2022-01-27 02:43:19,678 WARN org.apache.flink.runtime.jobmaster.JobMaster
> [] - Error while processing AcknowledgeCheckpoint message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete
> the pending checkpoint 39. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1227)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-s
> tream2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
> at java.lang.Thread.run(Thread.java:829) [?:?]
> Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
> at java.lang.Throwable.addSuppressed(Throwable.java:1054) ~[?:?]
> at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:627)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.serializeCheckpoint(KubernetesHaCheckpointStore.java:204)
> ~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.09
> 1138-2.jar:?]
> at
> com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.addCheckpoint(KubernetesHaCheckpointStore.java:83)
> ~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.091138-2.
> jar:?]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1209)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> ... 9 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 2022-01-27 03:41:39,970 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 40 (type=CHECKPOINT) @ 1643254899952 for job
> ab2217e5ce144087bbddf6bd6c3
> 668eb.
> 2022-01-27 03:43:22,326 WARN org.apache.flink.runtime.jobmaster.JobMaster
> [] - Error while processing AcknowledgeCheckpoint message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete
> the pending checkpoint 40. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1227)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
>
[GitHub] [flink] flinkbot edited a comment on pull request #18798: [FLINK-26177][tests] Disable PulsarSourceITCase rescaling tests temporarily
flinkbot edited a comment on pull request #18798: URL: https://github.com/apache/flink/pull/18798#issuecomment-1041265347 ## CI report: * 7b3b53570a9ac3f4b7159e7b1bf24eb7d29b2649 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31627) * d87a36d6c2290af48b79a325beeeb917fe3501ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18789: [FLINK-26168][runtime] The judgment of chainable ignores StreamExchangeMode when partitioner is ForwardForConsecutiveHashPartitioner
flinkbot edited a comment on pull request #18789: URL: https://github.com/apache/flink/pull/18789#issuecomment-1040944258 ## CI report: * cdab9f7d140989714263bc2a4e1a0908bb444a12 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31586) * 72851364f75d1b6e40ea1688408c3b85dccd6ec7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31633) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25571) Update Elasticsearch Sink to use decomposed interfaces
[ https://issues.apache.org/jira/browse/FLINK-25571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493133#comment-17493133 ] Fabian Paul commented on FLINK-25571: - Merged in master: 2d517950a9ae4a58b25f59fb6048f4c1a15225ac > Update Elasticsearch Sink to use decomposed interfaces > -- > > Key: FLINK-25571 > URL: https://issues.apache.org/jira/browse/FLINK-25571 > Project: Flink > Issue Type: Sub-task > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-25571) Update Elasticsearch Sink to use decomposed interfaces
[ https://issues.apache.org/jira/browse/FLINK-25571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul resolved FLINK-25571. - Fix Version/s: 1.15.0 Resolution: Fixed > Update Elasticsearch Sink to use decomposed interfaces > -- > > Key: FLINK-25571 > URL: https://issues.apache.org/jira/browse/FLINK-25571 > Project: Flink > Issue Type: Sub-task > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18785: [FLINK-26167][table-planner] Explicitly set the partitioner for the sql operators whose shuffle and sort are removed
flinkbot edited a comment on pull request #18785: URL: https://github.com/apache/flink/pull/18785#issuecomment-1040524147 ## CI report: * 50789f74f16c2bd3e42a61754f426877a3376056 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31608) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26183) Support kubernetes-operator metrics using the Flink metric system
[ https://issues.apache.org/jira/browse/FLINK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26183: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Support kubernetes-operator metrics using the Flink metric system > - > > Key: FLINK-26183 > URL: https://issues.apache.org/jira/browse/FLINK-26183 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Matyas Orhidi >Priority: Major > > We should leverage the existing Metric and Reporter infrastructure of Flink > to expose metrics of the Flink operator. > Users should be able to pass configuration to the operator that will control > the metric registry (like for regular Flink jobs) and should be able to > access the built in metric reporter plugins. > > Initially we should expose standard JVM metrics, and later we can add > operator specific metric groups if needed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26181) Support manual savepoint triggering in the operator
[ https://issues.apache.org/jira/browse/FLINK-26181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26181: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Support manual savepoint triggering in the operator > --- > > Key: FLINK-26181 > URL: https://issues.apache.org/jira/browse/FLINK-26181 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > We should allow users to trigger savepoints manually. There have been > different approaches to implement this in different operators: > - savepoint generation field (Google/Spotify operator) > - trigger savepoint via resource annotation [~wangyang0918] - native > operator poc > I think the annotation approach is more elegant but we should discuss this on > the ML before proceeding with the implementation -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wanglijie95 commented on a change in pull request #18757: [FLINK-25226][doc] Add documentation about the AdaptiveBatchScheduler
wanglijie95 commented on a change in pull request #18757:
URL: https://github.com/apache/flink/pull/18757#discussion_r807748519
##
File path: docs/content/docs/deployment/adaptive_batch_scheduler.md
##
@@ -0,0 +1,63 @@
+---
+title: Adaptive Batch Scheduler
+weight: 5
+type: docs
+
+---
+
+
+## Adaptive Batch Scheduler
+
+The Adaptive Batch Scheduler can automatically decide parallelisms of job
vertices for batch jobs. If a job vertex is not set with a parallelism, the
scheduler will decide parallelism for the job vertex according to the size of
its consumed datasets. This can bring many benefits:
+- Batch job users can be relieved from parallelism tuning
+- Automatically tuned parallelisms can be vertex level and can better fit
consumed datasets which have a varying volume size every day
+- Vertices from SQL batch jobs can be assigned with different parallelisms
which are automatically tuned
+
+### Usage
+
+To automatically decide parallelisms for job vertices through Adaptive Batch
Scheduler, you need to:
+- Configure to use Adaptive Batch Scheduler.
+- Set the parallelism of job vertices to `-1`.
+
+ Configure to use Adaptive Batch Scheduler
+To use Adaptive Batch Scheduler, you need to set the
[`jobmanager.scheduler`]({{< ref "docs/deployment/config"
>}}#jobmanager-scheduler) to `AdpaptiveBatch`. In addition, there are several
optional config options that might need adjustment when using Adaptive Batch
Scheduler:
+- [`jobmanager.scheduler.adaptive-batch.min-parallelism`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-min-parallelism): The lower bound of
allowed parallelism to set adaptively
+- [`jobmanager.scheduler.adaptive-batch.max-parallelism`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-max-parallelism): The upper bound of
allowed parallelism to set adaptively
+- [`jobmanager.scheduler.adaptive-batch.data-volume-per-task`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-data-volume-per-task): The size of data
volume to expect each task instance to process
+- [`jobmanager.scheduler.adaptive-batch.source-parallelism.default`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-source-parallelism-default): The
default parallelism of source vertices
+
+ Set the parallelism of job vertices to `-1`
+Adaptive Batch Scheduler will only decide parallelism for job vertices whose
parallelism is not specified by users (parallelism is `-1`). So if you want the
parallelism of vertices can be decided automatically, you should configure as
follows:
+- Set `paralleims.default` to `-1`
+- Set `table.exec.resource.default-parallelism` to -1 in SQL jobs.
+- Don't call `setParallelism()` for operators in datastream jobs.
+
+### Performance tuning
+
+1. It's recommended to use `Sort Shuffle` and set
[`taskmanager.network.memory.buffers-per-channel`]({{< ref
"docs/deployment/config" >}}#taskmanager-network-memory-buffers-per-channel) to
`0`. This can decouple the network memory consumption from parallelism, so for
large scale jobs, the possibility of "Insufficient number of network buffers"
error can be decreased.
Review comment:
+1 for this. I will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fapaul merged pull request #18666: [FLINK-25571] Update Elasticsearch Sink to use decomposed interfaces
fapaul merged pull request #18666: URL: https://github.com/apache/flink/pull/18666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18604: [FLINK-25926][Connectors][JDBC] Update org.postgresql:postgresql to 42.3.2
flinkbot edited a comment on pull request #18604: URL: https://github.com/apache/flink/pull/18604#issuecomment-1027886359 ## CI report: * 948f554e347ff5cb332d3273372718012d5b14fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18757: [FLINK-25226][doc] Add documentation about the AdaptiveBatchScheduler
wanglijie95 commented on a change in pull request #18757:
URL: https://github.com/apache/flink/pull/18757#discussion_r807748121
##
File path: docs/content/docs/deployment/adaptive_batch_scheduler.md
##
@@ -0,0 +1,63 @@
+---
+title: Adaptive Batch Scheduler
+weight: 5
+type: docs
+
+---
+
+
+## Adaptive Batch Scheduler
+
+The Adaptive Batch Scheduler can automatically decide parallelisms of job
vertices for batch jobs. If a job vertex is not set with a parallelism, the
scheduler will decide parallelism for the job vertex according to the size of
its consumed datasets. This can bring many benefits:
+- Batch job users can be relieved from parallelism tuning
+- Automatically tuned parallelisms can be vertex level and can better fit
consumed datasets which have a varying volume size every day
+- Vertices from SQL batch jobs can be assigned with different parallelisms
which are automatically tuned
+
+### Usage
+
+To automatically decide parallelisms for job vertices through Adaptive Batch
Scheduler, you need to:
+- Configure to use Adaptive Batch Scheduler.
+- Set the parallelism of job vertices to `-1`.
+
+ Configure to use Adaptive Batch Scheduler
+To use Adaptive Batch Scheduler, you need to set the
[`jobmanager.scheduler`]({{< ref "docs/deployment/config"
>}}#jobmanager-scheduler) to `AdpaptiveBatch`. In addition, there are several
optional config options that might need adjustment when using Adaptive Batch
Scheduler:
+- [`jobmanager.scheduler.adaptive-batch.min-parallelism`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-min-parallelism): The lower bound of
allowed parallelism to set adaptively
+- [`jobmanager.scheduler.adaptive-batch.max-parallelism`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-max-parallelism): The upper bound of
allowed parallelism to set adaptively
+- [`jobmanager.scheduler.adaptive-batch.data-volume-per-task`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-data-volume-per-task): The size of data
volume to expect each task instance to process
+- [`jobmanager.scheduler.adaptive-batch.source-parallelism.default`]({{< ref
"docs/deployment/config"
>}}#jobmanager-scheduler-adaptive-batch-source-parallelism-default): The
default parallelism of source vertices
+
+ Set the parallelism of job vertices to `-1`
Review comment:
If users explicitly configure the `parallelism.default` (with a value >
0) in `flink-conf`, but we override this value with `-1`, I think this may give
the users a feeling that the configuration does not take effect. Maybe we can
check the value of `parallelism.default` and then print an `ERROR` or
`WARNING` log if the value > 0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-26141) Support last-state upgrade mode
[ https://issues.apache.org/jira/browse/FLINK-26141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26141: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Support last-state upgrade mode > --- > > Key: FLINK-26141 > URL: https://issues.apache.org/jira/browse/FLINK-26141 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > The operator currently only implements savepoint and stateless upgrade > strategies for Flink jobs. > We should investigate if we can provide last-state upgrade strategy that > would use the latest available checkpoint the make job upgrades instead of > savepoints. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26142) Integrate flink-kubernetes-operator repo with CI/CD
[ https://issues.apache.org/jira/browse/FLINK-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26142: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Integrate flink-kubernetes-operator repo with CI/CD > --- > > Key: FLINK-26142 > URL: https://issues.apache.org/jira/browse/FLINK-26142 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Yang Wang >Priority: Major > > We should be able to run the unit/integration tests on the existing Flink CI > infra and publish docker images for new builds to dockerhub -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26157) Containers Should Not Run As Root
[ https://issues.apache.org/jira/browse/FLINK-26157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26157: --- Component/s: Kubernetes Operator > Containers Should Not Run As Root > - > > Key: FLINK-26157 > URL: https://issues.apache.org/jira/browse/FLINK-26157 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Matyas Orhidi >Assignee: Matyas Orhidi >Priority: Major > > Processes in a container should not run as root. Create a user in the > Dockerfile with a known UID:GID (e.g. flink:flink) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26179) Support periodic savepointing in the operator
[ https://issues.apache.org/jira/browse/FLINK-26179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26179: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Support periodic savepointing in the operator > - > > Key: FLINK-26179 > URL: https://issues.apache.org/jira/browse/FLINK-26179 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Automatic triggering of savepoints is a commonly requested feature. The > configuration should be part of the job spec. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26163) Refactor FlinkUtils#getEffectiveConfig into smaller pieces
[ https://issues.apache.org/jira/browse/FLINK-26163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26163: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Refactor FlinkUtils#getEffectiveConfig into smaller pieces > -- > > Key: FLINK-26163 > URL: https://issues.apache.org/jira/browse/FLINK-26163 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Biao Geng >Priority: Major > > There is currently one very large method dealing with Configuration > management. This logic probably deserves it's own utility class and a few > more modular methods. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26140) Add basic handling mechanism to deal with job upgrade errors
[ https://issues.apache.org/jira/browse/FLINK-26140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26140: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Add basic handling mechanism to deal with job upgrade errors > > > Key: FLINK-26140 > URL: https://issues.apache.org/jira/browse/FLINK-26140 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > There are various different ways how a stateful job upgrade can fail. > For example: > - Failure/timeout during savepoint > - Incompatible state > - Corrupted / not-found checkpoint > - Error after restart > We should allow some strategies for the user to declare how to handle the > different error scenarios (such as roll back to earlier state) and what > should be treated as a fatal error. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26139) Improve JobStatus tracking and handle different job states
[ https://issues.apache.org/jira/browse/FLINK-26139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26139: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Improve JobStatus tracking and handle different job states > -- > > Key: FLINK-26139 > URL: https://issues.apache.org/jira/browse/FLINK-26139 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Currently we do not handle any job status changes such as cancellations, > errors or job completions. > We should introduce some mechanism to react and deal with these changes and > expose them in the status as they can potentially affect upgrades. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26138) Create controller test
[ https://issues.apache.org/jira/browse/FLINK-26138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26138: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Create controller test > -- > > Key: FLINK-26138 > URL: https://issues.apache.org/jira/browse/FLINK-26138 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Add test to validate the general controller flow like observe -> reconcile, > deletions etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26137) Create webhook REST api test
[ https://issues.apache.org/jira/browse/FLINK-26137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26137: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Create webhook REST api test > > > Key: FLINK-26137 > URL: https://issues.apache.org/jira/browse/FLINK-26137 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Add test to validate the webhook rest endpoint and make sure it returns the > expected responses, status codes etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26135) Separate job and deployment errors in FlinkDeployment status
[ https://issues.apache.org/jira/browse/FLINK-26135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26135: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Separate job and deployment errors in FlinkDeployment status > > > Key: FLINK-26135 > URL: https://issues.apache.org/jira/browse/FLINK-26135 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Major > > At the moment the controller does not validate or tolerate any deployment > errors such as incorrect configurations etc. Those will lead to an exception > loop in the reconcile logic. > There are cases where the job deployment cannot be executed due to incorrect > configuration or other causes. In these cases the job can still be running > correctly so the job status should be OK but we should signal a deployment > error to the user that requires action. > There should be a shared validation logic between the controller and the > webhook that should be applied whenever a new FlinkDeployment update is > received by the controller. If an error is detected in the controller, set > the deployment status to error with a useful message and leave the current > job running. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26136) Implement shared validation logic for FlinkDeployment objects
[ https://issues.apache.org/jira/browse/FLINK-26136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26136: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Implement shared validation logic for FlinkDeployment objects > - > > Key: FLINK-26136 > URL: https://issues.apache.org/jira/browse/FLINK-26136 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > At the moment there is only a very basic “placeholder” validation logic > implemented in the webhook module: > org.apache.flink.kubernetes.operator.admission.FlinkDeploymentValidator > We should aim to validate parts of the FlinkDeployment that can be done > upfront, things like most common Flink config options, parallelism, resources > etc. > As described in https://issues.apache.org/jira/browse/FLINK-26135 this > validation should be part of the flink-kubernetes-operator module. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26078) Initial Kubernetes Operator Prototype contribution
[ https://issues.apache.org/jira/browse/FLINK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26078: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Initial Kubernetes Operator Prototype contribution > -- > > Key: FLINK-26078 > URL: https://issues.apache.org/jira/browse/FLINK-26078 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Major > Labels: pull-request-available > > This issue tracks the initial code contribution for the Flink Kubernetes > operator and any questions or concerns that might arise -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] dawidwys commented on a change in pull request #18745: [FLINK-24607] Make OperatorCoordinator closing sequence more robust.
dawidwys commented on a change in pull request #18745:
URL: https://github.com/apache/flink/pull/18745#discussion_r807684936
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/ComponentClosingUtils.java
##
@@ -95,8 +100,94 @@ private ComponentClosingUtils() {}
return future;
}
+/**
+ * A util method that tries to shut down an {@link ExecutorService}
elegantly within the given
+ * timeout. If the executor has not been shut down before it hits timeout
or the thread is
+ * interrupted when waiting for the termination, a forceful shutdown will
be attempted on the
+ * executor.
+ *
+ * @param executor the {@link ExecutorService} to shut down.
+ * @param timeout the timeout duration.
+ * @return true if the given executor has been successfully closed, false
otherwise.
+ */
+@SuppressWarnings("ResultOfMethodCallIgnored")
+public static boolean tryShutdownExecutorElegantly(ExecutorService
executor, Duration timeout) {
+try {
+executor.shutdown();
+executor.awaitTermination(timeout.toMillis(),
TimeUnit.MILLISECONDS);
+} catch (InterruptedException ie) {
+// Let it go.
+}
+if (!executor.isTerminated()) {
+shutdownExecutorForcefully(executor, Duration.ZERO, false);
+}
+return executor.isTerminated();
+}
+
+/**
+ * Shutdown the given executor forcefully within the given timeout. The
method returns if it is
+ * interrupted.
+ *
+ * @param executor the executor to shut down.
+ * @param timeout the timeout duration.
+ * @return true if the given executor is terminated, false otherwise.
+ */
+public static boolean shutdownExecutorForcefully(ExecutorService executor,
Duration timeout) {
+return shutdownExecutorForcefully(executor, timeout, true);
+}
+
+/**
+ * Shutdown the given executor forcefully within the given timeout.
+ *
+ * @param executor the executor to shut down.
+ * @param timeout the timeout duration.
+ * @param interruptable when set to true, the method cannot be
interrupted. Each interruption to
Review comment:
I'd expect it to behave the other way around. If something is
interruptable, I'd expect it can be properly interrupted. The way II understand
it, now if the flag is true (interruptable), you can not interrupt this method.
EDIT: Is it just that the javadoc is wrong?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorContext.java
##
@@ -82,7 +84,7 @@ Licensed to the Apache Software Foundation (ASF) under one
private static final Logger LOG =
LoggerFactory.getLogger(SourceCoordinatorContext.class);
-private final ExecutorService coordinatorExecutor;
+private final ScheduledExecutorService coordinatorExecutor;
Review comment:
nit: Could we create the `coordinatorExecutor` in the ctor? It's a bit
weird now that we close a `coordinatorExecutor` that we are given from the
outside of the class. As far as I checked it's always a new service created in
the `SourceCoordinatorProvider`.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/ComponentClosingUtils.java
##
@@ -95,8 +100,94 @@ private ComponentClosingUtils() {}
return future;
}
+/**
+ * A util method that tries to shut down an {@link ExecutorService}
elegantly within the given
+ * timeout. If the executor has not been shut down before it hits timeout
or the thread is
+ * interrupted when waiting for the termination, a forceful shutdown will
be attempted on the
+ * executor.
+ *
+ * @param executor the {@link ExecutorService} to shut down.
+ * @param timeout the timeout duration.
+ * @return true if the given executor has been successfully closed, false
otherwise.
+ */
+@SuppressWarnings("ResultOfMethodCallIgnored")
+public static boolean tryShutdownExecutorElegantly(ExecutorService
executor, Duration timeout) {
+try {
+executor.shutdown();
+executor.awaitTermination(timeout.toMillis(),
TimeUnit.MILLISECONDS);
+} catch (InterruptedException ie) {
+// Let it go.
+}
+if (!executor.isTerminated()) {
+shutdownExecutorForcefully(executor, Duration.ZERO, false);
+}
+return executor.isTerminated();
+}
+
+/**
+ * Shutdown the given executor forcefully within the given timeout. The
method returns if it is
+ * interrupted.
+ *
+ * @param executor the executor to shut down.
+ * @param timeout the timeout duration.
+ * @return true if the given executor is terminated, false otherwise.
+ */
+public static boolean shutdownExecutorForcefully(ExecutorService executor,
Duration timeout) {
+
[jira] [Commented] (FLINK-26178) Use the same enum for expected and observed jobstate (JobState / JobStatus.state)
[ https://issues.apache.org/jira/browse/FLINK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493129#comment-17493129 ] Gyula Fora commented on FLINK-26178: Done, thanks, will update all the tickets here > Use the same enum for expected and observed jobstate (JobState / > JobStatus.state) > - > > Key: FLINK-26178 > URL: https://issues.apache.org/jira/browse/FLINK-26178 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > We should consolidate these two and maybe even use Flink's own job state enum > here if makes sense -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25963) FLIP-212: Introduce Flink Kubernetes Operator
[ https://issues.apache.org/jira/browse/FLINK-25963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-25963: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > FLIP-212: Introduce Flink Kubernetes Operator > - > > Key: FLINK-25963 > URL: https://issues.apache.org/jira/browse/FLINK-25963 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-212%3A+Introduce+Flink+Kubernetes+Operator] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26178) Use the same enum for expected and observed jobstate (JobState / JobStatus.state)
[ https://issues.apache.org/jira/browse/FLINK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26178: --- Component/s: Kubernetes Operator (was: Deployment / Kubernetes) > Use the same enum for expected and observed jobstate (JobState / > JobStatus.state) > - > > Key: FLINK-26178 > URL: https://issues.apache.org/jira/browse/FLINK-26178 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > We should consolidate these two and maybe even use Flink's own job state enum > here if makes sense -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18798: [FLINK-26177][tests] Disable PulsarSourceITCase rescaling tests temporarily
flinkbot edited a comment on pull request #18798: URL: https://github.com/apache/flink/pull/18798#issuecomment-1041265347 ## CI report: * 7b3b53570a9ac3f4b7159e7b1bf24eb7d29b2649 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31627) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18789: [FLINK-26168][runtime] The judgment of chainable ignores StreamExchangeMode when partitioner is ForwardForConsecutiveHashPartitioner
flinkbot edited a comment on pull request #18789: URL: https://github.com/apache/flink/pull/18789#issuecomment-1040944258 ## CI report: * cdab9f7d140989714263bc2a4e1a0908bb444a12 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31586) * 72851364f75d1b6e40ea1688408c3b85dccd6ec7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18604: [FLINK-25926][Connectors][JDBC] Update org.postgresql:postgresql to 42.3.2
flinkbot edited a comment on pull request #18604: URL: https://github.com/apache/flink/pull/18604#issuecomment-1027886359 ## CI report: * 948f554e347ff5cb332d3273372718012d5b14fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #18604: [FLINK-25926][Connectors][JDBC] Update org.postgresql:postgresql to 42.3.2
MartijnVisser commented on pull request #18604: URL: https://github.com/apache/flink/pull/18604#issuecomment-1041308638 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26184) Migrate StoreSink to Sink V2
[ https://issues.apache.org/jira/browse/FLINK-26184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26184: --- Labels: pull-request-available (was: ) > Migrate StoreSink to Sink V2 > > > Key: FLINK-26184 > URL: https://issues.apache.org/jira/browse/FLINK-26184 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26178) Use the same enum for expected and observed jobstate (JobState / JobStatus.state)
[ https://issues.apache.org/jira/browse/FLINK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493126#comment-17493126 ] Chesnay Schepler commented on FLINK-26178: -- You should be able to add it here: https://issues.apache.org/jira/projects/FLINK?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page You can also do that by just typing the desired component into the tickets component field. > Use the same enum for expected and observed jobstate (JobState / > JobStatus.state) > - > > Key: FLINK-26178 > URL: https://issues.apache.org/jira/browse/FLINK-26178 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Gyula Fora >Priority: Major > > We should consolidate these two and maybe even use Flink's own job state enum > here if makes sense -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink-table-store] JingsongLi opened a new pull request #22: [FLINK-26184] Migrate StoreSink to Sink V2
JingsongLi opened a new pull request #22:
URL: https://github.com/apache/flink-table-store/pull/22
This PR creates a new Sink interface: `GlobalCommittingSink`.
```
/**
* The {@code GlobalCommitter} is responsible for creating and committing an
aggregated committable,
* which we call global committable (see {@link #combine}).
*
* The {@code GlobalCommitter} runs with parallelism equal to 1.
*
* @param The type of information needed to commit data staged by
the sink
* @param The type of the aggregated committable
*/
public interface GlobalCommitter extends AutoCloseable {
/** Find out which global committables need to be retried when
recovering from the failure. */
List filterRecoveredCommittables(List
globalCommittables)
throws IOException;
/** Compute an aggregated committable from a list of committables. */
GlobalCommT combine(long checkpointId, List committables) throws
IOException;
/** Commits the given {@link GlobalCommT}. */
void commit(List globalCommittables) throws IOException,
InterruptedException;
}
```
Because StoreSink committing needs to rely on checkpointId and needs to call
the `filterRecoveredCommittables` interface carefully, a global commit
mechanism similar to SinkV1 is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18798: [FLINK-26177][tests] Disable PulsarSourceITCase rescaling tests temporarily
flinkbot edited a comment on pull request #18798: URL: https://github.com/apache/flink/pull/18798#issuecomment-1041265347 ## CI report: * 7b3b53570a9ac3f4b7159e7b1bf24eb7d29b2649 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31627) * d87a36d6c2290af48b79a325beeeb917fe3501ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18718: [FLINK-25782] [docs] Translate datastream filesystem.md page into Chi…
flinkbot edited a comment on pull request #18718: URL: https://github.com/apache/flink/pull/18718#issuecomment-1035898573 ## CI report: * 8dbb8256563d48a7bdb13489d6670ae6215eebe1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-24246) Bump PulsarClient to 2.9.1 with better transaction management
[ https://issues.apache.org/jira/browse/FLINK-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul resolved FLINK-24246. - Fix Version/s: 1.15.0 Resolution: Fixed > Bump PulsarClient to 2.9.1 with better transaction management > - > > Key: FLINK-24246 > URL: https://issues.apache.org/jira/browse/FLINK-24246 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.14.0, 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0 > > > Pulsar 2.9.1 has been released, the hack for getting TxnID from Pulsar > Transaction instance could be removed after bumping flink-connector-pulsar's > pulsar-client-all to 2.9.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24246) Bump PulsarClient to 2.9.1 with better transaction management
[ https://issues.apache.org/jira/browse/FLINK-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493125#comment-17493125 ] Fabian Paul commented on FLINK-24246: - Merged in master: 1602e4b7d26cf52cea993c410769b7b15a672aff > Bump PulsarClient to 2.9.1 with better transaction management > - > > Key: FLINK-24246 > URL: https://issues.apache.org/jira/browse/FLINK-24246 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.14.0, 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > > Pulsar 2.9.1 has been released, the hack for getting TxnID from Pulsar > Transaction instance could be removed after bumping flink-connector-pulsar's > pulsar-client-all to 2.9.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26020) Unified Pulsar Connector config model
[ https://issues.apache.org/jira/browse/FLINK-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493124#comment-17493124 ] Fabian Paul commented on FLINK-26020: - Merged in master: 36de46dc9cf3144a02a4f99c973bfb6b0db60c7e > Unified Pulsar Connector config model > - > > Key: FLINK-26020 > URL: https://issues.apache.org/jira/browse/FLINK-26020 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > PulsarClient has built-in config model named ClientConfigurationData, > ConsumerConfigurationData and ProducerConfigurationData. We don't want to > expose all the config options. And some config options could conflict with > each other. > We decide to design a new config model based on Flink's Configuration. Which > could provide type checks and better integration with Flink Table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26020) Unified Pulsar Connector config model
[ https://issues.apache.org/jira/browse/FLINK-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul updated FLINK-26020: Affects Version/s: 1.15.0 > Unified Pulsar Connector config model > - > > Key: FLINK-26020 > URL: https://issues.apache.org/jira/browse/FLINK-26020 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > > PulsarClient has built-in config model named ClientConfigurationData, > ConsumerConfigurationData and ProducerConfigurationData. We don't want to > expose all the config options. And some config options could conflict with > each other. > We decide to design a new config model based on Flink's Configuration. Which > could provide type checks and better integration with Flink Table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-26020) Unified Pulsar Connector config model
[ https://issues.apache.org/jira/browse/FLINK-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul resolved FLINK-26020. - Fix Version/s: 1.15.0 Resolution: Fixed > Unified Pulsar Connector config model > - > > Key: FLINK-26020 > URL: https://issues.apache.org/jira/browse/FLINK-26020 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > > PulsarClient has built-in config model named ClientConfigurationData, > ConsumerConfigurationData and ProducerConfigurationData. We don't want to > expose all the config options. And some config options could conflict with > each other. > We decide to design a new config model based on Flink's Configuration. Which > could provide type checks and better integration with Flink Table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-26021) Pulsar topic deduplicated in both sink and source connector
[ https://issues.apache.org/jira/browse/FLINK-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul resolved FLINK-26021. - Fix Version/s: 1.15.0 Resolution: Fixed > Pulsar topic deduplicated in both sink and source connector > --- > > Key: FLINK-26021 > URL: https://issues.apache.org/jira/browse/FLINK-26021 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > > Both topics and partitions are regarded as topics in Pulsar. We have to make > the topic configuration more robust for deduplicating the partitions and > topics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wanglijie95 commented on a change in pull request #18757: [FLINK-25226][doc] Add documentation about the AdaptiveBatchScheduler
wanglijie95 commented on a change in pull request #18757: URL: https://github.com/apache/flink/pull/18757#discussion_r807733497 ## File path: docs/content/docs/deployment/adaptive_batch_scheduler.md ## @@ -0,0 +1,63 @@ +--- +title: Adaptive Batch Scheduler +weight: 5 +type: docs + +--- + + +## Adaptive Batch Scheduler + +The Adaptive Batch Scheduler can automatically decide parallelisms of job vertices for batch jobs. If a job vertex is not set with a parallelism, the scheduler will decide parallelism for the job vertex according to the size of its consumed datasets. This can bring many benefits: +- Batch job users can be relieved from parallelism tuning +- Automatically tuned parallelisms can be vertex level and can better fit consumed datasets which have a varying volume size every day +- Vertices from SQL batch jobs can be assigned with different parallelisms which are automatically tuned Review comment: > What's the target audience? Does regular Flink user supposed to know what the job vertex is? Overall this page feels bit too low level 樂. Thanks for pointing that out. Maybe `stage` is more appropriate？ > On the other hand I don't think that other pages withing this section are all much better in this regard 樂 I'll check the rest content. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18798: [FLINK-26177][tests] Disable PulsarSourceITCase rescaling tests temporarily
flinkbot edited a comment on pull request #18798: URL: https://github.com/apache/flink/pull/18798#issuecomment-1041265347 ## CI report: * 7b3b53570a9ac3f4b7159e7b1bf24eb7d29b2649 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31627) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26021) Pulsar topic deduplicated in both sink and source connector
[ https://issues.apache.org/jira/browse/FLINK-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493123#comment-17493123 ] Fabian Paul commented on FLINK-26021: - Merged in master: a195f729ba4d1ffe657d82888b26ab9db9121ab8 > Pulsar topic deduplicated in both sink and source connector > --- > > Key: FLINK-26021 > URL: https://issues.apache.org/jira/browse/FLINK-26021 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > Both topics and partitions are regarded as topics in Pulsar. We have to make > the topic configuration more robust for deduplicating the partitions and > topics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26023) Create a Pulsar sink config model for matching ProducerConfigurationData
[ https://issues.apache.org/jira/browse/FLINK-26023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493121#comment-17493121 ] Fabian Paul commented on FLINK-26023: - Merged in master: 9bc8b0f37bec419bcdc4b8cdee3abf5320df5399 > Create a Pulsar sink config model for matching ProducerConfigurationData > > > Key: FLINK-26023 > URL: https://issues.apache.org/jira/browse/FLINK-26023 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18791: [FLINK-26169][coordination] There is a typo in the annotation for the WatermarkAlignmentParams class
flinkbot edited a comment on pull request #18791: URL: https://github.com/apache/flink/pull/18791#issuecomment-1041038528 ## CI report: * 5abd99fbc044163e5ca8b4ddfb6c33e55df06b44 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31606) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-26023) Create a Pulsar sink config model for matching ProducerConfigurationData
[ https://issues.apache.org/jira/browse/FLINK-26023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul resolved FLINK-26023. - Resolution: Fixed > Create a Pulsar sink config model for matching ProducerConfigurationData > > > Key: FLINK-26023 > URL: https://issues.apache.org/jira/browse/FLINK-26023 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26178) Use the same enum for expected and observed jobstate (JobState / JobStatus.state)
[ https://issues.apache.org/jira/browse/FLINK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493122#comment-17493122 ] Gyula Fora commented on FLINK-26178: [~chesnay] thank you for jumping in, I was going to propose this. Do you know where to do this? > Use the same enum for expected and observed jobstate (JobState / > JobStatus.state) > - > > Key: FLINK-26178 > URL: https://issues.apache.org/jira/browse/FLINK-26178 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Gyula Fora >Priority: Major > > We should consolidate these two and maybe even use Flink's own job state enum > here if makes sense -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26023) Create a Pulsar sink config model for matching ProducerConfigurationData
[ https://issues.apache.org/jira/browse/FLINK-26023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul updated FLINK-26023: Affects Version/s: 1.15.0 > Create a Pulsar sink config model for matching ProducerConfigurationData > > > Key: FLINK-26023 > URL: https://issues.apache.org/jira/browse/FLINK-26023 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26021) Pulsar topic deduplicated in both sink and source connector
[ https://issues.apache.org/jira/browse/FLINK-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul updated FLINK-26021: Affects Version/s: 1.15.0 > Pulsar topic deduplicated in both sink and source connector > --- > > Key: FLINK-26021 > URL: https://issues.apache.org/jira/browse/FLINK-26021 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > Both topics and partitions are regarded as topics in Pulsar. We have to make > the topic configuration more robust for deduplicating the partitions and > topics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26023) Create a Pulsar sink config model for matching ProducerConfigurationData
[ https://issues.apache.org/jira/browse/FLINK-26023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul updated FLINK-26023: Fix Version/s: 1.15.0 > Create a Pulsar sink config model for matching ProducerConfigurationData > > > Key: FLINK-26023 > URL: https://issues.apache.org/jira/browse/FLINK-26023 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26024) Create a PulsarSerializationSchema for better records serialization
[ https://issues.apache.org/jira/browse/FLINK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul updated FLINK-26024: Affects Version/s: 1.15.0 > Create a PulsarSerializationSchema for better records serialization > --- > > Key: FLINK-26024 > URL: https://issues.apache.org/jira/browse/FLINK-26024 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26163) Refactor FlinkUtils#getEffectiveConfig into smaller pieces
[ https://issues.apache.org/jira/browse/FLINK-26163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493120#comment-17493120 ] Gyula Fora commented on FLINK-26163: Awesome, thanks [~bgeng777] > Refactor FlinkUtils#getEffectiveConfig into smaller pieces > -- > > Key: FLINK-26163 > URL: https://issues.apache.org/jira/browse/FLINK-26163 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Gyula Fora >Assignee: Biao Geng >Priority: Major > > There is currently one very large method dealing with Configuration > management. This logic probably deserves it's own utility class and a few > more modular methods. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18725: [FLINK-25532] Improve Documentation of Flink Docker-Compose
flinkbot edited a comment on pull request #18725: URL: https://github.com/apache/flink/pull/18725#issuecomment-1036065440 ## CI report: * b97354da0a65cd807b532a083f53c1c90b0c8a08 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31240) * 481c5adfee681c53c4d54ba2b2bd5a8746a615ad Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31631) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-26024) Create a PulsarSerializationSchema for better records serialization
[ https://issues.apache.org/jira/browse/FLINK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul resolved FLINK-26024. - Fix Version/s: 1.15.0 Resolution: Fixed > Create a PulsarSerializationSchema for better records serialization > --- > > Key: FLINK-26024 > URL: https://issues.apache.org/jira/browse/FLINK-26024 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26184) Migrate StoreSink to Sink V2
[ https://issues.apache.org/jira/browse/FLINK-26184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-26184: Assignee: Jingsong Lee > Migrate StoreSink to Sink V2 > > > Key: FLINK-26184 > URL: https://issues.apache.org/jira/browse/FLINK-26184 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: table-store-0.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26024) Create a PulsarSerializationSchema for better records serialization
[ https://issues.apache.org/jira/browse/FLINK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493119#comment-17493119 ] Fabian Paul commented on FLINK-26024: - Merged in master: 0e72bfede70a00146f466b3e7491fc0f83eb6c41 > Create a PulsarSerializationSchema for better records serialization > --- > > Key: FLINK-26024 > URL: https://issues.apache.org/jira/browse/FLINK-26024 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26022) Implement at-least-once and exactly-once Pulsar Sink
[ https://issues.apache.org/jira/browse/FLINK-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul updated FLINK-26022: Affects Version/s: 1.15.0 > Implement at-least-once and exactly-once Pulsar Sink > > > Key: FLINK-26022 > URL: https://issues.apache.org/jira/browse/FLINK-26022 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > Support both three types of DeliveryGuarantee in Pulsar sink. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26163) Refactor FlinkUtils#getEffectiveConfig into smaller pieces
[ https://issues.apache.org/jira/browse/FLINK-26163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora reassigned FLINK-26163: -- Assignee: Biao Geng > Refactor FlinkUtils#getEffectiveConfig into smaller pieces > -- > > Key: FLINK-26163 > URL: https://issues.apache.org/jira/browse/FLINK-26163 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Gyula Fora >Assignee: Biao Geng >Priority: Major > > There is currently one very large method dealing with Configuration > management. This logic probably deserves it's own utility class and a few > more modular methods. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-26022) Implement at-least-once and exactly-once Pulsar Sink
[ https://issues.apache.org/jira/browse/FLINK-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul resolved FLINK-26022. - Fix Version/s: 1.15.0 Resolution: Fixed > Implement at-least-once and exactly-once Pulsar Sink > > > Key: FLINK-26022 > URL: https://issues.apache.org/jira/browse/FLINK-26022 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Fix For: 1.15.0 > > > Support both three types of DeliveryGuarantee in Pulsar sink. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26022) Implement at-least-once and exactly-once Pulsar Sink
[ https://issues.apache.org/jira/browse/FLINK-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493118#comment-17493118 ] Fabian Paul commented on FLINK-26022: - Merged in master: 136add5d0c9c5b9b2869a9ee194f78449065b18e > Implement at-least-once and exactly-once Pulsar Sink > > > Key: FLINK-26022 > URL: https://issues.apache.org/jira/browse/FLINK-26022 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > Support both three types of DeliveryGuarantee in Pulsar sink. -- This message was sent by Atlassian Jira (v8.20.1#820001)
