[jira] [Commented] (FLINK-29845) ThroughputCalculator throws java.lang.IllegalArgumentException: Time should be non negative under very low throughput cluster
[
https://issues.apache.org/jira/browse/FLINK-29845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627523#comment-17627523
]
Chesnay Schepler commented on FLINK-29845:
--
Did you actually still see this issue 1.14.6 or a previous version?
> ThroughputCalculator throws java.lang.IllegalArgumentException: Time should
> be non negative under very low throughput cluster
> -
>
> Key: FLINK-29845
> URL: https://issues.apache.org/jira/browse/FLINK-29845
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Runtime / Task
>Affects Versions: 1.14.6
>Reporter: Jingxiao GU
>Priority: Major
>
> Our team are using Flink@1.14 to process data from Kafka.
> It works all fine unless the same job jar with same arguments deployed in an
> environment with{color:#FF} *very low kafka source throughput.*{color}
> The job crashed sometimes with the following Exception and could not be able
> to recover unless we restarted TaskManagers, which is unacceptable for a
> production environment.
> {code:java}
> [2022-10-31T15:33:57.153+08:00] [o.a.f.runtime.taskmanager.Task#cess
> (2/16)#244] - [WARN ] KeyedProcess (2/16)#244
> (b9b54f6445419fc43c4d58fcd95cee82) switched from RUNNING to FAILED with
> failure cause: java.lang.IllegalArgumentException: Time should be non negative
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
> at
> org.apache.flink.runtime.throughput.ThroughputEMA.calculateThroughput(ThroughputEMA.java:44)
> at
> org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:80)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:789)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$4(StreamTask.java:781)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:806)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:758)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> After checking the source code roughly, we found if buffer debloating is
> disabled
> ([https://github.com/apache/flink/blob/release-1.14.6/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L427]
> ), the buffer debloater will still be scheduled
> ([https://github.com/apache/flink/blob/release-1.14.6/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L755]
> ) so that the {{ThrouputCalculator}} keeps calculating the throughput
> ([https://github.com/apache/flink/blob/release-1.14.6/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L789]
> ) which causes the division of zero and seems useless as i suppose.
> Currently, we tried to workaround by setting
> {{taskmanager.network.memory.buffer-debloat.period: 365d}} to avoid the
> buffer debloater being scheduled frequently causing the random crash.
> P.S. We found a bug with similar stacktrace
> https://issues.apache.org/jira/browse/FLINK-25454 which was fixed in 1.14.6.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29812) Remove deprecated Netty API usages
[ https://issues.apache.org/jira/browse/FLINK-29812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29812. Resolution: Fixed master: 03c0f155c2f5198bf1fda35de43afc34a4b12f6e > Remove deprecated Netty API usages > -- > > Key: FLINK-29812 > URL: https://issues.apache.org/jira/browse/FLINK-29812 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / REST >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27341) TaskManager running together with JobManager are bind to 127.0.0.1
[ https://issues.apache.org/jira/browse/FLINK-27341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627128#comment-17627128 ] Chesnay Schepler commented on FLINK-27341: -- [~gaoyunhaii] What makes this so complicated? I thought we just have to drop the loopback address resolution strategy again. > TaskManager running together with JobManager are bind to 127.0.0.1 > -- > > Key: FLINK-27341 > URL: https://issues.apache.org/jira/browse/FLINK-27341 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.15.0, 1.16.0 >Reporter: Yun Gao >Assignee: Chesnay Schepler >Priority: Major > > If some TaskManagers running with JobManager on the same machine while some > other TaskManager not, the TaskManagers running together with JobManager > would bind to localhost or 127.0.01, which makes the Netty connections across > the TaskManagers fail. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-27341) TaskManager running together with JobManager are bind to 127.0.0.1
[ https://issues.apache.org/jira/browse/FLINK-27341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-27341: Assignee: (was: Chesnay Schepler) > TaskManager running together with JobManager are bind to 127.0.0.1 > -- > > Key: FLINK-27341 > URL: https://issues.apache.org/jira/browse/FLINK-27341 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.15.0, 1.16.0 >Reporter: Yun Gao >Priority: Major > > If some TaskManagers running with JobManager on the same machine while some > other TaskManager not, the TaskManagers running together with JobManager > would bind to localhost or 127.0.01, which makes the Netty connections across > the TaskManagers fail. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29693) MiniClusterExtension should respect DEFAULT_PARALLELISM if set
[
https://issues.apache.org/jira/browse/FLINK-29693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29693:
-
Fix Version/s: 1.17.0
> MiniClusterExtension should respect DEFAULT_PARALLELISM if set
> --
>
> Key: FLINK-29693
> URL: https://issues.apache.org/jira/browse/FLINK-29693
> Project: Flink
> Issue Type: Improvement
> Components: Tests
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.1
>
>
> MiniClusterExtension#registerEnv sets the default parallelism of the
> environment to the number of the slots the cluster has.
> This effectively prevents multiple jobs from running on the same MiniCluster
> unless they specify a parallelism via the API.
> This isn't ideal since it means you can't easily mix workloads during testing.
> It would be better if the cluster would check the config for whether
> {{DEFAULT_PARALLELISM}} was set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29834) Clear static Jackson TypeFactory cache on CL release
[ https://issues.apache.org/jira/browse/FLINK-29834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29834: - Description: The Jackson TypeFactory contains a singleton instance that is at times used by Jackson, potentially containing user-classes for longer than necessary. https://github.com/FasterXML/jackson-databind/issues/1363 We could clear this cache whenever a user code CL is being released similar to what was done in BEAM-6460. was: The Jackson TypeFactory contains a singleton instance that is at times used by Jackson, potentially containing user-classes for longer than necessary. https://github.com/FasterXML/jackson-databind/issues/1363 BEAM-6460 We could clear this cache whenever a user code CL is being released. > Clear static Jackson TypeFactory cache on CL release > > > Key: FLINK-29834 > URL: https://issues.apache.org/jira/browse/FLINK-29834 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0, 1.16.1 > > > The Jackson TypeFactory contains a singleton instance that is at times used > by Jackson, potentially containing user-classes for longer than necessary. > https://github.com/FasterXML/jackson-databind/issues/1363 > We could clear this cache whenever a user code CL is being released similar > to what was done in BEAM-6460. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29834) Clear static Jackson TypeFactory cache on CL release
Chesnay Schepler created FLINK-29834: Summary: Clear static Jackson TypeFactory cache on CL release Key: FLINK-29834 URL: https://issues.apache.org/jira/browse/FLINK-29834 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0, 1.16.1 The Jackson TypeFactory contains a singleton instance that is at times used by Jackson, potentially containing user-classes for longer than necessary. https://github.com/FasterXML/jackson-databind/issues/1363 BEAM-6460 We could clear this cache whenever a user code CL is being released. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29803) Table API Scala APIs lack proper source jars
[ https://issues.apache.org/jira/browse/FLINK-29803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29803. Resolution: Fixed master: 0bc9c538ac71249d39a27a0a6cca0ad9a0a87d8f 1.16: e9a3f7ca1002be5d119efce1aad79c7023bc1bf0 1.15: e45e5c3fa7d6c5d7d6b88e8a99cf0028987014a8 > Table API Scala APIs lack proper source jars > > > Key: FLINK-29803 > URL: https://issues.apache.org/jira/browse/FLINK-29803 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.15.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0, 1.15.3, 1.16.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29807) Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
[ https://issues.apache.org/jira/browse/FLINK-29807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29807: - Fix Version/s: 1.17.0 (was: 1.17) > Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < > 1.8.0 > --- > > Key: FLINK-29807 > URL: https://issues.apache.org/jira/browse/FLINK-29807 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > The motivation behind this move is two fold. One reason is that it > complicates our code base unnecessarily and creates confusion on how to > actually implement custom serializers. The immediate reason is that I wanted > to clean up Flink's configuration stack a bit and refactor the > ExecutionConfig class FLINK-29379. This refactor would keep the API > compatibility of the ExecutionConfig, but it would break savepoint > compatibility with snapshots written with some of the old serializers, which > had ExecutionConfig as a field and were serialized in the snapshot. This > issue has been resolved by the introduction of TypeSerializerSnapshot in > Flink 1.7 FLINK-9377, where serializers are no longer part of the snapshot. > TypeSerializerConfigSnapshot has been deprecated and no longer used by > built-in serializers since Flink 1.8 FLINK-9376 and FLINK-11323. Users were > encouraged to migrate to TypeSerializerSnapshot since then with their own > custom serializers. That has been plenty of time for the migration. > This proposal would have the following impact for the users: > 1. we would drop support for recovery from savepoints taken with Flink < > 1.7.0 for all built in types serializers > 2. we would drop support for recovery from savepoints taken with Flink < > 1.8.0 for built in kryo serializers > 3. we would drop support for recovery from savepoints taken with Flink < 1.17 > for custom serializers using deprecated TypeSerializerConfigSnapshot > 1. and 2. would have a simple migration path. Users migrating from those old > savepoints would have to first start his job using a Flink version from the > [1.8, 1.16] range, and take a new savepoint that would be compatible with > Flink 1.17. > 3. This is a bit more problematic, because users would have to first migrate > their own custom serializers to use TypeSerializerSnapshot (using a Flink > version from the [1.8, 1.16]), take a savepoint, and only then migrate to > Flink 1.17. However users had already 4 years to migrate, which in my opinion > has been plenty of time to do so. > *As discussed and vote is currently in progress:* > https://lists.apache.org/thread/x5d0p08pf2wx47njogsgqct0k5rpfrl4 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29646) SQL Gateway should return a simpler error message
[
https://issues.apache.org/jira/browse/FLINK-29646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29646:
-
Fix Version/s: 1.17.0
(was: 1.17)
> SQL Gateway should return a simpler error message
> -
>
> Key: FLINK-29646
> URL: https://issues.apache.org/jira/browse/FLINK-29646
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Gateway
>Affects Versions: 1.17
>Reporter: yuanfenghu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> sql gateway should return simpler exception information
> for example:
> If i execute a sql statement through sql gateway but my statement has
> syntax error ：[ inset into tablea select * from tableb ]
> When I get exception information. The abnormal information returned by the
> server is too redundant to quickly find the Key Information.
> {code:java}
> org.apache.flink.table.gateway.api.utils.SqlGatewayException:
> org.apache.flink.table.gateway.api.utils.SqlGatewayException: Failed to
> fetchResults.
> at
> org.apache.flink.table.gateway.rest.handler.statement.FetchResultsHandler.handleRequest(FetchResultsHandler.java:77)
> at
> org.apache.flink.table.gateway.rest.handler.AbstractSqlGatewayRestHandler.respondToRequest(AbstractSqlGatewayRestHandler.java:84)
> at
> org.apache.flink.table.gateway.rest.handler.AbstractSqlGatewayRestHandler.respondToRequest(AbstractSqlGatewayRestHandler.java:52)
> at
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196)
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83)
> at java.util.Optional.ifPresent(Optional.java:159)
> at
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80)
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115)
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94)
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:210)
> at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>
[jira] [Commented] (FLINK-29809) REST API for running a Flink job
[ https://issues.apache.org/jira/browse/FLINK-29809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626548#comment-17626548 ] Chesnay Schepler commented on FLINK-29809: -- How exactly do you expect the jar to transferred to the Flink server if you're unable to submit it via any existing mean? > REST API for running a Flink job > > > Key: FLINK-29809 > URL: https://issues.apache.org/jira/browse/FLINK-29809 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: Nguyễn Minh Phú >Priority: Major > > When I want to submit a Flink job, I have to run `flink run ...` or submit a > jar via Flink web. But in our production environment, we cannot connect to > the flink server and run the command or submit a jar file via the web. So I > need a REST API to trigger a jar file in Flink server. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29611) Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest
[ https://issues.apache.org/jira/browse/FLINK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29611. Resolution: Won't Fix > Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest > -- > > Key: FLINK-29611 > URL: https://issues.apache.org/jira/browse/FLINK-29611 > Project: Flink > Issue Type: Bug >Reporter: Sopan Phaltankar >Priority: Minor > Labels: pull-request-available > > The test > _org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport_ > has the following failure: > Failures: > [ERROR] CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport:74 > Wrong Side Output: arrays first differed at element [0]; expected: 15 : 9:key.6->6> but was:5> > I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to > find this flaky test. > Command: mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest='Fully > Qualified Test Name' > I analyzed the assertion failure and found that the root cause is because the > test method calls ctx.getBroadcastState(STATE_DESCRIPTOR).immutableEntries() > which calls the entrySet() method of the underlying HashMap. entrySet() > returns the entries in a non-deterministic way, causing the test to be flaky. > The fix would be to change _HashMap_ to _LinkedHashMap_ where the Map is > getting initialized. > On further analysis, it was found that the Map is getting initialized on line > 53 of org.apache.flink.runtime.state.HeapBroadcastState class. > After changing from HashMap to LinkedHashMap, the above test is passing. > Edit: Upon making this change and running the CI, it was found that the tests > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchKeyedBroadcastExecution > and > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchBroadcastExecution > were failing. Upon further investigation, I found that these tests were also > flaky and depended on the earlier made change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29611) Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest
[ https://issues.apache.org/jira/browse/FLINK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626538#comment-17626538 ] Chesnay Schepler commented on FLINK-29611: -- Sorry but we are busy enough without chasing theoretical test instabilities. Closing this issue and PR. > Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest > -- > > Key: FLINK-29611 > URL: https://issues.apache.org/jira/browse/FLINK-29611 > Project: Flink > Issue Type: Bug >Reporter: Sopan Phaltankar >Priority: Minor > Labels: pull-request-available > > The test > _org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport_ > has the following failure: > Failures: > [ERROR] CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport:74 > Wrong Side Output: arrays first differed at element [0]; expected: 15 : 9:key.6->6> but was:5> > I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to > find this flaky test. > Command: mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest='Fully > Qualified Test Name' > I analyzed the assertion failure and found that the root cause is because the > test method calls ctx.getBroadcastState(STATE_DESCRIPTOR).immutableEntries() > which calls the entrySet() method of the underlying HashMap. entrySet() > returns the entries in a non-deterministic way, causing the test to be flaky. > The fix would be to change _HashMap_ to _LinkedHashMap_ where the Map is > getting initialized. > On further analysis, it was found that the Map is getting initialized on line > 53 of org.apache.flink.runtime.state.HeapBroadcastState class. > After changing from HashMap to LinkedHashMap, the above test is passing. > Edit: Upon making this change and running the CI, it was found that the tests > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchKeyedBroadcastExecution > and > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchBroadcastExecution > were failing. Upon further investigation, I found that these tests were also > flaky and depended on the earlier made change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29812) Remove deprecated Netty API usages
Chesnay Schepler created FLINK-29812: Summary: Remove deprecated Netty API usages Key: FLINK-29812 URL: https://issues.apache.org/jira/browse/FLINK-29812 Project: Flink Issue Type: Technical Debt Components: Runtime / REST Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29813) Remove deprecated Curator API usages
Chesnay Schepler created FLINK-29813: Summary: Remove deprecated Curator API usages Key: FLINK-29813 URL: https://issues.apache.org/jira/browse/FLINK-29813 Project: Flink Issue Type: Technical Debt Components: Runtime / Coordination Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29803) Table API Scala APIs lack proper source jars
Chesnay Schepler created FLINK-29803: Summary: Table API Scala APIs lack proper source jars Key: FLINK-29803 URL: https://issues.apache.org/jira/browse/FLINK-29803 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.15.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0, 1.15.3, 1.16.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29716) Separate slf4j jar in the lib folder from the distribution
[
https://issues.apache.org/jira/browse/FLINK-29716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622313#comment-17622313
]
Chesnay Schepler commented on FLINK-29716:
--
The log4j jars are bundled separately to allow users to switch logging backends
or even go back to log4j1.
There's no such requirement for slf4j.
Should we upgrade to slf4j 2.x then that is just what Flink will require like
any other direct dependency.
Given that it is a compile dependency (unlike log4j) replacing it isn't as
trivially safe as log4j is; for example if we were to start using the new
fluent logging API then replacing it with slf4j v1 is just not an option.
> Separate slf4j jar in the lib folder from the distribution
> --
>
> Key: FLINK-29716
> URL: https://issues.apache.org/jira/browse/FLINK-29716
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.15.2
>Reporter: Alexis Sarda-Espinosa
>Priority: Major
>
> Flink's binary distribution includes several jars under the {{lib}} folder,
> which has individual jars for all log4j artifacts. This makes it relatively
> easy to swap out those logging jars when necessary, for example when critical
> vulnerabilities are found (as was recently the case).
> With SLF4J 2.+, some breaking changes mean that many implementations are not
> directly backwards compatible, see for example the [notes for
> log4j2|https://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html].
> This means that, in the future, if swapping logging jars were necessary, the
> SLF4J jar might have to be changed as well.
> Right now the SLF4J jar is not included separately in the distribution, I
> believe it's packed inside the {{flink-dist}} jar, although I'm not sure. It
> would be better to separate that as it is done for the default log4j2 jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29472) Create shared release scripts
[ https://issues.apache.org/jira/browse/FLINK-29472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622145#comment-17622145 ] Chesnay Schepler commented on FLINK-29472: -- I've created a [repository|https://github.com/apache/flink-connector-shared-utils/] but there's some permission issue with GitHub. Once that is resolved I'll publish the scripts I wrote into said repository. > Create shared release scripts > - > > Key: FLINK-29472 > URL: https://issues.apache.org/jira/browse/FLINK-29472 > Project: Flink > Issue Type: Sub-task > Components: Release System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > > With the versioning & branching model being identical we should be able to > share all release scripts. Put them into a central location that projects > can rely on (e.g., via a git submodule). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29706) Remove japicmp dependency bumps
[ https://issues.apache.org/jira/browse/FLINK-29706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29706. Resolution: Fixed master: b8672a230a3d34e09fddc7f506e090910e2d202e > Remove japicmp dependency bumps > --- > > Key: FLINK-29706 > URL: https://issues.apache.org/jira/browse/FLINK-29706 > Project: Flink > Issue Type: Technical Debt > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > Way back when we worked on Java 11 support we bumped several dependencies > from japicmp. > These are no longer required for the latest version that we're using. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29582) SavepointWriter should be usable without any transformation
[ https://issues.apache.org/jira/browse/FLINK-29582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29582. Resolution: Fixed master: d17ceaf2f8cb0a36c2b629b9f6f87e020ce79395 > SavepointWriter should be usable without any transformation > --- > > Key: FLINK-29582 > URL: https://issues.apache.org/jira/browse/FLINK-29582 > Project: Flink > Issue Type: Improvement > Components: API / State Processor >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > The SavepointWriter of the state processor API currently enforces at least > one transformation to be defined be the user. > This is an irritating limitation; this means you can't use the API to delete > a state or use the new uid remapping function from FLINK-29457 without > specifying some dummy transformation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29693) MiniClusterExtension should respect DEFAULT_PARALLELISM if set
[
https://issues.apache.org/jira/browse/FLINK-29693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29693:
-
Fix Version/s: 1.16.1
(was: 1.16.0)
> MiniClusterExtension should respect DEFAULT_PARALLELISM if set
> --
>
> Key: FLINK-29693
> URL: https://issues.apache.org/jira/browse/FLINK-29693
> Project: Flink
> Issue Type: Improvement
> Components: Tests
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.1
>
>
> MiniClusterExtension#registerEnv sets the default parallelism of the
> environment to the number of the slots the cluster has.
> This effectively prevents multiple jobs from running on the same MiniCluster
> unless they specify a parallelism via the API.
> This isn't ideal since it means you can't easily mix workloads during testing.
> It would be better if the cluster would check the config for whether
> {{DEFAULT_PARALLELISM}} was set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29706) Remove japicmp dependency bumps
Chesnay Schepler created FLINK-29706: Summary: Remove japicmp dependency bumps Key: FLINK-29706 URL: https://issues.apache.org/jira/browse/FLINK-29706 Project: Flink Issue Type: Technical Debt Components: Build System Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0 Way back when we worked on Java 11 support we bumped several dependencies from japicmp. These are no longer required for the latest version that we're using. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29217) CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint failed with AssertionFailedError
[
https://issues.apache.org/jira/browse/FLINK-29217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621040#comment-17621040
]
Chesnay Schepler commented on FLINK-29217:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42281=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8
> CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint
> failed with AssertionFailedError
> -
>
> Key: FLINK-29217
> URL: https://issues.apache.org/jira/browse/FLINK-29217
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Xingbo Huang
>Assignee: Yunfeng Zhou
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-09-07T02:00:50.2507464Z Sep 07 02:00:50 [ERROR]
> org.apache.flink.streaming.runtime.tasks.CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint
> Time elapsed: 2.137 s <<< FAILURE!
> 2022-09-07T02:00:50.2508673Z Sep 07 02:00:50
> org.opentest4j.AssertionFailedError:
> 2022-09-07T02:00:50.2509309Z Sep 07 02:00:50
> 2022-09-07T02:00:50.2509945Z Sep 07 02:00:50 Expecting value to be false but
> was true
> 2022-09-07T02:00:50.2511950Z Sep 07 02:00:50 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 2022-09-07T02:00:50.2513254Z Sep 07 02:00:50 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 2022-09-07T02:00:50.2514621Z Sep 07 02:00:50 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 2022-09-07T02:00:50.2516342Z Sep 07 02:00:50 at
> org.apache.flink.streaming.runtime.tasks.CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint(CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.java:173)
> 2022-09-07T02:00:50.2517852Z Sep 07 02:00:50 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-09-07T02:00:50.251Z Sep 07 02:00:50 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-09-07T02:00:50.2520065Z Sep 07 02:00:50 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-09-07T02:00:50.2521153Z Sep 07 02:00:50 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-09-07T02:00:50.2522747Z Sep 07 02:00:50 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-09-07T02:00:50.2523973Z Sep 07 02:00:50 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-09-07T02:00:50.2525158Z Sep 07 02:00:50 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-09-07T02:00:50.2526347Z Sep 07 02:00:50 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-09-07T02:00:50.2527525Z Sep 07 02:00:50 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-09-07T02:00:50.2528646Z Sep 07 02:00:50 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-09-07T02:00:50.2529708Z Sep 07 02:00:50 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-09-07T02:00:50.2530744Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-09-07T02:00:50.2532008Z Sep 07 02:00:50 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-09-07T02:00:50.2533137Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-09-07T02:00:50.2544265Z Sep 07 02:00:50 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-09-07T02:00:50.2545595Z Sep 07 02:00:50 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-09-07T02:00:50.2546782Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-09-07T02:00:50.2547810Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-09-07T02:00:50.2548890Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-09-07T02:00:50.2549932Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-09-07T02:00:50.2550933Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-09-07T02:00:50.2552325Z Sep 07 02:00:50 at
>
[jira] [Reopened] (FLINK-29217) CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint failed with AssertionFailedError
[
https://issues.apache.org/jira/browse/FLINK-29217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reopened FLINK-29217:
--
> CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint
> failed with AssertionFailedError
> -
>
> Key: FLINK-29217
> URL: https://issues.apache.org/jira/browse/FLINK-29217
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Xingbo Huang
>Assignee: Yunfeng Zhou
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-09-07T02:00:50.2507464Z Sep 07 02:00:50 [ERROR]
> org.apache.flink.streaming.runtime.tasks.CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint
> Time elapsed: 2.137 s <<< FAILURE!
> 2022-09-07T02:00:50.2508673Z Sep 07 02:00:50
> org.opentest4j.AssertionFailedError:
> 2022-09-07T02:00:50.2509309Z Sep 07 02:00:50
> 2022-09-07T02:00:50.2509945Z Sep 07 02:00:50 Expecting value to be false but
> was true
> 2022-09-07T02:00:50.2511950Z Sep 07 02:00:50 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 2022-09-07T02:00:50.2513254Z Sep 07 02:00:50 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 2022-09-07T02:00:50.2514621Z Sep 07 02:00:50 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 2022-09-07T02:00:50.2516342Z Sep 07 02:00:50 at
> org.apache.flink.streaming.runtime.tasks.CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.testConcurrentCheckpoint(CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase.java:173)
> 2022-09-07T02:00:50.2517852Z Sep 07 02:00:50 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-09-07T02:00:50.251Z Sep 07 02:00:50 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-09-07T02:00:50.2520065Z Sep 07 02:00:50 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-09-07T02:00:50.2521153Z Sep 07 02:00:50 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-09-07T02:00:50.2522747Z Sep 07 02:00:50 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-09-07T02:00:50.2523973Z Sep 07 02:00:50 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-09-07T02:00:50.2525158Z Sep 07 02:00:50 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-09-07T02:00:50.2526347Z Sep 07 02:00:50 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-09-07T02:00:50.2527525Z Sep 07 02:00:50 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-09-07T02:00:50.2528646Z Sep 07 02:00:50 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-09-07T02:00:50.2529708Z Sep 07 02:00:50 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-09-07T02:00:50.2530744Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-09-07T02:00:50.2532008Z Sep 07 02:00:50 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-09-07T02:00:50.2533137Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-09-07T02:00:50.2544265Z Sep 07 02:00:50 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-09-07T02:00:50.2545595Z Sep 07 02:00:50 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-09-07T02:00:50.2546782Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-09-07T02:00:50.2547810Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-09-07T02:00:50.2548890Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-09-07T02:00:50.2549932Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-09-07T02:00:50.2550933Z Sep 07 02:00:50 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-09-07T02:00:50.2552325Z Sep 07 02:00:50 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-09-07T02:00:50.2553660Z Sep 07 02:00:50 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2022-09-07T02:00:50.2554661Z Sep 07 02:00:50 at
>
[jira] [Closed] (FLINK-29691) Adjust org.apache.flink.test.junit5.MiniClusterExtension#registerEnv to honor DEFAULT_PARALLELISM if set
[ https://issues.apache.org/jira/browse/FLINK-29691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29691. Resolution: Duplicate > Adjust org.apache.flink.test.junit5.MiniClusterExtension#registerEnv to honor > DEFAULT_PARALLELISM if set > - > > Key: FLINK-29691 > URL: https://issues.apache.org/jira/browse/FLINK-29691 > Project: Flink > Issue Type: Technical Debt > Components: Tests >Reporter: Martijn Visser >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29693) MiniClusterExtension should respect DEFAULT_PARALLELISM if set
[
https://issues.apache.org/jira/browse/FLINK-29693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-29693.
Resolution: Fixed
master: a6db6ee5d0d6e9b50c6d110793e2efbd0d57cc38
1.16: 1ed2494cf570e6e137f870b7d4a13ef18a9c381c
> MiniClusterExtension should respect DEFAULT_PARALLELISM if set
> --
>
> Key: FLINK-29693
> URL: https://issues.apache.org/jira/browse/FLINK-29693
> Project: Flink
> Issue Type: Improvement
> Components: Tests
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> MiniClusterExtension#registerEnv sets the default parallelism of the
> environment to the number of the slots the cluster has.
> This effectively prevents multiple jobs from running on the same MiniCluster
> unless they specify a parallelism via the API.
> This isn't ideal since it means you can't easily mix workloads during testing.
> It would be better if the cluster would check the config for whether
> {{DEFAULT_PARALLELISM}} was set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29659) Deduplicate SavepointWriter factory method code
[ https://issues.apache.org/jira/browse/FLINK-29659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29659. Resolution: Fixed master: 8feddfbec21b3f3a2a5ddbea06e9be54b43a9ca0 > Deduplicate SavepointWriter factory method code > --- > > Key: FLINK-29659 > URL: https://issues.apache.org/jira/browse/FLINK-29659 > Project: Flink > Issue Type: Technical Debt > Components: API / State Processor >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29693) MiniClusterExtension should respect DEFAULT_PARALLELISM if set
Chesnay Schepler created FLINK-29693:
Summary: MiniClusterExtension should respect DEFAULT_PARALLELISM
if set
Key: FLINK-29693
URL: https://issues.apache.org/jira/browse/FLINK-29693
Project: Flink
Issue Type: Improvement
Components: Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.16.0
MiniClusterExtension#registerEnv sets the default parallelism of the
environment to the number of the slots the cluster has.
This effectively prevents multiple jobs from running on the same MiniCluster
unless they specify a parallelism via the API.
This isn't ideal since it means you can't easily mix workloads during testing.
It would be better if the cluster would check the config for whether
{{DEFAULT_PARALLELISM}} was set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27721) Slack: set up archive
[ https://issues.apache.org/jira/browse/FLINK-27721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619422#comment-17619422 ] Chesnay Schepler commented on FLINK-27721: -- oooh, ok that changes things a bit. Then +1 to use linen.dev > Slack: set up archive > - > > Key: FLINK-27721 > URL: https://issues.apache.org/jira/browse/FLINK-27721 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-27721) Slack: set up archive
[ https://issues.apache.org/jira/browse/FLINK-27721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619348#comment-17619348 ] Chesnay Schepler edited comment on FLINK-27721 at 10/18/22 8:32 AM: ??Slack is storing all data for us, the risk is mostly around loosing URLs, once people start linking into the linen.dev archive.?? I thought Slack doesn't since we're on a free plan? was (Author: zentol): > I agree that linen is a fairly new service, but since Slack is storing all > data for us, the risk is mostly around loosing URLs, once people start > linking into the linen.dev archive. I thought Slack doesn't since we're on a free plan? > Slack: set up archive > - > > Key: FLINK-27721 > URL: https://issues.apache.org/jira/browse/FLINK-27721 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27721) Slack: set up archive
[ https://issues.apache.org/jira/browse/FLINK-27721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619348#comment-17619348 ] Chesnay Schepler commented on FLINK-27721: -- > I agree that linen is a fairly new service, but since Slack is storing all > data for us, the risk is mostly around loosing URLs, once people start > linking into the linen.dev archive. I thought Slack doesn't since we're on a free plan? > Slack: set up archive > - > > Key: FLINK-27721 > URL: https://issues.apache.org/jira/browse/FLINK-27721 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-20873) Upgrade Calcite version to 1.27
[ https://issues.apache.org/jira/browse/FLINK-20873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reopened FLINK-20873: -- > Upgrade Calcite version to 1.27 > --- > > Key: FLINK-20873 > URL: https://issues.apache.org/jira/browse/FLINK-20873 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Table SQL / Planner >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > > The following files should be removed from the Flink code base during an > upgrade: > - org.apache.calcite.rex.RexSimplify > - org.apache.calcite.sql.SqlMatchRecognize > - org.apache.calcite.sql.SqlTableRef > - org.apache.calcite.sql2rel.RelDecorrelator > - org.apache.flink.table.planner.functions.sql.SqlJsonObjectFunction (added > in FLINK-16203) > - Adopt calcite's behaviour and add SQL tests once > [https://github.com/apache/calcite/pull/2555] is merged, (check FLINK-24576 ) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29629) FlameGraph is empty for Legacy Source Threads
[
https://issues.apache.org/jira/browse/FLINK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17618831#comment-17618831
]
Chesnay Schepler commented on FLINK-29629:
--
We shouldn't invest additional resources into legacy APIs.
> FlameGraph is empty for Legacy Source Threads
> -
>
> Key: FLINK-29629
> URL: https://issues.apache.org/jira/browse/FLINK-29629
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Web Frontend
>Reporter: Peter Vary
>Priority: Major
>
> Thread dump gets the stack trace for the {{Custom Source}} thread, but this
> thread is always in {{TIMED_WAITING}}:
> {code}
> "Source: Custom Source -> A random source (1/2)#0" ...
>java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
> - parking to wait for <0xea775750> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos()
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take()
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:335)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
> [..]
> {code}
> The actual code is run in the {{Legacy Source Thread}}:
> {code}
> "Legacy Source Thread - Source: Custom Source -> A random source (1/2)#0" ...
>java.lang.Thread.State: RUNNABLE
> {code}
> This causes the WebUI FlameGraph to be empty of any useful data.
> This is an example code to reproduce:
> {code}
> DataStream inputStream = env.addSource(new
> RandomRecordSource(recordSize));
> inputStream = inputStream.map(new CounterMapper());
> FlinkSink.forRowData(inputStream).tableLoader(loader).append();
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29659) Deduplicate SavepointWriter factory method code
Chesnay Schepler created FLINK-29659: Summary: Deduplicate SavepointWriter factory method code Key: FLINK-29659 URL: https://issues.apache.org/jira/browse/FLINK-29659 Project: Flink Issue Type: Technical Debt Components: API / State Processor Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27101) Add REST API to manually trigger checkpoints
[ https://issues.apache.org/jira/browse/FLINK-27101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-27101: - Summary: Add REST API to manually trigger checkpoints (was: Periodically break the chain of incremental checkpoint (trigger checkpoints via REST API)) > Add REST API to manually trigger checkpoints > > > Key: FLINK-27101 > URL: https://issues.apache.org/jira/browse/FLINK-27101 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing, Runtime / REST >Reporter: Steven Zhen Wu >Assignee: Jiale Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > Incremental checkpoint is almost a must for large-state jobs. It greatly > reduces the bytes uploaded to DFS per checkpoint. However, there are a few > implications from incremental checkpoint that are problematic for production > operations. Will use S3 as an example DFS for the rest of description. > 1. Because there is no way to deterministically know how far back the > incremental checkpoint can refer to files uploaded to S3, it is very > difficult to set S3 bucket/object TTL. In one application, we have observed > Flink checkpoint referring to files uploaded over 6 months ago. S3 TTL can > corrupt the Flink checkpoints. > S3 TTL is important for a few reasons > - purge orphaned files (like external checkpoints from previous deployments) > to keep the storage cost in check. This problem can be addressed by > implementing proper garbage collection (similar to JVM) by traversing the > retained checkpoints from all jobs and traverse the file references. But that > is an expensive solution from engineering cost perspective. > - Security and privacy. E.g., there may be requirement that Flink state can't > keep the data for more than some duration threshold (hours/days/weeks). > Application is expected to purge keys to satisfy the requirement. However, > with incremental checkpoint and how deletion works in RocksDB, it is hard to > set S3 TTL to purge S3 files. Even though those old S3 files don't contain > live keys, they may still be referrenced by retained Flink checkpoints. > 2. Occasionally, corrupted checkpoint files (on S3) are observed. As a > result, restoring from checkpoint failed. With incremental checkpoint, it > usually doesn't help to try other older checkpoints, because they may refer > to the same corrupted file. It is unclear whether the corruption happened > before or during S3 upload. This risk can be mitigated with periodical > savepoints. > It all boils down to periodical full snapshot (checkpoint or savepoint) to > deterministically break the chain of incremental checkpoints. Search the jira > history, the behavior that FLINK-23949 [1] trying to fix is actually close to > what we would need here. > There are a few options > 1. Periodically trigger savepoints (via control plane). This is actually not > a bad practice and might be appealing to some people. The problem is that it > requires a job deployment to break the chain of incremental checkpoint. > periodical job deployment may sound hacky. If we make the behavior of full > checkpoint after a savepoint (fixed in FLINK-23949) configurable, it might be > an acceptable compromise. The benefit is that no job deployment is required > after savepoints. > 2. Build the feature in Flink incremental checkpoint. Periodically (with some > cron style config) trigger a full checkpoint to break the incremental chain. > If the full checkpoint failed (due to whatever reason), the following > checkpoints should attempt full checkpoint as well until one successful full > checkpoint is completed. > 3. For the security/privacy requirement, the main thing is to apply > compaction on the deleted keys. That could probably avoid references to the > old files. Is there any RocksDB compation can achieve full compaction of > removing old delete markers. Recent delete markers are fine > [1] https://issues.apache.org/jira/browse/FLINK-23949 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27101) Periodically break the chain of incremental checkpoint (trigger checkpoints via REST API)
[ https://issues.apache.org/jira/browse/FLINK-27101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-27101. Fix Version/s: 1.17.0 Resolution: Fixed master: 5d66e82915eace9342c175163b17f610bfbf7fa4 > Periodically break the chain of incremental checkpoint (trigger checkpoints > via REST API) > - > > Key: FLINK-27101 > URL: https://issues.apache.org/jira/browse/FLINK-27101 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing, Runtime / REST >Reporter: Steven Zhen Wu >Assignee: Jiale Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > Incremental checkpoint is almost a must for large-state jobs. It greatly > reduces the bytes uploaded to DFS per checkpoint. However, there are a few > implications from incremental checkpoint that are problematic for production > operations. Will use S3 as an example DFS for the rest of description. > 1. Because there is no way to deterministically know how far back the > incremental checkpoint can refer to files uploaded to S3, it is very > difficult to set S3 bucket/object TTL. In one application, we have observed > Flink checkpoint referring to files uploaded over 6 months ago. S3 TTL can > corrupt the Flink checkpoints. > S3 TTL is important for a few reasons > - purge orphaned files (like external checkpoints from previous deployments) > to keep the storage cost in check. This problem can be addressed by > implementing proper garbage collection (similar to JVM) by traversing the > retained checkpoints from all jobs and traverse the file references. But that > is an expensive solution from engineering cost perspective. > - Security and privacy. E.g., there may be requirement that Flink state can't > keep the data for more than some duration threshold (hours/days/weeks). > Application is expected to purge keys to satisfy the requirement. However, > with incremental checkpoint and how deletion works in RocksDB, it is hard to > set S3 TTL to purge S3 files. Even though those old S3 files don't contain > live keys, they may still be referrenced by retained Flink checkpoints. > 2. Occasionally, corrupted checkpoint files (on S3) are observed. As a > result, restoring from checkpoint failed. With incremental checkpoint, it > usually doesn't help to try other older checkpoints, because they may refer > to the same corrupted file. It is unclear whether the corruption happened > before or during S3 upload. This risk can be mitigated with periodical > savepoints. > It all boils down to periodical full snapshot (checkpoint or savepoint) to > deterministically break the chain of incremental checkpoints. Search the jira > history, the behavior that FLINK-23949 [1] trying to fix is actually close to > what we would need here. > There are a few options > 1. Periodically trigger savepoints (via control plane). This is actually not > a bad practice and might be appealing to some people. The problem is that it > requires a job deployment to break the chain of incremental checkpoint. > periodical job deployment may sound hacky. If we make the behavior of full > checkpoint after a savepoint (fixed in FLINK-23949) configurable, it might be > an acceptable compromise. The benefit is that no job deployment is required > after savepoints. > 2. Build the feature in Flink incremental checkpoint. Periodically (with some > cron style config) trigger a full checkpoint to break the incremental chain. > If the full checkpoint failed (due to whatever reason), the following > checkpoints should attempt full checkpoint as well until one successful full > checkpoint is completed. > 3. For the security/privacy requirement, the main thing is to apply > compaction on the deleted keys. That could probably avoid references to the > old files. Is there any RocksDB compation can achieve full compaction of > removing old delete markers. Recent delete markers are fine > [1] https://issues.apache.org/jira/browse/FLINK-23949 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29502) Update the Hadoop implementation for filesystems to 3.3.4
[ https://issues.apache.org/jira/browse/FLINK-29502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29502. Fix Version/s: 1.17.0 Resolution: Fixed master: 62e48fc40bb510c5767b87054c870515b6680162 > Update the Hadoop implementation for filesystems to 3.3.4 > - > > Key: FLINK-29502 > URL: https://issues.apache.org/jira/browse/FLINK-29502 > Project: Flink > Issue Type: Technical Debt > Components: FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > Flink currently uses Hadoop version 3.3.2 for the Flink filesystem > implementations. Upgrading this to version 3.3.4 will resolve some CVEs like > https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-25168 (which Flink is > not affected by) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-24999) flink-python doesn't work on Java 17
[
https://issues.apache.org/jira/browse/FLINK-24999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17616950#comment-17616950
]
Chesnay Schepler commented on FLINK-24999:
--
What's the state here?
> flink-python doesn't work on Java 17
>
>
> Key: FLINK-24999
> URL: https://issues.apache.org/jira/browse/FLINK-24999
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Chesnay Schepler
>Assignee: Luning Wang
>Priority: Major
>
> {code:java}
> java.lang.UnsupportedOperationException: sun.misc.Unsafe or
> java.nio.DirectByteBuffer.(long, int) not available
> at
> io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:490)
> at io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:257)
> at io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:247)
> at io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:248)
> at
> org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:228)
> at
> org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:242)
> at
> org.apache.arrow.vector.ipc.ArrowWriter.writeRecordBatch(ArrowWriter.java:132)
> at org.apache.arrow.vector.ipc.ArrowWriter.writeBatch(ArrowWriter.java:120)
> at
> org.apache.flink.table.runtime.arrow.ArrowUtilsTest.testReadArrowBatches(ArrowUtilsTest.java:389)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29548) Remove deprecated MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-29548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29548: - Summary: Remove deprecated MiniClusterResource (was: Remove deprecated class files of the 'flink-test-utils' module.) > Remove deprecated MiniClusterResource > - > > Key: FLINK-29548 > URL: https://issues.apache.org/jira/browse/FLINK-29548 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > Labels: pull-request-available > Fix For: 1.17.0 > > > Remove > > flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/MiniClusterResource.java > flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/MiniClusterResourceConfiguration.java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29548) Remove deprecated MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-29548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29548: - Issue Type: Technical Debt (was: Improvement) > Remove deprecated MiniClusterResource > - > > Key: FLINK-29548 > URL: https://issues.apache.org/jira/browse/FLINK-29548 > Project: Flink > Issue Type: Technical Debt > Components: Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > Labels: pull-request-available > Fix For: 1.17.0 > > > Remove > > flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/MiniClusterResource.java > flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/MiniClusterResourceConfiguration.java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29548) Remove deprecated class files of the 'flink-test-utils' module.
[ https://issues.apache.org/jira/browse/FLINK-29548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29548. Release Note: The deprecated MiniClusterResource in flink-test-utils has been removed. The MiniClusterWithClientResource is a drop-in replacement. Assignee: RocMarshal Resolution: Fixed master: 7a135c299f13c62832b93d7ceadb91663c7d1b53 > Remove deprecated class files of the 'flink-test-utils' module. > --- > > Key: FLINK-29548 > URL: https://issues.apache.org/jira/browse/FLINK-29548 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > Labels: pull-request-available > Fix For: 1.17.0 > > > Remove > > flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/MiniClusterResource.java > flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/MiniClusterResourceConfiguration.java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29570) Bump org.jsoup:jsoup to v1.15.3
[ https://issues.apache.org/jira/browse/FLINK-29570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29570. Fix Version/s: 1.17.0 Resolution: Fixed master: 2a00737a1a68e7d4b16525042a78ee75ae234ecc > Bump org.jsoup:jsoup to v1.15.3 > --- > > Key: FLINK-29570 > URL: https://issues.apache.org/jira/browse/FLINK-29570 > Project: Flink > Issue Type: Technical Debt > Components: Documentation >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Fix For: 1.17.0 > > > Bump JSoup to avoid getting flagged for CVE-2022-36033 (which doesn't affect > Flink directly) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21708) CloserCleaner warning
[
https://issues.apache.org/jira/browse/FLINK-21708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17616412#comment-17616412
]
Chesnay Schepler commented on FLINK-21708:
--
[~tapan_halani] It's not possible to hide this warning; the JDK forces it to be
logged at least once.
> CloserCleaner warning
> -
>
> Key: FLINK-21708
> URL: https://issues.apache.org/jira/browse/FLINK-21708
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: Avi Levi
>Priority: Minor
>
> Getting this warning :
> {noformat}
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.flink.api.java.ClosureCleaner
> (file:/Users/Foouser/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/flink/flink-core/1.12.2/flink-core-1.12.2.jar)
> to field java.lang.String.value
> WARNING: Please consider reporting this to the maintainers of
> org.apache.flink.api.java.ClosureCleaner{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-28768) Update JUnit5 to v5.9.0
[ https://issues.apache.org/jira/browse/FLINK-28768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-28768. Fix Version/s: 1.17.0 Resolution: Fixed master: 51fc20db30d001a95de95b3b9993eeb06f558f6c > Update JUnit5 to v5.9.0 > --- > > Key: FLINK-28768 > URL: https://issues.apache.org/jira/browse/FLINK-28768 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Minor > Labels: pull-request-available, stale-assigned > Fix For: 1.17.0 > > > Junit 5.9.0 is released > with release notes > https://junit.org/junit5/docs/current/release-notes/#release-notes-5.9.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28768) Update JUnit5 to v5.9.1
[ https://issues.apache.org/jira/browse/FLINK-28768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-28768: - Summary: Update JUnit5 to v5.9.1 (was: Update JUnit5 to v5.9.0) > Update JUnit5 to v5.9.1 > --- > > Key: FLINK-28768 > URL: https://issues.apache.org/jira/browse/FLINK-28768 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Minor > Labels: pull-request-available, stale-assigned > Fix For: 1.17.0 > > > Junit 5.9.0 is released > with release notes > https://junit.org/junit5/docs/current/release-notes/#release-notes-5.9.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29510) Add NoticeFileChecker tests
[ https://issues.apache.org/jira/browse/FLINK-29510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29510: - Fix Version/s: 1.16.0 (was: 1.17.0) > Add NoticeFileChecker tests > --- > > Key: FLINK-29510 > URL: https://issues.apache.org/jira/browse/FLINK-29510 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > The NoticeFileChecker is too important to not be covered by tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29510) Add NoticeFileChecker tests
[ https://issues.apache.org/jira/browse/FLINK-29510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613980#comment-17613980 ] Chesnay Schepler edited comment on FLINK-29510 at 10/12/22 9:51 AM: master: 40f0540a3fa7892e18d897eb35cbdbd091d4fe19..42b00efdc7c78aa3f01a6e0f6f3cf0fdfd093582 1.16: a76d8b8b5e6d39f08a0eb4f9e0d809d04f160645..8318717a3eb53f47bbf3222093224618281c083e was (Author: zentol): master: 40f0540a3fa7892e18d897eb35cbdbd091d4fe19..42b00efdc7c78aa3f01a6e0f6f3cf0fdfd093582 > Add NoticeFileChecker tests > --- > > Key: FLINK-29510 > URL: https://issues.apache.org/jira/browse/FLINK-29510 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > The NoticeFileChecker is too important to not be covered by tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29508) Some NOTICE files are not checked for correctness
[ https://issues.apache.org/jira/browse/FLINK-29508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29508. Resolution: Fixed master: 01d7b37f06b3d6da178c214335d9801db76f6383 1.16: d269e22b30c0758e905ebc3262c4463949df54e6 > Some NOTICE files are not checked for correctness > - > > Key: FLINK-29508 > URL: https://issues.apache.org/jira/browse/FLINK-29508 > Project: Flink > Issue Type: Technical Debt > Components: Build System >Affects Versions: 1.16.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > We have 3 modules that are not being deployed (and thus auto-excluded since > FLINK-29301) which are still relevant for production though. > We should amend the checker to take into account whether the non-deployed > module is bundled by another deployed module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29589) Data Loss in Sink GlobalCommitter during Task Manager recovery
[ https://issues.apache.org/jira/browse/FLINK-29589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17616298#comment-17616298 ] Chesnay Schepler commented on FLINK-29589: -- Does this also happen in 1.15 / 1.16-SNAPSHOT and/or the v2 Sink API? > Data Loss in Sink GlobalCommitter during Task Manager recovery > -- > > Key: FLINK-29589 > URL: https://issues.apache.org/jira/browse/FLINK-29589 > Project: Flink > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Krzysztof Chmielewski >Priority: Blocker > > Flink's Sink architecture with global committer seems to be vulnerable for > data loss during Task Manager recovery. The entire checkpoint can be lost by > _GlobalCommitter_ resulting with data loss. > Issue was observed in Delta Sink connector on a real 1.14.x cluster and was > replicated using Flink's 1.14.6 Test Utils classes. > Scenario: > # Streaming source emitting constant number of events per checkpoint (20 > events per commit for 5 commits in total, that gives 100 records). > # Sink with parallelism > 1 with committer and _GlobalCommitter_ elements. > # _Commiters_ processed committables for *checkpointId 2*. > # _GlobalCommitter_ throws exception (desired exception) during > *checkpointId 2* (third commit) while processing data from *checkpoint 1* (it > is expected to global committer architecture lag one commit behind in > reference to rest of the pipeline). > # Task Manager recovery, source resumes sending data. > # Streaming source ends. > # We are missing 20 records (one checkpoint). > What is happening is that during recovery, committers are performing "retry" > on committables for *checkpointId 2*, however those committables, reprocessed > from "retry" task are not emit downstream to the global committer. > The issue can be reproduced using Junit Test build with Flink's TestSink. > The test was [implemented > here|https://github.com/kristoffSC/flink/blob/Flink_1.14_DataLoss_SinkGlobalCommitter/flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/SinkITCase.java#:~:text=testGlobalCommitterMissingRecordsDuringRecovery] > and it is based on other tests from `SinkITCase.java` class. > The test reproduces the issue in more than 90% of runs. > I believe that problem is somewhere around > *SinkOperator::notifyCheckpointComplete* method. In there we see that Retry > async task is scheduled however its result is never emitted downstream like > it is done for regular flow one line above. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29589) Data Loss in Sink GlobalCommitter during Task Manager recovery
[ https://issues.apache.org/jira/browse/FLINK-29589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29589: - Affects Version/s: (was: 1.14.2) (was: 1.14.3) (was: 1.14.4) (was: 1.14.5) (was: 1.14.6) > Data Loss in Sink GlobalCommitter during Task Manager recovery > -- > > Key: FLINK-29589 > URL: https://issues.apache.org/jira/browse/FLINK-29589 > Project: Flink > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Krzysztof Chmielewski >Priority: Blocker > > Flink's Sink architecture with global committer seems to be vulnerable for > data loss during Task Manager recovery. The entire checkpoint can be lost by > _GlobalCommitter_ resulting with data loss. > Issue was observed in Delta Sink connector on a real 1.14.x cluster and was > replicated using Flink's 1.14.6 Test Utils classes. > Scenario: > # Streaming source emitting constant number of events per checkpoint (20 > events per commit for 5 commits in total, that gives 100 records). > # Sink with parallelism > 1 with committer and _GlobalCommitter_ elements. > # _Commiters_ processed committables for *checkpointId 2*. > # _GlobalCommitter_ throws exception (desired exception) during > *checkpointId 2* (third commit) while processing data from *checkpoint 1* (it > is expected to global committer architecture lag one commit behind in > reference to rest of the pipeline). > # Task Manager recovery, source resumes sending data. > # Streaming source ends. > # We are missing 20 records (one checkpoint). > What is happening is that during recovery, committers are performing "retry" > on committables for *checkpointId 2*, however those committables, reprocessed > from "retry" task are not emit downstream to the global committer. > The issue can be reproduced using Junit Test build with Flink's TestSink. > The test was [implemented > here|https://github.com/kristoffSC/flink/blob/Flink_1.14_DataLoss_SinkGlobalCommitter/flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/SinkITCase.java#:~:text=testGlobalCommitterMissingRecordsDuringRecovery] > and it is based on other tests from `SinkITCase.java` class. > The test reproduces the issue in more than 90% of runs. > I believe that problem is somewhere around > *SinkOperator::notifyCheckpointComplete* method. In there we see that Retry > async task is scheduled however its result is never emitted downstream like > it is done for regular flow one line above. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29576) Serialized OperatorCoordinators are collected in a not thread-safe ArrayList even though serialization was parallelized in FLINK-26675
[
https://issues.apache.org/jira/browse/FLINK-29576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29576:
-
Fix Version/s: 1.16.0
> Serialized OperatorCoordinators are collected in a not thread-safe ArrayList
> even though serialization was parallelized in FLINK-26675
> --
>
> Key: FLINK-29576
> URL: https://issues.apache.org/jira/browse/FLINK-29576
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
> Attachments:
> SourceNAryInputChainingITCase.testDirectSourcesOnlyExecution.log
>
>
> There's a [build
> failure|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41843=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=8523]
> being caused by
> {{SourceNAryInputChainingITCase.testDirectSourcesOnlyExecution}} on
> {{master}}:
> {code}
> Oct 11 01:45:36 [ERROR] Errors:
> Oct 11 01:45:36 [ERROR]
> SourceNAryInputChainingITCase.testDirectSourcesOnlyExecution:89 » Runtime
> Fail...
> Oct 11 01:45:36 [INFO]
> Oct 11 01:45:36 [ERROR] Tests run: 1931, Failures: 0, Errors: 1, Skipped: 4
> {code}
> The actual cause might be a missing {{OperatorCoordinatorHolder}} in
> {{DefaultOperatorCoordinatorHandler}} (see attached Maven logs that were
> extracted from the linked build):
> {code}
> 01:44:28,248 [flink-akka.actor.default-dispatcher-5] WARN
> org.apache.flink.runtime.taskmanager.Task[] -
> MultipleInputOperator [Source: source-1, Source: source-2, Source: source-3]
> (1/4)#0
> (9babb402557eb959216c28116aabddbe_1dd2eb40b0971d6d849b9e4a69494c88_0_0)
> switched from RUNNING to FAILED with failure cause:
> org.apache.flink.util.FlinkException: No coordinator registered for operator
> bc764cd8ddf7a0cff126f51c16239658
> at
> org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.deliverOperatorEventToCoordinator(DefaultOperatorCoordinatorHandler.java:117)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.deliverOperatorEventToCoordinator(SchedulerBase.java:1031)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.sendOperatorEventToCoordinator(JobMaster.java:588)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:309)
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:307)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:222)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at akka.actor.Actor.aroundReceive(Actor.scala:537)
> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
>
[jira] [Closed] (FLINK-29569) Replace usages of deprecated expand shortcode
[
https://issues.apache.org/jira/browse/FLINK-29569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-29569.
Resolution: Fixed
master: 4934bd69052f2a69e8021d337373f4480c802359
> Replace usages of deprecated expand shortcode
> -
>
> Key: FLINK-29569
> URL: https://issues.apache.org/jira/browse/FLINK-29569
> Project: Flink
> Issue Type: Technical Debt
> Components: Documentation
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> The expand shortcode is deprecated; use {{}} instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29563) SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case the number of records counter is wrong
[ https://issues.apache.org/jira/browse/FLINK-29563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615858#comment-17615858 ] Chesnay Schepler edited comment on FLINK-29563 at 10/11/22 2:22 PM: ??As a more general comment, maybe we should introduce junit tags to specify tests targeted to unbounded sources and dynamically ignore them for bounded sources. Chesnay Schepler WDYT ? Should I create a new ticket for that??? ehhh I'm not a fan of that. So far we only used categories/tags for exceptional circumstances, like known bugs or incompatibilities (e.g., java 11). To me it should be obvious whether a test is run or not. Why is it relevant whether the source is bounded or not for this test? was (Author: zentol): ??As a more general comment, maybe we should introduce junit tags to specify tests targeted to unbounded sources and dynamically ignore them for bounded sources. Chesnay Schepler WDYT ? Should I create a new ticket for that??? ehhh I'm not a fan of that. So far we only used categories/tags for exceptional circumstances, like known bugs or incompatibilities (e.g., java 11). Why is it relevant whether the source is bounded or not for this test? > SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case > the number of records counter is wrong > > > Key: FLINK-29563 > URL: https://issues.apache.org/jira/browse/FLINK-29563 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > The call to _CommonTestUtils#waitUntilCondition_ (1) makes the test wait for > the condition _Precision.equals(allRecordSize, sumNumRecordsIn)_. In case the > reported number of records is incorrect, the waiting loop never ends. > [1] > https://github.com/apache/flink/blob/a6092b1176d15a7af32a7eb19f59cdfeab172034/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/testsuites/SourceTestSuiteBase.java#L451 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29563) SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case the number of records counter is wrong
[ https://issues.apache.org/jira/browse/FLINK-29563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615858#comment-17615858 ] Chesnay Schepler edited comment on FLINK-29563 at 10/11/22 2:21 PM: ??As a more general comment, maybe we should introduce junit tags to specify tests targeted to unbounded sources and dynamically ignore them for bounded sources. Chesnay Schepler WDYT ? Should I create a new ticket for that??? ehhh I'm not a fan of that. So far we only used categories/tags for exceptional circumstances, like known bugs or incompatibilities (e.g., java 11). Why is it relevant whether the source is bounded or not for this test? was (Author: zentol): ??As a more general comment, maybe we should introduce junit tags to specify tests targeted to unbounded sources and dynamically ignore them for bounded sources. Chesnay Schepler WDYT ? Should I create a new ticket for that ??? ehhh I'm not a fan of that. So far we only used categories/tags for exceptional circumstances, like known bugs or incompatibilities (e.g., java 11). Why is it relevant whether the source is bounded or not for this test? > SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case > the number of records counter is wrong > > > Key: FLINK-29563 > URL: https://issues.apache.org/jira/browse/FLINK-29563 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > The call to _CommonTestUtils#waitUntilCondition_ (1) makes the test wait for > the condition _Precision.equals(allRecordSize, sumNumRecordsIn)_. In case the > reported number of records is incorrect, the waiting loop never ends. > [1] > https://github.com/apache/flink/blob/a6092b1176d15a7af32a7eb19f59cdfeab172034/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/testsuites/SourceTestSuiteBase.java#L451 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29563) SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case the number of records counter is wrong
[ https://issues.apache.org/jira/browse/FLINK-29563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615858#comment-17615858 ] Chesnay Schepler commented on FLINK-29563: -- > As a more general comment, maybe we should introduce junit tags to specify > tests targeted to unbounded sources and dynamically ignore them for bounded > sources. Chesnay Schepler WDYT ? Should I create a new ticket for that ? ehhh I'm not a fan of that. So far we only used categories/tags for exceptional circumstances, like known bugs or incompatibilities (e.g., java 11). Why is it relevant whether the source is bounded or not for this test? > SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case > the number of records counter is wrong > > > Key: FLINK-29563 > URL: https://issues.apache.org/jira/browse/FLINK-29563 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > The call to _CommonTestUtils#waitUntilCondition_ (1) makes the test wait for > the condition _Precision.equals(allRecordSize, sumNumRecordsIn)_. In case the > reported number of records is incorrect, the waiting loop never ends. > [1] > https://github.com/apache/flink/blob/a6092b1176d15a7af32a7eb19f59cdfeab172034/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/testsuites/SourceTestSuiteBase.java#L451 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29563) SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case the number of records counter is wrong
[ https://issues.apache.org/jira/browse/FLINK-29563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615858#comment-17615858 ] Chesnay Schepler edited comment on FLINK-29563 at 10/11/22 2:21 PM: ??As a more general comment, maybe we should introduce junit tags to specify tests targeted to unbounded sources and dynamically ignore them for bounded sources. Chesnay Schepler WDYT ? Should I create a new ticket for that ??? ehhh I'm not a fan of that. So far we only used categories/tags for exceptional circumstances, like known bugs or incompatibilities (e.g., java 11). Why is it relevant whether the source is bounded or not for this test? was (Author: zentol): > As a more general comment, maybe we should introduce junit tags to specify > tests targeted to unbounded sources and dynamically ignore them for bounded > sources. Chesnay Schepler WDYT ? Should I create a new ticket for that ? ehhh I'm not a fan of that. So far we only used categories/tags for exceptional circumstances, like known bugs or incompatibilities (e.g., java 11). Why is it relevant whether the source is bounded or not for this test? > SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case > the number of records counter is wrong > > > Key: FLINK-29563 > URL: https://issues.apache.org/jira/browse/FLINK-29563 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > The call to _CommonTestUtils#waitUntilCondition_ (1) makes the test wait for > the condition _Precision.equals(allRecordSize, sumNumRecordsIn)_. In case the > reported number of records is incorrect, the waiting loop never ends. > [1] > https://github.com/apache/flink/blob/a6092b1176d15a7af32a7eb19f59cdfeab172034/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/testsuites/SourceTestSuiteBase.java#L451 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29584) Upgrade java 11 version on the microbenchmark worker
[ https://issues.apache.org/jira/browse/FLINK-29584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615745#comment-17615745 ] Chesnay Schepler commented on FLINK-29584: -- Do you think this is also relevant for production? > Upgrade java 11 version on the microbenchmark worker > > > Key: FLINK-29584 > URL: https://issues.apache.org/jira/browse/FLINK-29584 > Project: Flink > Issue Type: Improvement > Components: Benchmarks >Affects Versions: 1.17.0 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski >Priority: Major > Fix For: 1.17.0 > > > It looks like the older JDK 11 builds have problems with JIT in the > microbenchmarks, as for example visible > [here|http://codespeed.dak8s.net:8000/timeline/?ben=globalWindow=2]. > Locally I was able to reproduce this problem and the issue goes away after > upgrading to a newer JDK 11 build. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29408) HiveCatalogITCase failed with NPE
[
https://issues.apache.org/jira/browse/FLINK-29408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-29408.
Fix Version/s: 1.17.0
1.16.1
Resolution: Fixed
master: 23e027230838ac5728b2974f02235f512c97010c
1.16: 4403ea5811d9d1edc5896818081005c4d6237efa
> HiveCatalogITCase failed with NPE
> -
>
> Key: FLINK-29408
> URL: https://issues.apache.org/jira/browse/FLINK-29408
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Huang Xingbo
>Assignee: luoyuxia
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0, 1.16.1
>
>
> {code:java}
> 2022-09-25T03:41:07.4212129Z Sep 25 03:41:07 [ERROR]
> org.apache.flink.table.catalog.hive.HiveCatalogUdfITCase.testFlinkUdf Time
> elapsed: 0.098 s <<< ERROR!
> 2022-09-25T03:41:07.4212662Z Sep 25 03:41:07 java.lang.NullPointerException
> 2022-09-25T03:41:07.4213189Z Sep 25 03:41:07 at
> org.apache.flink.table.catalog.hive.HiveCatalogUdfITCase.testFlinkUdf(HiveCatalogUdfITCase.java:109)
> 2022-09-25T03:41:07.4213753Z Sep 25 03:41:07 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-09-25T03:41:07.4224643Z Sep 25 03:41:07 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-09-25T03:41:07.4225311Z Sep 25 03:41:07 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-09-25T03:41:07.4225879Z Sep 25 03:41:07 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-09-25T03:41:07.4226405Z Sep 25 03:41:07 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-09-25T03:41:07.4227201Z Sep 25 03:41:07 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-09-25T03:41:07.4227807Z Sep 25 03:41:07 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-09-25T03:41:07.4228394Z Sep 25 03:41:07 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-09-25T03:41:07.4228966Z Sep 25 03:41:07 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-09-25T03:41:07.4229514Z Sep 25 03:41:07 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-09-25T03:41:07.4230066Z Sep 25 03:41:07 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-09-25T03:41:07.4230587Z Sep 25 03:41:07 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-09-25T03:41:07.4231258Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-09-25T03:41:07.4231823Z Sep 25 03:41:07 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-09-25T03:41:07.4232384Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-09-25T03:41:07.4232930Z Sep 25 03:41:07 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-09-25T03:41:07.4233511Z Sep 25 03:41:07 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-09-25T03:41:07.4234039Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-09-25T03:41:07.4234546Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-09-25T03:41:07.4235057Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-09-25T03:41:07.4235573Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-09-25T03:41:07.4236087Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-09-25T03:41:07.4236635Z Sep 25 03:41:07 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-09-25T03:41:07.4237314Z Sep 25 03:41:07 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-09-25T03:41:07.4238211Z Sep 25 03:41:07 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-09-25T03:41:07.4238775Z Sep 25 03:41:07 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-09-25T03:41:07.4239277Z Sep 25 03:41:07 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2022-09-25T03:41:07.4239769Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-09-25T03:41:07.4240265Z Sep 25 03:41:07 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2022-09-25T03:41:07.4240731Z Sep 25 03:41:07 at
>
[jira] [Created] (FLINK-29582) SavepointWriter should be usable without any transformation
Chesnay Schepler created FLINK-29582: Summary: SavepointWriter should be usable without any transformation Key: FLINK-29582 URL: https://issues.apache.org/jira/browse/FLINK-29582 Project: Flink Issue Type: Improvement Components: API / State Processor Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0 The SavepointWriter of the state processor API currently enforces at least one transformation to be defined be the user. This is an irritating limitation; this means you can't use the API to delete a state or use the new uid remapping function from FLINK-29457 without specifying some dummy transformation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29457) Add a uid(hash) remapping function
[ https://issues.apache.org/jira/browse/FLINK-29457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29457. Resolution: Fixed master: 8ccca78ca0db23f7965fe77ea9b57d8391b6f583 > Add a uid(hash) remapping function > -- > > Key: FLINK-29457 > URL: https://issues.apache.org/jira/browse/FLINK-29457 > Project: Flink > Issue Type: Sub-task > Components: API / State Processor >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > Expose functionality for modifying the uid[hash] of a state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29453) Add uidHash support to State Processor API
[ https://issues.apache.org/jira/browse/FLINK-29453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29453. Resolution: Fixed > Add uidHash support to State Processor API > --- > > Key: FLINK-29453 > URL: https://issues.apache.org/jira/browse/FLINK-29453 > Project: Flink > Issue Type: New Feature > Components: API / State Processor >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0 > > > The state process API is currently limited to working with uids. > We should change this since this is a good application for the API. > The API should be extended to support uidHashes wherever a uid is support, > and we should add a method to map uid[hashes] to a different uid[hash]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29568) Remove unnecessary whitespace in request/response blocks
[ https://issues.apache.org/jira/browse/FLINK-29568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29568. Resolution: Fixed master: 6934032acd09cfaae9945ca9c36ea1077c9e1eeb > Remove unnecessary whitespace in request/response blocks > > > Key: FLINK-29568 > URL: https://issues.apache.org/jira/browse/FLINK-29568 > Project: Flink > Issue Type: Technical Debt > Components: Documentation, Runtime / REST >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29503) Add backpressureLevel field without hyphens
[
https://issues.apache.org/jira/browse/FLINK-29503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-29503.
Fix Version/s: 1.17.0
1.16.1
(was: 1.16.0)
Resolution: Fixed
master: a743812edd7e19cf5b43d9efd1222471def90d18
1.16: fc708c2fe212826a25fa3151b5b8571e3993f15b
1.15: 5ad5618821f4b004f5de95004280981f1de6d17a
> Add backpressureLevel field without hyphens
> ---
>
> Key: FLINK-29503
> URL: https://issues.apache.org/jira/browse/FLINK-29503
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.15.2
>Reporter: Tiger (Apache) Wang
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: openapi, pull-request-available
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> Install nodejs and run
> {{$ npx --yes --package openapi-typescript-codegen openapi --input
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/generated/rest_v1_dispatcher.yml]
> --output .}}
> {{$ npx --package typescript tsc }}
> The only thing it complains about is:
> {{{}src/models/JobVertexBackPressureInfo.ts:21:17 - error TS1003: Identifier
> expected.{}}}{{{}21 export enum 'backpressure-level' {{}}}
> This is because for TypeScript, enum name should not have a hyphen in it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29504) Jar upload spec should define a schema
[
https://issues.apache.org/jira/browse/FLINK-29504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-29504.
Resolution: Fixed
master: e188cb2d66cc68e5eb822c5cca4fe0ee85173157
1.16: 141eeffb6b2c05d7c139c94273c2e5c468b44ec9
1.15: 383d451595f364196010351205f2ee8fd7f073c7
> Jar upload spec should define a schema
> --
>
> Key: FLINK-29504
> URL: https://issues.apache.org/jira/browse/FLINK-29504
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.15.2
>Reporter: Tiger (Apache) Wang
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: openapi, pull-request-available
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> Install nodejs and run
> {{$ npx --yes @openapitools/openapi-generator-cli generate -i
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/generated/rest_v1_dispatcher.yml]
> -g typescript-axios -o .}}
>
> Then it outputs error
>
> {{Caused by: java.lang.RuntimeException: Request body cannot be null.
> Possible cause: missing schema in body parameter (OAS v2): class RequestBody
> {}}
> {{ description: null}}
> {{ content: class Content {}}
> {{ {application/x-java-archive=class MediaType {}}
> {{ schema: null}}
> {{ examples: null}}
> {{ example: null}}
> {{ encoding: null}}
> {\{
> {\{ }}}
> {{ required: true}}
> {{}}}
>
> This is because in the YAML:
> {{}}{{ requestBody:}}
> {{ content:}}
> {{{} application/x-java-archive: {{
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29504) Jar upload spec should define a schema
[
https://issues.apache.org/jira/browse/FLINK-29504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29504:
-
Fix Version/s: 1.17.0
> Jar upload spec should define a schema
> --
>
> Key: FLINK-29504
> URL: https://issues.apache.org/jira/browse/FLINK-29504
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.15.2
>Reporter: Tiger (Apache) Wang
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: openapi, pull-request-available
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> Install nodejs and run
> {{$ npx --yes @openapitools/openapi-generator-cli generate -i
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/generated/rest_v1_dispatcher.yml]
> -g typescript-axios -o .}}
>
> Then it outputs error
>
> {{Caused by: java.lang.RuntimeException: Request body cannot be null.
> Possible cause: missing schema in body parameter (OAS v2): class RequestBody
> {}}
> {{ description: null}}
> {{ content: class Content {}}
> {{ {application/x-java-archive=class MediaType {}}
> {{ schema: null}}
> {{ examples: null}}
> {{ example: null}}
> {{ encoding: null}}
> {\{
> {\{ }}}
> {{ required: true}}
> {{}}}
>
> This is because in the YAML:
> {{}}{{ requestBody:}}
> {{ content:}}
> {{{} application/x-java-archive: {{
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29504) Jar upload spec should define a schema
[
https://issues.apache.org/jira/browse/FLINK-29504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29504:
-
Fix Version/s: 1.16.1
(was: 1.16.0)
> Jar upload spec should define a schema
> --
>
> Key: FLINK-29504
> URL: https://issues.apache.org/jira/browse/FLINK-29504
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.15.2
>Reporter: Tiger (Apache) Wang
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: openapi, pull-request-available
> Fix For: 1.15.3, 1.16.1
>
>
> Install nodejs and run
> {{$ npx --yes @openapitools/openapi-generator-cli generate -i
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/generated/rest_v1_dispatcher.yml]
> -g typescript-axios -o .}}
>
> Then it outputs error
>
> {{Caused by: java.lang.RuntimeException: Request body cannot be null.
> Possible cause: missing schema in body parameter (OAS v2): class RequestBody
> {}}
> {{ description: null}}
> {{ content: class Content {}}
> {{ {application/x-java-archive=class MediaType {}}
> {{ schema: null}}
> {{ examples: null}}
> {{ example: null}}
> {{ encoding: null}}
> {\{
> {\{ }}}
> {{ required: true}}
> {{}}}
>
> This is because in the YAML:
> {{}}{{ requestBody:}}
> {{ content:}}
> {{{} application/x-java-archive: {{
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29571) Flink configuration only supports Map
[
https://issues.apache.org/jira/browse/FLINK-29571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615147#comment-17615147
]
Chesnay Schepler commented on FLINK-29571:
--
That'd also imply that T can also be a Map/List. I'm not sure if the parser can
handle that.
> Flink configuration only supports Map
> -
>
> Key: FLINK-29571
> URL: https://issues.apache.org/jira/browse/FLINK-29571
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> The Flink configuration parsing logic only supports maps with string key and
> value types. It should support {{Map}}. T should allow the same
> value types that are already supported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27721) Slack: set up archive
[ https://issues.apache.org/jira/browse/FLINK-27721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615134#comment-17615134 ] Chesnay Schepler commented on FLINK-27721: -- I'd still like a clarification about whether it stores all data and whether all data can be exported on demand. I don't want to end up in a situation where everything is lost because they go down / mess up. In particular since this services appears to be relatively new, and is (apparently) only backed by few people. > Slack: set up archive > - > > Key: FLINK-27721 > URL: https://issues.apache.org/jira/browse/FLINK-27721 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29562) JM/SQl gateway OpenAPI specs should have different titles
[ https://issues.apache.org/jira/browse/FLINK-29562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29562. Resolution: Fixed master: 1c2fd3584b899412f6a94303ecdd29513664e9dc 1.16: ae5c7046f11504d5399f11e5be1762cf378e3c30 > JM/SQl gateway OpenAPI specs should have different titles > - > > Key: FLINK-29562 > URL: https://issues.apache.org/jira/browse/FLINK-29562 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / REST >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0, 1.16.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29569) Replace usages of deprecated expand shortcode
Chesnay Schepler created FLINK-29569:
Summary: Replace usages of deprecated expand shortcode
Key: FLINK-29569
URL: https://issues.apache.org/jira/browse/FLINK-29569
Project: Flink
Issue Type: Technical Debt
Components: Documentation
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.17.0
The expand shortcode is deprecated; use {{}} instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29568) Remove unnecessary whitespace in request/response blocks
Chesnay Schepler created FLINK-29568: Summary: Remove unnecessary whitespace in request/response blocks Key: FLINK-29568 URL: https://issues.apache.org/jira/browse/FLINK-29568 Project: Flink Issue Type: Technical Debt Components: Documentation, Runtime / REST Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29563) SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case the number of records counter is wrong
[ https://issues.apache.org/jira/browse/FLINK-29563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-29563: Assignee: Etienne Chauchot > SourceTestSuiteBase#testSourceMetrics enters an infinite waiting loop in case > the number of records counter is wrong > > > Key: FLINK-29563 > URL: https://issues.apache.org/jira/browse/FLINK-29563 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > The call to _CommonTestUtils#waitUntilCondition_ (1) makes the test wait for > the condition _Precision.equals(allRecordSize, sumNumRecordsIn)_. In case the > reported number of records is incorrect, the waiting loop never ends. > [1] > https://github.com/apache/flink/blob/a6092b1176d15a7af32a7eb19f59cdfeab172034/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/testsuites/SourceTestSuiteBase.java#L451 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29562) JM/SQl gatewa OpenAPI specs should have different titles
Chesnay Schepler created FLINK-29562: Summary: JM/SQl gatewa OpenAPI specs should have different titles Key: FLINK-29562 URL: https://issues.apache.org/jira/browse/FLINK-29562 Project: Flink Issue Type: Bug Components: Documentation, Runtime / REST Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0, 1.16.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29562) JM/SQl gateway OpenAPI specs should have different titles
[ https://issues.apache.org/jira/browse/FLINK-29562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29562: - Summary: JM/SQl gateway OpenAPI specs should have different titles (was: JM/SQl gatewa OpenAPI specs should have different titles) > JM/SQl gateway OpenAPI specs should have different titles > - > > Key: FLINK-29562 > URL: https://issues.apache.org/jira/browse/FLINK-29562 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / REST >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0, 1.16.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-27721) Slack: set up archive
[ https://issues.apache.org/jira/browse/FLINK-27721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614998#comment-17614998 ] Chesnay Schepler edited comment on FLINK-27721 at 10/10/22 9:28 AM: ??What else do you expect from an archive??? >From the description on the page I'd conclude they just display what is >_currently_ stored in the Slack instance in a searchable manner. I'd expect an archive to store _all_ data, even if it's no longer available in slack or the slack instance was removed altogether. was (Author: zentol): > What else do you expect from an archive? >From the description on the page I'd conclude they just display what is >_currently_ stored in the Slack instance in a searchable manner. I'd expect an archive to store _all_ data, even if it's no longer available in slack or the slack instance was removed altogether. > Slack: set up archive > - > > Key: FLINK-27721 > URL: https://issues.apache.org/jira/browse/FLINK-27721 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27721) Slack: set up archive
[ https://issues.apache.org/jira/browse/FLINK-27721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614998#comment-17614998 ] Chesnay Schepler commented on FLINK-27721: -- > What else do you expect from an archive? >From the description on the page I'd conclude they just display what is >_currently_ stored in the Slack instance in a searchable manner. I'd expect an archive to store _all_ data, even if it's no longer available in slack or the slack instance was removed altogether. > Slack: set up archive > - > > Key: FLINK-27721 > URL: https://issues.apache.org/jira/browse/FLINK-27721 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29503) Add backpressureLevel field without hyphens
[
https://issues.apache.org/jira/browse/FLINK-29503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29503:
-
Summary: Add backpressureLevel field without hyphens (was: Add
backpressureLevel field wiithout hyphens)
> Add backpressureLevel field without hyphens
> ---
>
> Key: FLINK-29503
> URL: https://issues.apache.org/jira/browse/FLINK-29503
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.15.2
>Reporter: Tiger (Apache) Wang
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: openapi
> Fix For: 1.16.0, 1.15.3
>
>
> Install nodejs and run
> {{$ npx --yes --package openapi-typescript-codegen openapi --input
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/generated/rest_v1_dispatcher.yml]
> --output .}}
> {{$ npx --package typescript tsc }}
> The only thing it complains about is:
> {{{}src/models/JobVertexBackPressureInfo.ts:21:17 - error TS1003: Identifier
> expected.{}}}{{{}21 export enum 'backpressure-level' {{}}}
> This is because for TypeScript, enum name should not have a hyphen in it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27721) Slack: set up archive
[ https://issues.apache.org/jira/browse/FLINK-27721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614968#comment-17614968 ] Chesnay Schepler commented on FLINK-27721: -- Is that actually an _archive_ though? It _sounds_ more like a front to support searches via google. > Slack: set up archive > - > > Key: FLINK-27721 > URL: https://issues.apache.org/jira/browse/FLINK-27721 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29501) Allow overriding JobVertex parallelisms during job submission
[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614120#comment-17614120
]
Chesnay Schepler commented on FLINK-29501:
--
??the Recale API is both disabled and broken at the moment??
In practice the entire mechanism doesn't exist. I'd ignore that there are still
some legacy API fragments around that we haven't removed IIRC exclusively so
that users don't hit a 404 when attempting to do a rescale.
??If you think it is already robust enough to support rescaling requests, we
can re-enable the rescale rest API and also add job vertex overrides to it??
It's definitely robust enough to be used in production I think.
IIRC Till had a prototype for adding a REST endpoint that adjusts the target
parallelism somewhere.
I don't think it worked on a per-vertex basis though; just a global parallelism
increase for all vertices (since in reactive mode everything scales uniformly
anyway). But that shouldn't be difficult to change.
> Allow overriding JobVertex parallelisms during job submission
> -
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29400) Default Value of env.log.max in documentation is incorrect
[ https://issues.apache.org/jira/browse/FLINK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29400. Fix Version/s: 1.17.0 Resolution: Fixed master: 912d00a8816efafb43b5fd4f3e7545df6133ce90 > Default Value of env.log.max in documentation is incorrect > -- > > Key: FLINK-29400 > URL: https://issues.apache.org/jira/browse/FLINK-29400 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Dhruv Patel >Assignee: Chesnay Schepler >Priority: Minor > Fix For: 1.17.0 > > > The default value of env.log.max is 10 as per the code in master > ([https://github.com/apache/flink/blob/master/flink-dist/src/main/flink-bin/bin/config.sh#L137).] > However the Flink Documentation says the default value is 5 > (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#env-log-max) > which is incorrect -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29400) Default Value of env.log.max in documentation is incorrect
[ https://issues.apache.org/jira/browse/FLINK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-29400: Assignee: Chesnay Schepler > Default Value of env.log.max in documentation is incorrect > -- > > Key: FLINK-29400 > URL: https://issues.apache.org/jira/browse/FLINK-29400 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Dhruv Patel >Assignee: Chesnay Schepler >Priority: Minor > > The default value of env.log.max is 10 as per the code in master > ([https://github.com/apache/flink/blob/master/flink-dist/src/main/flink-bin/bin/config.sh#L137).] > However the Flink Documentation says the default value is 5 > (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#env-log-max) > which is incorrect -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29497) Provide an option to publish the flink-dist jar file artifact
[ https://issues.apache.org/jira/browse/FLINK-29497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614031#comment-17614031 ] Chesnay Schepler commented on FLINK-29497: -- Can you explain why you'd like this to be configurable, and what parts of flink-dist you're actually interested in? I'm asking since one my long-time wishes was to split flink-dist into the 2 modules; one module that creates the Flink fat jar (==flink-dist.jar) that would be published (to stream-line the dependency setup of users) and one module that assembles the binary distribution (with all of it's optional dependencies etc) that would still not be published. > Provide an option to publish the flink-dist jar file artifact > - > > Key: FLINK-29497 > URL: https://issues.apache.org/jira/browse/FLINK-29497 > Project: Flink > Issue Type: Improvement > Components: Build System >Affects Versions: 1.16.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > > Currently deployment is skipped for the flink-dist jar file. Instead of > hardcoding that in pom.xml, use a property that can control this behavior > from the maven command line. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29534) @TypeInfo on field requires field type to be valid Pojo
[
https://issues.apache.org/jira/browse/FLINK-29534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614015#comment-17614015
]
Chesnay Schepler edited comment on FLINK-29534 at 10/7/22 10:46 AM:
It's not the field most be a POJO; the annotated class must actually go through
Flinks serialization stack.
If the ObjectNode is within a List then the whole thing goes through Kryo; the
List is serialized with Kryo and Flink neither does nor can further analyze the
contents.
If you'd annotate the list with {{@TypeInfo}} and set it up to go through
Flinks ListSerializer than TypeInfo will also work on the contained ObjectNodes.
was (Author: zentol):
It's not the field most be a POJO; the annotated class must actually go through
Flinks serialization stack.
If the ObjectNode is within a List then it the whole thing goes through Kryo.
If you'd annotate the list with TypeInfo and set it up to go through Flinks
ListSerializer than TypeInfo will also work on the contained ObjectNodes.
> @TypeInfo on field requires field type to be valid Pojo
>
>
> Key: FLINK-29534
> URL: https://issues.apache.org/jira/browse/FLINK-29534
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Exidex
>Priority: Major
>
> The ability to place @TypeInfo on field was added in
> [https://github.com/apache/flink/pull/8344] . But it seams like the fact that
> it requires field to be a valid POJO was overlooked. In my case I was trying
> to add custom serializer for Jackson's ObjectNode (wrapped in List but not
> sure if this is relevant https://issues.apache.org/jira/browse/FLINK-26470)
> which is not a valid POJO, and this requirement seams to defeat the whole
> purpose of such feature. It also doesn't look like like there's a way to
> register {{org.apache.flink.api.common.typeutils.TypeSerializer}} globally on
> 3rd-party types
> code snippet from TypeExtractor:
> {code:java}
> Type fieldType = field.getGenericType();
> if (!isValidPojoField(field, clazz, typeHierarchy) && clazz != Row.class) {
> LOG.info(
> "Class "
> + clazz
> + " cannot be used as a POJO type because not all fields
> are valid POJO fields, "
> + "and must be processed as GenericType. {}",
> GENERIC_TYPE_DOC_HINT);
> return null;
> }
> try {
> final TypeInformation typeInfo;
> List fieldTypeHierarchy = new ArrayList<>(typeHierarchy);
> TypeInfoFactory factory = getTypeInfoFactory(field);
> if (factory != null) {{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29534) @TypeInfo on field requires field type to be valid Pojo
[
https://issues.apache.org/jira/browse/FLINK-29534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614016#comment-17614016
]
Chesnay Schepler edited comment on FLINK-29534 at 10/7/22 10:45 AM:
??It also doesn't look like like there's a way to register
org.apache.flink.api.common.typeutils.TypeSerializer globally on 3rd-party
types??
Yes, this was removed at some point.
was (Author: zentol):
> It also doesn't look like like there's a way to register
> org.apache.flink.api.common.typeutils.TypeSerializer globally on 3rd-party
> types
Yes, this was removed at some point.
> @TypeInfo on field requires field type to be valid Pojo
>
>
> Key: FLINK-29534
> URL: https://issues.apache.org/jira/browse/FLINK-29534
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Exidex
>Priority: Major
>
> The ability to place @TypeInfo on field was added in
> [https://github.com/apache/flink/pull/8344] . But it seams like the fact that
> it requires field to be a valid POJO was overlooked. In my case I was trying
> to add custom serializer for Jackson's ObjectNode (wrapped in List but not
> sure if this is relevant https://issues.apache.org/jira/browse/FLINK-26470)
> which is not a valid POJO, and this requirement seams to defeat the whole
> purpose of such feature. It also doesn't look like like there's a way to
> register {{org.apache.flink.api.common.typeutils.TypeSerializer}} globally on
> 3rd-party types
> code snippet from TypeExtractor:
> {code:java}
> Type fieldType = field.getGenericType();
> if (!isValidPojoField(field, clazz, typeHierarchy) && clazz != Row.class) {
> LOG.info(
> "Class "
> + clazz
> + " cannot be used as a POJO type because not all fields
> are valid POJO fields, "
> + "and must be processed as GenericType. {}",
> GENERIC_TYPE_DOC_HINT);
> return null;
> }
> try {
> final TypeInformation typeInfo;
> List fieldTypeHierarchy = new ArrayList<>(typeHierarchy);
> TypeInfoFactory factory = getTypeInfoFactory(field);
> if (factory != null) {{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29534) @TypeInfo on field requires field type to be valid Pojo
[
https://issues.apache.org/jira/browse/FLINK-29534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614016#comment-17614016
]
Chesnay Schepler commented on FLINK-29534:
--
> It also doesn't look like like there's a way to register
> org.apache.flink.api.common.typeutils.TypeSerializer globally on 3rd-party
> types
Yes, this was removed at some point.
> @TypeInfo on field requires field type to be valid Pojo
>
>
> Key: FLINK-29534
> URL: https://issues.apache.org/jira/browse/FLINK-29534
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Exidex
>Priority: Major
>
> The ability to place @TypeInfo on field was added in
> [https://github.com/apache/flink/pull/8344] . But it seams like the fact that
> it requires field to be a valid POJO was overlooked. In my case I was trying
> to add custom serializer for Jackson's ObjectNode (wrapped in List but not
> sure if this is relevant https://issues.apache.org/jira/browse/FLINK-26470)
> which is not a valid POJO, and this requirement seams to defeat the whole
> purpose of such feature. It also doesn't look like like there's a way to
> register {{org.apache.flink.api.common.typeutils.TypeSerializer}} globally on
> 3rd-party types
> code snippet from TypeExtractor:
> {code:java}
> Type fieldType = field.getGenericType();
> if (!isValidPojoField(field, clazz, typeHierarchy) && clazz != Row.class) {
> LOG.info(
> "Class "
> + clazz
> + " cannot be used as a POJO type because not all fields
> are valid POJO fields, "
> + "and must be processed as GenericType. {}",
> GENERIC_TYPE_DOC_HINT);
> return null;
> }
> try {
> final TypeInformation typeInfo;
> List fieldTypeHierarchy = new ArrayList<>(typeHierarchy);
> TypeInfoFactory factory = getTypeInfoFactory(field);
> if (factory != null) {{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29534) @TypeInfo on field requires field type to be valid Pojo
[
https://issues.apache.org/jira/browse/FLINK-29534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614015#comment-17614015
]
Chesnay Schepler commented on FLINK-29534:
--
It's not the field most be a POJO; the annotated class must actually go through
Flinks serialization stack.
If the ObjectNode is within a List then it the whole thing goes through Kryo.
If you'd annotate the list with TypeInfo and set it up to go through Flinks
ListSerializer than TypeInfo will also work on the contained ObjectNodes.
> @TypeInfo on field requires field type to be valid Pojo
>
>
> Key: FLINK-29534
> URL: https://issues.apache.org/jira/browse/FLINK-29534
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Exidex
>Priority: Major
>
> The ability to place @TypeInfo on field was added in
> [https://github.com/apache/flink/pull/8344] . But it seams like the fact that
> it requires field to be a valid POJO was overlooked. In my case I was trying
> to add custom serializer for Jackson's ObjectNode (wrapped in List but not
> sure if this is relevant https://issues.apache.org/jira/browse/FLINK-26470)
> which is not a valid POJO, and this requirement seams to defeat the whole
> purpose of such feature. It also doesn't look like like there's a way to
> register {{org.apache.flink.api.common.typeutils.TypeSerializer}} globally on
> 3rd-party types
> code snippet from TypeExtractor:
> {code:java}
> Type fieldType = field.getGenericType();
> if (!isValidPojoField(field, clazz, typeHierarchy) && clazz != Row.class) {
> LOG.info(
> "Class "
> + clazz
> + " cannot be used as a POJO type because not all fields
> are valid POJO fields, "
> + "and must be processed as GenericType. {}",
> GENERIC_TYPE_DOC_HINT);
> return null;
> }
> try {
> final TypeInformation typeInfo;
> List fieldTypeHierarchy = new ArrayList<>(typeHierarchy);
> TypeInfoFactory factory = getTypeInfoFactory(field);
> if (factory != null) {{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29510) Add NoticeFileChecker tests
[ https://issues.apache.org/jira/browse/FLINK-29510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29510. Resolution: Fixed master: 40f0540a3fa7892e18d897eb35cbdbd091d4fe19..42b00efdc7c78aa3f01a6e0f6f3cf0fdfd093582 > Add NoticeFileChecker tests > --- > > Key: FLINK-29510 > URL: https://issues.apache.org/jira/browse/FLINK-29510 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > The NoticeFileChecker is too important to not be covered by tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29501) Allow overriding JobVertex parallelisms during job submission
[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613764#comment-17613764
]
Chesnay Schepler edited comment on FLINK-29501 at 10/6/22 7:35 PM:
---
??The upside may be shorter downtime.??
The upside is significantly more potential for optimizations down-the-line,
like deferring a rescaling to a job failure (== low cost rescaling) or it being
properly integrated into batch jobs (==rescale on next stage, similar to the
AdaptiveBatchScheduler).
??rescaling with an api call is not a "persistent" operation. If a user later
wants to upgrade the job using a savepoint, they would have to then tweak the
parallelism of the entire pipeline from the main method or rescale again after
upgrade.??
If the parallelism is being tuned as part of an auto-scaler, wouldn't it make
sense to start from a blank slate if a job is fully resubmitted? I'd assume the
initial parallelism to be a reasonable base-line.
??Flink already provides a myriad of config options including default
parallelism, max parallelism and various execution settings, this is not
different from that in my view.??
This is quite different to me since it introduces the concept of
vertex-specific configurations.
It introduces _yet another_ way how the parallelism can be _configured_.
Next you're gonna propose to be able to set the max parallelism, uid, name etc
etc when submitting a job.
was (Author: zentol):
?? The upside may be shorter downtime.??
The upside is significantly more potential for optimizations down-the-line,
like deferring a rescaling to a job failure (== low cost rescaling) or it being
properly integrated into batch jobs (==rescale on next stage, similar to the
AdaptiveBatchScheduler).
??rescaling with an api call is not a "persistent" operation. If a user later
wants to upgrade the job using a savepoint, they would have to then tweak the
parallelism of the entire pipeline from the main method or rescale again after
upgrade.??
If the parallelism is being tuned as part of an auto-scaler, wouldn't it make
sense to start from a blank slate if a job is fully resubmitted? I'd assume the
initial parallelism to be a reasonable base-line.
??Flink already provides a myriad of config options including default
parallelism, max parallelism and various execution settings, this is not
different from that in my view.??
This is quite different to me since it introduces the concept of
vertex-specific configurations.
It introduces _yet another_ way how the parallelism can be _configured_.
Next you're gonna propose to be able to set the max parallelism, uid, name etc
etc when submitting a job.
> Allow overriding JobVertex parallelisms during job submission
> -
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29501) Allow overriding JobVertex parallelisms during job submission
[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613764#comment-17613764
]
Chesnay Schepler commented on FLINK-29501:
--
?? The upside may be shorter downtime.??
The upside is significantly more potential for optimizations down-the-line,
like deferring a rescaling to a job failure (== low cost rescaling) or it being
properly integrated into batch jobs (==rescale on next stage, similar to the
AdaptiveBatchScheduler).
??rescaling with an api call is not a "persistent" operation. If a user later
wants to upgrade the job using a savepoint, they would have to then tweak the
parallelism of the entire pipeline from the main method or rescale again after
upgrade.??
If the parallelism is being tuned as part of an auto-scaler, wouldn't it make
sense to start from a blank slate if a job is fully resubmitted? I'd assume the
initial parallelism to be a reasonable base-line.
??Flink already provides a myriad of config options including default
parallelism, max parallelism and various execution settings, this is not
different from that in my view.??
This is quite different to me since it introduces the concept of
vertex-specific configurations.
It introduces _yet another_ way how the parallelism can be _configured_.
Next you're gonna propose to be able to set the max parallelism, uid, name etc
etc when submitting a job.
> Allow overriding JobVertex parallelisms during job submission
> -
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29511) Sort properties/schemas in OpenAPI spec
[ https://issues.apache.org/jira/browse/FLINK-29511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29511. Resolution: Fixed master: c8fb99beeccd81f7b2ebfdc7ce178fcc04d100a1 fa8b1232d8eaee0679b6fe160f66db73cf5a4af2 > Sort properties/schemas in OpenAPI spec > --- > > Key: FLINK-29511 > URL: https://issues.apache.org/jira/browse/FLINK-29511 > Project: Flink > Issue Type: Technical Debt > Components: Documentation, Runtime / REST >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > The properties/schema order is currently based on whatever order they were > looked up, which varies as the spec is being extended. > Sort them by name to prevent this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29530) CommittableMessageTypeInfo#toString may force client to have S3 credentials
[
https://issues.apache.org/jira/browse/FLINK-29530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613593#comment-17613593
]
Chesnay Schepler edited comment on FLINK-29530 at 10/6/22 3:06 PM:
---
Maybe the issue isn't that they load the serializer (which one should always be
able to do client-side), but that {{FileSink#getCommittableSerializer}} derives
its serializers from a {{BulkBucketWriter}} instance that it has to create
first and which needs a filesystem path.
was (Author: zentol):
Maybe the issue isn't that they load the serializer (which one should always be
able to do client-side), but that {{FileSink#getCommittableSerializer}} derives
its serializers from a {{BulkBucketWriter}} instance that it has to create
first.
> CommittableMessageTypeInfo#toString may force client to have S3 credentials
> ---
>
> Key: FLINK-29530
> URL: https://issues.apache.org/jira/browse/FLINK-29530
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / FileSystem
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Priority: Critical
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> The {{toString}} implementation calls the committable serializer factory,
> which in the case of the {{FileSink}} loads a filesystem and thus may require
> credentials.
> As a result something as basic as logging the type info can force the client
> to require S3 credentials.
> It shouldn't have such side-effects..
> Stacktrace that a user provided for another issue:
> {code}
> Exception in thread "main" org.apache.flink.util.FlinkRuntimeException: Could
> not create committable serializer.
> at
> org.apache.flink.connector.file.sink.FileSink.getCommittableSerializer(FileSink.java:180)
> at
> org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo.toString(CommittableMessageTypeInfo.java:120)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
> at
> org.apache.flink.api.dag.Transformation.toString(Transformation.java:556)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:506)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.access$200(StreamGraphGenerator.java:131)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator$ContextImpl.transform(StreamGraphGenerator.java:933)
> at java.base/java.lang.Iterable.forEach(Iterable.java:75)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.expand(SinkTransformationTranslator.java:157)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.access$000(SinkTransformationTranslator.java:99)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateInternal(SinkTransformationTranslator.java:89)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:77)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:61)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.translate(StreamGraphGenerator.java:825)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:555)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.generate(StreamGraphGenerator.java:316)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2135)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2121)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1967)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29530) CommittableMessageTypeInfo#toString may force client to have S3 credentials
[
https://issues.apache.org/jira/browse/FLINK-29530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613593#comment-17613593
]
Chesnay Schepler commented on FLINK-29530:
--
Maybe the issue isn't that they load the serializer (which one should always be
able to do client-side), but that {{FileSink#getCommittableSerializer}} derives
its serializers from a {{BulkBucketWriter}} instance that it has to create
first.
> CommittableMessageTypeInfo#toString may force client to have S3 credentials
> ---
>
> Key: FLINK-29530
> URL: https://issues.apache.org/jira/browse/FLINK-29530
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / FileSystem
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Priority: Critical
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> The {{toString}} implementation calls the committable serializer factory,
> which in the case of the {{FileSink}} loads a filesystem and thus may require
> credentials.
> As a result something as basic as logging the type info can force the client
> to require S3 credentials.
> It shouldn't have such side-effects..
> Stacktrace that a user provided for another issue:
> {code}
> Exception in thread "main" org.apache.flink.util.FlinkRuntimeException: Could
> not create committable serializer.
> at
> org.apache.flink.connector.file.sink.FileSink.getCommittableSerializer(FileSink.java:180)
> at
> org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo.toString(CommittableMessageTypeInfo.java:120)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
> at
> org.apache.flink.api.dag.Transformation.toString(Transformation.java:556)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:506)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.access$200(StreamGraphGenerator.java:131)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator$ContextImpl.transform(StreamGraphGenerator.java:933)
> at java.base/java.lang.Iterable.forEach(Iterable.java:75)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.expand(SinkTransformationTranslator.java:157)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.access$000(SinkTransformationTranslator.java:99)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateInternal(SinkTransformationTranslator.java:89)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:77)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:61)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.translate(StreamGraphGenerator.java:825)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:555)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.generate(StreamGraphGenerator.java:316)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2135)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2121)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1967)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29530) CommittableMessageTypeInfo#toString may force client to have S3 credentials
[
https://issues.apache.org/jira/browse/FLINK-29530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613592#comment-17613592
]
Chesnay Schepler commented on FLINK-29530:
--
{{equals()}} and {{hashcode()}} also load the serializer.
> CommittableMessageTypeInfo#toString may force client to have S3 credentials
> ---
>
> Key: FLINK-29530
> URL: https://issues.apache.org/jira/browse/FLINK-29530
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / FileSystem
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Priority: Critical
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> The {{toString}} implementation calls the committable serializer factory,
> which in the case of the {{FileSink}} loads a filesystem and thus may require
> credentials.
> As a result something as basic as logging the type info can force the client
> to require S3 credentials.
> It shouldn't have such side-effects..
> Stacktrace that a user provided for another issue:
> {code}
> Exception in thread "main" org.apache.flink.util.FlinkRuntimeException: Could
> not create committable serializer.
> at
> org.apache.flink.connector.file.sink.FileSink.getCommittableSerializer(FileSink.java:180)
> at
> org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo.toString(CommittableMessageTypeInfo.java:120)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
> at
> org.apache.flink.api.dag.Transformation.toString(Transformation.java:556)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:506)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.access$200(StreamGraphGenerator.java:131)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator$ContextImpl.transform(StreamGraphGenerator.java:933)
> at java.base/java.lang.Iterable.forEach(Iterable.java:75)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.expand(SinkTransformationTranslator.java:157)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.access$000(SinkTransformationTranslator.java:99)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateInternal(SinkTransformationTranslator.java:89)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:77)
> at
> org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:61)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.translate(StreamGraphGenerator.java:825)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:555)
> at
> org.apache.flink.streaming.api.graph.StreamGraphGenerator.generate(StreamGraphGenerator.java:316)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2135)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2121)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1967)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29530) CommittableMessageTypeInfo#toString may force client to have S3 credentials
[
https://issues.apache.org/jira/browse/FLINK-29530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-29530:
-
Description:
The {{toString}} implementation calls the committable serializer factory, which
in the case of the {{FileSink}} loads a filesystem and thus may require
credentials.
As a result something as basic as logging the type info can force the client to
require S3 credentials.
It shouldn't have such side-effects..
Stacktrace that a user provided for another issue:
{code}
Exception in thread "main" org.apache.flink.util.FlinkRuntimeException: Could
not create committable serializer.
at
org.apache.flink.connector.file.sink.FileSink.getCommittableSerializer(FileSink.java:180)
at
org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo.toString(CommittableMessageTypeInfo.java:120)
at java.base/java.lang.String.valueOf(String.java:2951)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
at
org.apache.flink.api.dag.Transformation.toString(Transformation.java:556)
at java.base/java.lang.String.valueOf(String.java:2951)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
at
org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:506)
at
org.apache.flink.streaming.api.graph.StreamGraphGenerator.access$200(StreamGraphGenerator.java:131)
at
org.apache.flink.streaming.api.graph.StreamGraphGenerator$ContextImpl.transform(StreamGraphGenerator.java:933)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at
org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.expand(SinkTransformationTranslator.java:157)
at
org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator$SinkExpander.access$000(SinkTransformationTranslator.java:99)
at
org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateInternal(SinkTransformationTranslator.java:89)
at
org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:77)
at
org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:61)
at
org.apache.flink.streaming.api.graph.StreamGraphGenerator.translate(StreamGraphGenerator.java:825)
at
org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:555)
at
org.apache.flink.streaming.api.graph.StreamGraphGenerator.generate(StreamGraphGenerator.java:316)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2135)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2121)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1967)
{code}
was:
The {{toString}} implementation calls the committable serializer factory, which
in the case of the {{FileSink}} loads a filesystem and thus may require
credentials.
As a result something as basic as logging the type info can force the client to
require S3 credentials.
It shouldn't have such side-effects..
> CommittableMessageTypeInfo#toString may force client to have S3 credentials
> ---
>
> Key: FLINK-29530
> URL: https://issues.apache.org/jira/browse/FLINK-29530
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / FileSystem
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Priority: Critical
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> The {{toString}} implementation calls the committable serializer factory,
> which in the case of the {{FileSink}} loads a filesystem and thus may require
> credentials.
> As a result something as basic as logging the type info can force the client
> to require S3 credentials.
> It shouldn't have such side-effects..
> Stacktrace that a user provided for another issue:
> {code}
> Exception in thread "main" org.apache.flink.util.FlinkRuntimeException: Could
> not create committable serializer.
> at
> org.apache.flink.connector.file.sink.FileSink.getCommittableSerializer(FileSink.java:180)
> at
> org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo.toString(CommittableMessageTypeInfo.java:120)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:172)
> at
>
[jira] [Created] (FLINK-29530) CommittableMessageTypeInfo#toString may force client to have S3 credentials
Chesnay Schepler created FLINK-29530:
Summary: CommittableMessageTypeInfo#toString may force client to
have S3 credentials
Key: FLINK-29530
URL: https://issues.apache.org/jira/browse/FLINK-29530
Project: Flink
Issue Type: Technical Debt
Components: Connectors / FileSystem
Affects Versions: 1.15.0
Reporter: Chesnay Schepler
Fix For: 1.17.0, 1.15.3, 1.16.1
The {{toString}} implementation calls the committable serializer factory, which
in the case of the {{FileSink}} loads a filesystem and thus may require
credentials.
As a result something as basic as logging the type info can force the client to
require S3 credentials.
It shouldn't have such side-effects..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29528) Check if core-default-shaded is still required
[ https://issues.apache.org/jira/browse/FLINK-29528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29528: - Description: fs-hadoop-shaded contains a core-default-shaded.xml that was originally meant to point Hadoop to relocated classes (back when we were relocating Hadoop as well), and to that end we even copied classes from Hadoop. Now that the filesystems no longer relocate Hadoop I'm wondering if we even need this anymore. was: fs-hadoop-shaded contains a core-default-shaded.xml that was originally meant to point Hadoop to relocated classes (back when we were relocating Hadoop as well), and to that end we even copied classes from Hadoop. Now that the filesystems no longer relocate Hadoop I'm wondering if we even needs this anymore. > Check if core-default-shaded is still required > -- > > Key: FLINK-29528 > URL: https://issues.apache.org/jira/browse/FLINK-29528 > Project: Flink > Issue Type: Technical Debt > Components: FileSystems >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.17.0 > > > fs-hadoop-shaded contains a core-default-shaded.xml that was originally meant > to point Hadoop to relocated classes (back when we were relocating Hadoop as > well), and to that end we even copied classes from Hadoop. > Now that the filesystems no longer relocate Hadoop I'm wondering if we even > need this anymore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29528) Check if core-default-shaded is still required
Chesnay Schepler created FLINK-29528: Summary: Check if core-default-shaded is still required Key: FLINK-29528 URL: https://issues.apache.org/jira/browse/FLINK-29528 Project: Flink Issue Type: Technical Debt Components: FileSystems Reporter: Chesnay Schepler Fix For: 1.17.0 fs-hadoop-shaded contains a core-default-shaded.xml that was originally meant to point Hadoop to relocated classes (back when we were relocating Hadoop as well), and to that end we even copied classes from Hadoop. Now that the filesystems no longer relocate Hadoop I'm wondering if we even needs this anymore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29501) Allow overriding JobVertex parallelisms during job submission
[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613509#comment-17613509
]
Chesnay Schepler edited comment on FLINK-29501 at 10/6/22 1:16 PM:
---
I don't really see why we should make this a feature of the job submission.
Ultimately this is just a rescale request. Down the line this will duplicate
the logic of externally triggered rescalings via the REST API in the adaptive
scheduler (which is supposed to be the default for streaming at some point),
and thus in practice adds technical debt.
I'd rather add such a REST endpoint and revisit rescaling for the default
scheduler (if it is required at all; maybe you could just use the adaptive
scheduler).
was (Author: zentol):
I don't really see why we should make this a feature of the job submission.
Ultimately this is just a rescale request. Down the line this will duplicate
the logic of externally triggered rescalings via the REST API in the adaptive
scheduler (which is supposed to be the default for streaming at some point),
and thus in practice adds technical debt.
I'd rather add such a REST endpoint and revisit rescaling for the default
scheduler.
> Allow overriding JobVertex parallelisms during job submission
> -
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29501) Allow overriding JobVertex parallelisms during job submission
[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613509#comment-17613509
]
Chesnay Schepler commented on FLINK-29501:
--
I don't really see why we should make this a feature of the job submission.
Ultimately this is just a rescale request. Down the line this will duplicate
the logic of externally triggered rescalings via the REST API in the adaptive
scheduler (which is supposed to be the default for streaming at some point),
and thus in practice adds technical debt.
I'd rather add such a REST endpoint and revisit rescaling for the default
scheduler.
> Allow overriding JobVertex parallelisms during job submission
> -
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29453) Add uidHash support to State Processor API
[ https://issues.apache.org/jira/browse/FLINK-29453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-29453: - Issue Type: New Feature (was: Improvement) > Add uidHash support to State Processor API > --- > > Key: FLINK-29453 > URL: https://issues.apache.org/jira/browse/FLINK-29453 > Project: Flink > Issue Type: New Feature > Components: API / State Processor >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0 > > > The state process API is currently limited to working with uids. > We should change this since this is a good application for the API. > The API should be extended to support uidHashes wherever a uid is support, > and we should add a method to map uid[hashes] to a different uid[hash]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
