[jira] [Updated] (FLINK-34984) FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)
[ https://issues.apache.org/jira/browse/FLINK-34984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-34984: - Description: The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0 This new architecture is aimed to solve the following challenges brought in the cloud-native era for Flink. 1. Local Disk Constraints in containerization 2. Spiky Resource Usage caused by compaction in the current state model 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) 4. Light and Fast Checkpoint in a native way Design Details can be found in [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855] Proposed changes can be found here: * [Asynchronous State APIs (FLIP-424) |https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-AsynchronousStateAPIs(FLIP-424)] * [Non-blocking Asynchronous Execution Model: Parallel I/O (FLIP-425)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-Non-blockingAsynchronousExecutionModel:ParallelI/O(FLIP-425)] * [Batching for Network I/O: Beyond Parallel I/O (FLIP-426)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-BatchingforNetworkI/O:BeyondParallelI/O(FLIP-426)] * [Disaggregated State Store: ForSt (FLIP-427)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-DisaggregatedStateStore:ForSt(FLIP-427)] * [Faster Checkpoint/Restore/Rescale: Leverage Shared DFS (FLIP-428)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-FasterCheckpoint/Restore/Rescale:LeverageSharedDFS(FLIP-428)] was: The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0 This new architecture is aimed to solve the following challenges brought in the cloud-native era for Flink. 1. Local Disk Constraints in containerization 2. Spiky Resource Usage caused by compaction in the current state model 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) 4. Light and Fast Checkpoint in a native way Design Details can be found in [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855] > FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP) > > > Key: FLINK-34984 > URL: https://issues.apache.org/jira/browse/FLINK-34984 > Project: Flink > Issue Type: New Feature > Components: API / Core, API / DataStream, Runtime / Checkpointing, > Runtime / State Backends >Reporter: Yuan Mei >Priority: Major > > The past decade has witnessed a dramatic shift in Flink's deployment mode, > workload patterns, and hardware improvements. We've moved from the map-reduce > era where workers are computation-storage tightly coupled nodes to a > cloud-native world where containerized deployments on Kubernetes become > standard. To enable Flink's Cloud-Native future, we introduce Disaggregated > State Storage and Management that uses DFS as primary storage in Flink 2.0 > This new architecture is aimed to solve the following challenges brought in > the cloud-native era for Flink. > 1. Local Disk Constraints in containerization > 2. Spiky Resource Usage caused by compaction in the current state model > 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) > 4. Light and Fast Checkpoint in a native way > > Design Details can be found in > [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855] > Proposed changes can be found here: > * [Asynchronous State APIs (FLIP-424) > |https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-AsynchronousStateAPIs(FLIP-424)] > *
[jira] [Updated] (FLINK-34984) FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)
[ https://issues.apache.org/jira/browse/FLINK-34984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-34984: - Description: The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0 This new architecture is aimed to solve the following challenges brought in the cloud-native era for Flink. 1. Local Disk Constraints in containerization 2. Spiky Resource Usage caused by compaction in the current state model 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) 4. Light and Fast Checkpoint in a native way Design Details can be found in [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855] was: The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0 Design Details can be found in [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855]. This new architecture is aimed to solve the following challenges brought in the cloud-native era for Flink. 1. Local Disk Constraints in containerization 2. Spiky Resource Usage caused by compaction in the current state model 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) 4. Light and Fast Checkpoint in a native way > FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP) > > > Key: FLINK-34984 > URL: https://issues.apache.org/jira/browse/FLINK-34984 > Project: Flink > Issue Type: New Feature > Components: API / Core, API / DataStream, Runtime / Checkpointing, > Runtime / State Backends >Reporter: Yuan Mei >Priority: Major > > The past decade has witnessed a dramatic shift in Flink's deployment mode, > workload patterns, and hardware improvements. We've moved from the map-reduce > era where workers are computation-storage tightly coupled nodes to a > cloud-native world where containerized deployments on Kubernetes become > standard. To enable Flink's Cloud-Native future, we introduce Disaggregated > State Storage and Management that uses DFS as primary storage in Flink 2.0 > This new architecture is aimed to solve the following challenges brought in > the cloud-native era for Flink. > 1. Local Disk Constraints in containerization > 2. Spiky Resource Usage caused by compaction in the current state model > 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) > 4. Light and Fast Checkpoint in a native way > > Design Details can be found in > [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34984) FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)
[ https://issues.apache.org/jira/browse/FLINK-34984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-34984: - Description: The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0 Design Details can be found in [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855]. This new architecture is aimed to solve the following challenges brought in the cloud-native era for Flink. 1. Local Disk Constraints in containerization 2. Spiky Resource Usage caused by compaction in the current state model 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) 4. Light and Fast Checkpoint in a native way was: The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0, as promised in the Flink 2.0 Roadmap. Detailed design and story: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855 Also sub-FLIPs: - Asynchronous State APIs ([FLIP-424|https://cwiki.apache.org/confluence/x/SYp3EQ]): Introduce new APIs for asynchronous state access. - Asynchronous Execution Model ([FLIP-425|https://cwiki.apache.org/confluence/x/S4p3EQ]): Implement a non-blocking execution model leveraging the asynchronous APIs introduced in FLIP-424. - Grouping Remote State Access ([FLIP-426|https://cwiki.apache.org/confluence/x/TYp3EQ]): Enable retrieval of remote state data in batches to avoid unnecessary round-trip costs for remote access. - Disaggregated State Store ([FLIP-427|https://cwiki.apache.org/confluence/x/T4p3EQ]): Introduce the initial version of the ForSt disaggregated state store. - Fault Tolerance/Rescale Integration ([FLIP-428|https://cwiki.apache.org/confluence/x/UYp3EQ]): Integrate checkpointing mechanisms with the disaggregated state store for fault tolerance and fast rescaling. > FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP) > > > Key: FLINK-34984 > URL: https://issues.apache.org/jira/browse/FLINK-34984 > Project: Flink > Issue Type: New Feature > Components: API / Core, API / DataStream, Runtime / Checkpointing, > Runtime / State Backends >Reporter: Yuan Mei >Priority: Major > > The past decade has witnessed a dramatic shift in Flink's deployment mode, > workload patterns, and hardware improvements. We've moved from the map-reduce > era where workers are computation-storage tightly coupled nodes to a > cloud-native world where containerized deployments on Kubernetes become > standard. To enable Flink's Cloud-Native future, we introduce Disaggregated > State Storage and Management that uses DFS as primary storage in Flink 2.0 > > Design Details can be found in > [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855]. > This new architecture is aimed to solve the following challenges brought in > the cloud-native era for Flink. > 1. Local Disk Constraints in containerization > 2. Spiky Resource Usage caused by compaction in the current state model > 3. Fast Rescaling for jobs with large states (hundreds of Terabytes) > 4. Light and Fast Checkpoint in a native way -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34984) Disaggregated State Storage and Management (Umbrella FLIP)
[ https://issues.apache.org/jira/browse/FLINK-34984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-34984: - Description: The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0, as promised in the Flink 2.0 Roadmap. > Disaggregated State Storage and Management (Umbrella FLIP) > -- > > Key: FLINK-34984 > URL: https://issues.apache.org/jira/browse/FLINK-34984 > Project: Flink > Issue Type: New Feature > Components: API / Core, API / DataStream, Runtime / Checkpointing, > Runtime / State Backends >Reporter: Yuan Mei >Priority: Major > > The past decade has witnessed a dramatic shift in Flink's deployment mode, > workload patterns, and hardware improvements. We've moved from the map-reduce > era where workers are computation-storage tightly coupled nodes to a > cloud-native world where containerized deployments on Kubernetes become > standard. To enable Flink's Cloud-Native future, we introduce Disaggregated > State Storage and Management that uses DFS as primary storage in Flink 2.0, > as promised in the Flink 2.0 Roadmap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34984) Disaggregated State Storage and Management (Umbrella FLIP)
Yuan Mei created FLINK-34984: Summary: Disaggregated State Storage and Management (Umbrella FLIP) Key: FLINK-34984 URL: https://issues.apache.org/jira/browse/FLINK-34984 Project: Flink Issue Type: New Feature Components: API / Core, API / DataStream, Runtime / Checkpointing, Runtime / State Backends Reporter: Yuan Mei -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32073) Implement file merging in snapshot
[ https://issues.apache.org/jira/browse/FLINK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei closed FLINK-32073. Resolution: Done > Implement file merging in snapshot > -- > > Key: FLINK-32073 > URL: https://issues.apache.org/jira/browse/FLINK-32073 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32073) Implement file merging in snapshot
[
https://issues.apache.org/jira/browse/FLINK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809461#comment-17809461
]
Yuan Mei commented on FLINK-32073:
--
merged commit
[{{f9f9299}}|https://github.com/apache/flink/commit/f9f9299f6e25080c6f869b46ec0bdc5e3e19e00d]
into apache:master
> Implement file merging in snapshot
> --
>
> Key: FLINK-32073
> URL: https://issues.apache.org/jira/browse/FLINK-32073
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.18.0
>Reporter: Zakelly Lan
>Assignee: Han Yin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26490) Adjust the MaxParallelism or remove the MaxParallelism check when unnecessary.
[ https://issues.apache.org/jira/browse/FLINK-26490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785394#comment-17785394 ] Yuan Mei commented on FLINK-26490: -- Sorry to respond late. Just saw this ticket recently. I think we should be cautious about loosing checks for MaxParallelism. # In general, MaxParallelism is not expected to be changed (at least changed often) after starting a job. A more strict check on "MaxParallelism" is not too big a problem. # The other thing is should OPstate be completely not affected by "MaxParallelism". I know it is not affected today, but if we change the contract between "MaxParallelism" and "State", we need to think of all these corner cases if we make any changes later. I am concerned about making such changes. > Adjust the MaxParallelism or remove the MaxParallelism check when unnecessary. > -- > > Key: FLINK-26490 > URL: https://issues.apache.org/jira/browse/FLINK-26490 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: chenfengLiu >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > pull-request-available > > Since Flink introduce key group and MaxParallelism, Flink can rescale with > less cost. > But when we want to update the job parallelism bigger than the > MaxParallelism, it 's impossible cause there are so many MaxParallelism check > that require new parallelism should not bigger than MaxParallelism. > Actually, when an operator which don't contain keyed state, there should be > no problem when update the parallelism bigger than the MaxParallelism,, cause > only keyed state need MaxParallelism and key group. > So should we remove this check or auto adjust the MaxParallelism when we > restore an operator state that don't contain keyed state? > It can make job restore from checkpoint easier. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-17916) Provide API to separate KafkaShuffle's Producer and Consumer to different jobs
[ https://issues.apache.org/jira/browse/FLINK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei closed FLINK-17916. Resolution: Won't Do > Provide API to separate KafkaShuffle's Producer and Consumer to different jobs > -- > > Key: FLINK-17916 > URL: https://issues.apache.org/jira/browse/FLINK-17916 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / Kafka >Affects Versions: 1.11.0 >Reporter: Yuan Mei >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > > Follow up of FLINK-15670 > *Separate sink (producer) and source (consumer) to different jobs* > * In the same job, a sink and a source are recovered independently according > to regional failover. However, they share the same checkpoint coordinator and > correspondingly, share the same global checkpoint snapshot. > * That says if the consumer fails, the producer can not commit written data > because of two-phase commit set-up (the producer needs a checkpoint-complete > signal to complete the second stage) > * Same applies to the producer > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-17916) Provide API to separate KafkaShuffle's Producer and Consumer to different jobs
[ https://issues.apache.org/jira/browse/FLINK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776099#comment-17776099 ] Yuan Mei commented on FLINK-17916: -- Yes, I think so. Thanks for reminding [~martijnvisser] > Provide API to separate KafkaShuffle's Producer and Consumer to different jobs > -- > > Key: FLINK-17916 > URL: https://issues.apache.org/jira/browse/FLINK-17916 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / Kafka >Affects Versions: 1.11.0 >Reporter: Yuan Mei >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > > Follow up of FLINK-15670 > *Separate sink (producer) and source (consumer) to different jobs* > * In the same job, a sink and a source are recovered independently according > to regional failover. However, they share the same checkpoint coordinator and > correspondingly, share the same global checkpoint snapshot. > * That says if the consumer fails, the producer can not commit written data > because of two-phase commit set-up (the producer needs a checkpoint-complete > signal to complete the second stage) > * Same applies to the producer > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33052) codespeed and benchmark server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774779#comment-17774779 ] Yuan Mei commented on FLINK-33052: -- As the pipeline has been running daily, and the reported results look OK for 1.18 (as referred to in this email discussion[1]), this ticket is no longer a blocker for the 1.18 release, so I've moved the fix version to 1.19.0 > codespeed and benchmark server is down > -- > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.19.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed and benchmark server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774779#comment-17774779 ] Yuan Mei edited comment on FLINK-33052 at 10/13/23 4:19 AM: As the pipeline has been running daily, and the reported results look OK for 1.18 (as referred to in this email discussion[1]), this ticket is no longer a blocker for the 1.18 release, so I've moved the fix version to 1.19.0 [1] https://www.mail-archive.com/dev@flink.apache.org/msg69920.html was (Author: ym): As the pipeline has been running daily, and the reported results look OK for 1.18 (as referred to in this email discussion[1]), this ticket is no longer a blocker for the 1.18 release, so I've moved the fix version to 1.19.0 > codespeed and benchmark server is down > -- > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.19.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33052) codespeed and benchmark server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-33052: - Affects Version/s: 1.19.0 (was: 1.18.0) > codespeed and benchmark server is down > -- > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.19.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 2:14 PM: --- Hey [~pnowojski] # Strictly speaking, yes it is a blocker for release. # But since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged), maybe we should not make this a blocker issue for RC? # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # I agree we should have this set up as soon as possible, best before 1.18 release. was (Author: ym): Hey [~pnowojski] # Strictly speaking, yes it is a blocker for release. # But since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged), maybe we should not make this a blocker issue for RC? # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # I agree we should have this set up before 1.18 release. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:44 PM: --- Hey [~pnowojski] # Strictly speaking, yes it is a blocker for release. # But since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged), maybe we should not make this a blocker issue for RC? # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # I agree we should have this set up before 1.18 release. was (Author: ym): Hey [~pnowojski] # Strictly speaking, yes it is a blocker for release. # But since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged), maybe we should not make this a blocker issue for RC1 of 1.18 (RC0 is out, waiting for RC1)? # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # I agree we should have this set up before 1.18 release. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:41 PM: --- Hey [~pnowojski] # Strictly speaking, yes it is a blocker for release. # But since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged), maybe we should not make this a blocker issue for RC1 of 1.18 (RC0 is out, waiting for RC1)? # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # I agree we should have this set up before 1.18 release. was (Author: ym): Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 (RC0 is out, waiting for RC1) # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker for 1.18, we can chat off-line as well. Or at least I do not want this to block the RC? > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:39 PM: --- Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 (RC0 is out, waiting for RC1) # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker for 1.18, we can chat off-line as well. Or at least I do not want this to block the RC? was (Author: ym): Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 (RC0 is out, waiting for RC1) # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker for 1.18, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:35 PM: --- Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker for 1.18, we can chat off-line as well. was (Author: ym): Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:35 PM: --- Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 (RC0 is out, waiting for RC1) # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker for 1.18, we can chat off-line as well. was (Author: ym): Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker for 1.18, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:31 PM: --- Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. was (Author: ym): Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:31 PM: --- Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. was (Author: ym): Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:31 PM: --- Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. was (Author: ym): Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 1:30 PM: --- Hey [~pnowojski] # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. was (Author: ym): # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764675#comment-17764675 ] Yuan Mei commented on FLINK-33052: -- # I think this issue is very critical as you mentioned # Since 1.18 has been branch cut and most of the tests have already been done (No feature is allowed to be merged). I do not see why this should be a blocker issue for release 1.18 # [~Zakelly] has already been working on this, but as you can see this issue takes time (applying grants for buying new machines and set up everything, e.t.c) # If you think this is indeed a blocker, we can chat off-line as well. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-33052: - Priority: Critical (was: Blocker) > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Critical > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764570#comment-17764570 ] Yuan Mei commented on FLINK-33052: -- [~pnowojski] We have the historical data backed up. But just in case, we will still have the worker node there before the new setup. Also, I would say this ticket is not a blocker for release 1.18, since 1.18 is already branch-cut and in the stage of manual testing? > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei reassigned FLINK-33052: Assignee: Zakelly Lan (was: Yuan Mei) > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Test Infrastructure >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Major > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764484#comment-17764484 ] Yuan Mei edited comment on FLINK-33052 at 9/13/23 4:44 AM: --- Let me see whether i can get a new machine for this purpose and re-set up the environment. was (Author: ym): Let me see whether i can get a new machine for this purpose. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Test Infrastructure >Reporter: Jing Ge >Assignee: Yuan Mei >Priority: Major > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764484#comment-17764484 ] Yuan Mei commented on FLINK-33052: -- Let me see whether i can get a new machine for this purpose. > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Test Infrastructure >Reporter: Jing Ge >Assignee: Yuan Mei >Priority: Major > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33052) codespeed server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei reassigned FLINK-33052: Assignee: Yuan Mei (was: Yanfei Lei) > codespeed server is down > > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Test Infrastructure >Reporter: Jing Ge >Assignee: Yuan Mei >Priority: Major > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-25842: - Description: Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] 1st iteration: FLINK-21352 == Updated on July 23: The second half of Changelog is done in the 1.17 release. There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. Moved to FLINK-32651 We've had thorough testing and performance evaluation. This feature is prod ready. More details can be found in [Flink 1.17 release note|https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#checkpoint-improvements] and [this blog post|https://www.ververica.com/blog/generic-log-based-incremental-checkpoint] was: Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] 1st iteration: FLINK-21352 > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.18.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 > == > Updated on July 23: > The second half of Changelog is done in the 1.17 release. > There are two remaining tasks related to 1) DFS writer benchmarking and 2) > rescaling benchmarking, which I believe should not be a block for this. Moved > to FLINK-32651 > We've had thorough testing and performance evaluation. This feature is prod > ready. > More details can be found in [Flink 1.17 release > note|https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#checkpoint-improvements] > and [this blog > post|https://www.ververica.com/blog/generic-log-based-incremental-checkpoint] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746065#comment-17746065 ] Yuan Mei edited comment on FLINK-25842 at 7/23/23 12:31 PM: The second half of Changelog is done in the 1.17 release. There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. Moved to FLINK-32651 We've had thorough testing and performance evaluation. This feature is prod ready. More details can be found in [Flink 1.17 release note|https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#checkpoint-improvements] and [this blog post|https://www.ververica.com/blog/generic-log-based-incremental-checkpoint] was (Author: ym): The second half of Changelog is done in the 1.17 release. There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. Moved to We've had thorough testing and performance evaluation. This feature is prod ready. More details can be found in [Flink 1.17 release note|https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#checkpoint-improvements] and [this blog post|https://www.ververica.com/blog/generic-log-based-incremental-checkpoint] > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.18.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842 ] Yuan Mei deleted comment on FLINK-25842: -- was (Author: ym): The second half of Changelog is done in the 1.17 release. There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. We've had thorough testing and performance evaluation. This feature is prod ready. More details can be found in [Flink 1.17 release note|https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#checkpoint-improvements] and [this blog post|https://www.ververica.com/blog/generic-log-based-incremental-checkpoint] > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.18.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746065#comment-17746065 ] Yuan Mei commented on FLINK-25842: -- The second half of Changelog is done in the 1.17 release. There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. Moved to We've had thorough testing and performance evaluation. This feature is prod ready. More details can be found in [Flink 1.17 release note|https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#checkpoint-improvements] and [this blog post|https://www.ververica.com/blog/generic-log-based-incremental-checkpoint] > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.18.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746064#comment-17746064 ] Yuan Mei commented on FLINK-25842: -- The second half of Changelog is done in the 1.17 release. There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. We've had thorough testing and performance evaluation. This feature is prod ready. More details can be found in [Flink 1.17 release note|https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#checkpoint-improvements] and [this blog post|https://www.ververica.com/blog/generic-log-based-incremental-checkpoint] > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.18.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei resolved FLINK-25842. -- Resolution: Fixed Move the remaining two benchmarks to follow-ups. > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.18.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-23485) Add a benchmark for the Changelog DFS writer
[ https://issues.apache.org/jira/browse/FLINK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-23485: - Parent Issue: FLINK-32651 (was: FLINK-25842) > Add a benchmark for the Changelog DFS writer > > > Key: FLINK-23485 > URL: https://issues.apache.org/jira/browse/FLINK-23485 > Project: Flink > Issue Type: Sub-task > Components: Benchmarks >Reporter: Roman Khachatryan >Priority: Major > > http://codespeed.dak8s.net:8000/ > Besides of adding a benchmark, update (or better re-write) the script for > regression detection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32364) Add Rescaling benchmark for ChangelogStateBackend
[ https://issues.apache.org/jira/browse/FLINK-32364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32364: - Parent Issue: FLINK-32651 (was: FLINK-25842) > Add Rescaling benchmark for ChangelogStateBackend > - > > Key: FLINK-32364 > URL: https://issues.apache.org/jira/browse/FLINK-32364 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Hangxiang Yu >Assignee: Hangxiang Yu >Priority: Minor > > After FLINK-23484, we could Supports rescaling benchmark just like HEAP and > ROCKSDB. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32651) Benchmark Support for Changelog Statebackend
Yuan Mei created FLINK-32651: Summary: Benchmark Support for Changelog Statebackend Key: FLINK-32651 URL: https://issues.apache.org/jira/browse/FLINK-32651 Project: Flink Issue Type: Improvement Reporter: Yuan Mei -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32364) Add Rescaling benchmark for ChangelogStateBackend
[ https://issues.apache.org/jira/browse/FLINK-32364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32364: - Priority: Minor (was: Major) > Add Rescaling benchmark for ChangelogStateBackend > - > > Key: FLINK-32364 > URL: https://issues.apache.org/jira/browse/FLINK-32364 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Hangxiang Yu >Assignee: Hangxiang Yu >Priority: Minor > > After FLINK-23484, we could Supports rescaling benchmark just like HEAP and > ROCKSDB. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-28581) Test Changelog StateBackend V2 Manually
[ https://issues.apache.org/jira/browse/FLINK-28581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei reassigned FLINK-28581: Assignee: Hangxiang Yu > Test Changelog StateBackend V2 Manually > --- > > Key: FLINK-28581 > URL: https://issues.apache.org/jira/browse/FLINK-28581 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Hangxiang Yu >Assignee: Hangxiang Yu >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-32485) Flink State Backend Changelog should support build test-jar
[
https://issues.apache.org/jira/browse/FLINK-32485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738861#comment-17738861
]
Yuan Mei edited comment on FLINK-32485 at 6/30/23 3:34 AM:
---
I can take a look
"test-jar" is usually used to share test code between modules.
If I remember correctly, there should not be any other module sharing the test
code of the changelog (It has been changed quite a lot, so I may make a
mistake).
Do you know where "ChangelogStateBackendTestUtils" is used (other than
flink-statebackend-changelog)?
was (Author: ym):
I can take a look
"test-jar" is usually used to share test code between modules.
If I remember correctly, there should not be another module sharing the test
code of the changelog (It has been changed quite a lot, so I may make a
mistake).
Do you know where "ChangelogStateBackendTestUtils" is used (other than
flink-statebackend-changelog)?
> Flink State Backend Changelog should support build test-jar
> ---
>
> Key: FLINK-32485
> URL: https://issues.apache.org/jira/browse/FLINK-32485
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.17.1
>Reporter: Ran Tao
>Priority: Major
> Labels: pull-request-available
>
> In some scenarios, executing unit tests will report the following errors. In
> fact, since flink-state-backend-changelog test contains some util classes, we
> should build test jar like flink-rocks-db backend.
> {code:java}
> /Users/xxx/github/flink/flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java:29:37
> java: Package org.apache.flink.changelog.fs not exist {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-32485) Flink State Backend Changelog should support build test-jar
[
https://issues.apache.org/jira/browse/FLINK-32485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738861#comment-17738861
]
Yuan Mei edited comment on FLINK-32485 at 6/30/23 3:34 AM:
---
I can take a look
"test-jar" is usually used to share test code between modules.
If I remember correctly, there should not be any other module sharing the test
code of the changelog (It has been changed quite a lot, so I may make a
mistake).
Do you know where `ChangelogStateBackendTestUtils` is used (other than
flink-statebackend-changelog)?
was (Author: ym):
I can take a look
"test-jar" is usually used to share test code between modules.
If I remember correctly, there should not be any other module sharing the test
code of the changelog (It has been changed quite a lot, so I may make a
mistake).
Do you know where "ChangelogStateBackendTestUtils" is used (other than
flink-statebackend-changelog)?
> Flink State Backend Changelog should support build test-jar
> ---
>
> Key: FLINK-32485
> URL: https://issues.apache.org/jira/browse/FLINK-32485
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.17.1
>Reporter: Ran Tao
>Priority: Major
> Labels: pull-request-available
>
> In some scenarios, executing unit tests will report the following errors. In
> fact, since flink-state-backend-changelog test contains some util classes, we
> should build test jar like flink-rocks-db backend.
> {code:java}
> /Users/xxx/github/flink/flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java:29:37
> java: Package org.apache.flink.changelog.fs not exist {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32485) Flink State Backend Changelog should support build test-jar
[
https://issues.apache.org/jira/browse/FLINK-32485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738861#comment-17738861
]
Yuan Mei commented on FLINK-32485:
--
I can take a look
"test-jar" is usually used to share test code between modules.
If I remember correctly, there should not be another module sharing the test
code of the changelog (It has been changed quite a lot, so I may make a
mistake).
Do you know where "ChangelogStateBackendTestUtils" is used (other than
flink-statebackend-changelog)?
> Flink State Backend Changelog should support build test-jar
> ---
>
> Key: FLINK-32485
> URL: https://issues.apache.org/jira/browse/FLINK-32485
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.17.1
>Reporter: Ran Tao
>Priority: Major
> Labels: pull-request-available
>
> In some scenarios, executing unit tests will report the following errors. In
> fact, since flink-state-backend-changelog test contains some util classes, we
> should build test jar like flink-rocks-db backend.
> {code:java}
> /Users/xxx/github/flink/flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java:29:37
> java: Package org.apache.flink.changelog.fs not exist {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32071) Implement the snapshot manager for merged checkpoint files in TM
[
https://issues.apache.org/jira/browse/FLINK-32071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736556#comment-17736556
]
Yuan Mei commented on FLINK-32071:
--
commit
[{{01560a1}}|https://github.com/apache/flink/commit/01560a154fba41af4a9354da2588f78ea98089b7]
into apache:master
> Implement the snapshot manager for merged checkpoint files in TM
>
>
> Key: FLINK-32071
> URL: https://issues.apache.org/jira/browse/FLINK-32071
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.18.0
>Reporter: Zakelly Lan
>Assignee: Zakelly Lan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> As the first part of FLIP-306, which aims to provide an unified file merging
> mechanism for checkpoint files, this ticket/PR implements the basic file
> manager for merged files, providing file system directory layout and its
> initialization. It also provides the implementation of physical and logical
> files and the reference counting relationship between them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-32071) Implement the snapshot manager for merged checkpoint files in TM
[ https://issues.apache.org/jira/browse/FLINK-32071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei closed FLINK-32071. Resolution: Done > Implement the snapshot manager for merged checkpoint files in TM > > > Key: FLINK-32071 > URL: https://issues.apache.org/jira/browse/FLINK-32071 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > > As the first part of FLIP-306, which aims to provide an unified file merging > mechanism for checkpoint files, this ticket/PR implements the basic file > manager for merged files, providing file system directory layout and its > initialization. It also provides the implementation of physical and logical > files and the reference counting relationship between them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32092) Integrate snapshot file-merging with existing IT cases
[ https://issues.apache.org/jira/browse/FLINK-32092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32092: - Affects Version/s: 1.18.0 > Integrate snapshot file-merging with existing IT cases > -- > > Key: FLINK-32092 > URL: https://issues.apache.org/jira/browse/FLINK-32092 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Rui Xia >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32092) Integrate snapshot file-merging with existing IT cases
[ https://issues.apache.org/jira/browse/FLINK-32092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32092: - Component/s: Runtime / Checkpointing Runtime / State Backends > Integrate snapshot file-merging with existing IT cases > -- > > Key: FLINK-32092 > URL: https://issues.apache.org/jira/browse/FLINK-32092 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Rui Xia >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32091) Add necessary metrics for file-merging
[ https://issues.apache.org/jira/browse/FLINK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32091: - Component/s: Runtime / Checkpointing Runtime / State Backends > Add necessary metrics for file-merging > -- > > Key: FLINK-32091 > URL: https://issues.apache.org/jira/browse/FLINK-32091 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32091) Add necessary metrics for file-merging
[ https://issues.apache.org/jira/browse/FLINK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32091: - Affects Version/s: 1.18.0 > Add necessary metrics for file-merging > -- > > Key: FLINK-32091 > URL: https://issues.apache.org/jira/browse/FLINK-32091 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32090) Python API for enabling and configuring file merging snapshot
[ https://issues.apache.org/jira/browse/FLINK-32090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32090: - Component/s: Runtime / Checkpointing Runtime / State Backends > Python API for enabling and configuring file merging snapshot > - > > Key: FLINK-32090 > URL: https://issues.apache.org/jira/browse/FLINK-32090 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32087) Space amplification statistics of file merging
[ https://issues.apache.org/jira/browse/FLINK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32087: - Component/s: Runtime / Checkpointing Runtime / State Backends > Space amplification statistics of file merging > -- > > Key: FLINK-32087 > URL: https://issues.apache.org/jira/browse/FLINK-32087 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Rui Xia >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32090) Python API for enabling and configuring file merging snapshot
[ https://issues.apache.org/jira/browse/FLINK-32090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32090: - Affects Version/s: 1.18.0 > Python API for enabling and configuring file merging snapshot > - > > Key: FLINK-32090 > URL: https://issues.apache.org/jira/browse/FLINK-32090 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32089) Do fast copy in best-effort during first checkpoint after restoration
[ https://issues.apache.org/jira/browse/FLINK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32089: - Affects Version/s: 1.18.0 > Do fast copy in best-effort during first checkpoint after restoration > - > > Key: FLINK-32089 > URL: https://issues.apache.org/jira/browse/FLINK-32089 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32089) Do fast copy in best-effort during first checkpoint after restoration
[ https://issues.apache.org/jira/browse/FLINK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32089: - Component/s: Runtime / Checkpointing Runtime / State Backends > Do fast copy in best-effort during first checkpoint after restoration > - > > Key: FLINK-32089 > URL: https://issues.apache.org/jira/browse/FLINK-32089 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32088) Re-uploading in state file-merging for space amplification control
[ https://issues.apache.org/jira/browse/FLINK-32088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32088: - Affects Version/s: 1.18.0 > Re-uploading in state file-merging for space amplification control > -- > > Key: FLINK-32088 > URL: https://issues.apache.org/jira/browse/FLINK-32088 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32087) Space amplification statistics of file merging
[ https://issues.apache.org/jira/browse/FLINK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32087: - Affects Version/s: 1.18.0 > Space amplification statistics of file merging > -- > > Key: FLINK-32087 > URL: https://issues.apache.org/jira/browse/FLINK-32087 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Rui Xia >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32088) Re-uploading in state file-merging for space amplification control
[ https://issues.apache.org/jira/browse/FLINK-32088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32088: - Component/s: Runtime / Checkpointing Runtime / State Backends > Re-uploading in state file-merging for space amplification control > -- > > Key: FLINK-32088 > URL: https://issues.apache.org/jira/browse/FLINK-32088 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32086) Cleanup non-reported managed directory on exit of TM
[ https://issues.apache.org/jira/browse/FLINK-32086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32086: - Component/s: Runtime / Checkpointing Runtime / State Backends > Cleanup non-reported managed directory on exit of TM > > > Key: FLINK-32086 > URL: https://issues.apache.org/jira/browse/FLINK-32086 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32086) Cleanup non-reported managed directory on exit of TM
[ https://issues.apache.org/jira/browse/FLINK-32086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32086: - Affects Version/s: 1.18.0 > Cleanup non-reported managed directory on exit of TM > > > Key: FLINK-32086 > URL: https://issues.apache.org/jira/browse/FLINK-32086 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32085) Implement and migrate batch uploading in changelog files into the file merging framework
[ https://issues.apache.org/jira/browse/FLINK-32085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32085: - Affects Version/s: 1.18.0 > Implement and migrate batch uploading in changelog files into the file > merging framework > > > Key: FLINK-32085 > URL: https://issues.apache.org/jira/browse/FLINK-32085 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32085) Implement and migrate batch uploading in changelog files into the file merging framework
[ https://issues.apache.org/jira/browse/FLINK-32085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32085: - Component/s: Runtime / Checkpointing Runtime / State Backends > Implement and migrate batch uploading in changelog files into the file > merging framework > > > Key: FLINK-32085 > URL: https://issues.apache.org/jira/browse/FLINK-32085 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32083) Chinese translation of documentation of checkpoint file-merging
[ https://issues.apache.org/jira/browse/FLINK-32083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32083: - Component/s: Runtime / Checkpointing Runtime / State Backends > Chinese translation of documentation of checkpoint file-merging > --- > > Key: FLINK-32083 > URL: https://issues.apache.org/jira/browse/FLINK-32083 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32084) Migrate current file merging of channel state into the file merging framework
[ https://issues.apache.org/jira/browse/FLINK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32084: - Affects Version/s: 1.18.0 > Migrate current file merging of channel state into the file merging framework > - > > Key: FLINK-32084 > URL: https://issues.apache.org/jira/browse/FLINK-32084 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32084) Migrate current file merging of channel state into the file merging framework
[ https://issues.apache.org/jira/browse/FLINK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32084: - Component/s: Runtime / Checkpointing Runtime / State Backends > Migrate current file merging of channel state into the file merging framework > - > > Key: FLINK-32084 > URL: https://issues.apache.org/jira/browse/FLINK-32084 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32083) Chinese translation of documentation of checkpoint file-merging
[ https://issues.apache.org/jira/browse/FLINK-32083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32083: - Affects Version/s: 1.18.0 > Chinese translation of documentation of checkpoint file-merging > --- > > Key: FLINK-32083 > URL: https://issues.apache.org/jira/browse/FLINK-32083 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32082) Documentation of checkpoint file-merging
[ https://issues.apache.org/jira/browse/FLINK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32082: - Component/s: Runtime / Checkpointing Runtime / State Backends > Documentation of checkpoint file-merging > > > Key: FLINK-32082 > URL: https://issues.apache.org/jira/browse/FLINK-32082 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32082) Documentation of checkpoint file-merging
[ https://issues.apache.org/jira/browse/FLINK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32082: - Affects Version/s: 1.18.0 > Documentation of checkpoint file-merging > > > Key: FLINK-32082 > URL: https://issues.apache.org/jira/browse/FLINK-32082 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32081) Compatibility between file-merging on and off across job runs
[ https://issues.apache.org/jira/browse/FLINK-32081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32081: - Component/s: Runtime / Checkpointing Runtime / State Backends > Compatibility between file-merging on and off across job runs > - > > Key: FLINK-32081 > URL: https://issues.apache.org/jira/browse/FLINK-32081 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32081) Compatibility between file-merging on and off across job runs
[ https://issues.apache.org/jira/browse/FLINK-32081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32081: - Affects Version/s: 1.18.0 > Compatibility between file-merging on and off across job runs > - > > Key: FLINK-32081 > URL: https://issues.apache.org/jira/browse/FLINK-32081 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32080) Restoration of FileMergingSnapshotManager
[ https://issues.apache.org/jira/browse/FLINK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32080: - Component/s: Runtime / Checkpointing Runtime / State Backends > Restoration of FileMergingSnapshotManager > - > > Key: FLINK-32080 > URL: https://issues.apache.org/jira/browse/FLINK-32080 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32080) Restoration of FileMergingSnapshotManager
[ https://issues.apache.org/jira/browse/FLINK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32080: - Affects Version/s: 1.18.0 > Restoration of FileMergingSnapshotManager > - > > Key: FLINK-32080 > URL: https://issues.apache.org/jira/browse/FLINK-32080 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32079) Read/write checkpoint metadata of merged files
[ https://issues.apache.org/jira/browse/FLINK-32079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32079: - Affects Version/s: 1.18.0 > Read/write checkpoint metadata of merged files > -- > > Key: FLINK-32079 > URL: https://issues.apache.org/jira/browse/FLINK-32079 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32079) Read/write checkpoint metadata of merged files
[ https://issues.apache.org/jira/browse/FLINK-32079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32079: - Component/s: Runtime / Checkpointing Runtime / State Backends > Read/write checkpoint metadata of merged files > -- > > Key: FLINK-32079 > URL: https://issues.apache.org/jira/browse/FLINK-32079 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Hangxiang Yu >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32078) Implement private state file merging
[ https://issues.apache.org/jira/browse/FLINK-32078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32078: - Component/s: Runtime / Checkpointing Runtime / State Backends > Implement private state file merging > > > Key: FLINK-32078 > URL: https://issues.apache.org/jira/browse/FLINK-32078 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32077) Implement shared state file merging
[ https://issues.apache.org/jira/browse/FLINK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32077: - Component/s: Runtime / Checkpointing Runtime / State Backends > Implement shared state file merging > --- > > Key: FLINK-32077 > URL: https://issues.apache.org/jira/browse/FLINK-32077 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32077) Implement shared state file merging
[ https://issues.apache.org/jira/browse/FLINK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32077: - Affects Version/s: 1.18.0 > Implement shared state file merging > --- > > Key: FLINK-32077 > URL: https://issues.apache.org/jira/browse/FLINK-32077 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32078) Implement private state file merging
[ https://issues.apache.org/jira/browse/FLINK-32078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32078: - Affects Version/s: 1.18.0 > Implement private state file merging > > > Key: FLINK-32078 > URL: https://issues.apache.org/jira/browse/FLINK-32078 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32076) Add file pool for concurrent file reusing
[ https://issues.apache.org/jira/browse/FLINK-32076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32076: - Affects Version/s: 1.18.0 > Add file pool for concurrent file reusing > - > > Key: FLINK-32076 > URL: https://issues.apache.org/jira/browse/FLINK-32076 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Rui Xia >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32076) Add file pool for concurrent file reusing
[ https://issues.apache.org/jira/browse/FLINK-32076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32076: - Component/s: Runtime / Checkpointing Runtime / State Backends > Add file pool for concurrent file reusing > - > > Key: FLINK-32076 > URL: https://issues.apache.org/jira/browse/FLINK-32076 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Rui Xia >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32074) Support file merging across checkpoints
[ https://issues.apache.org/jira/browse/FLINK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32074: - Component/s: Runtime / Checkpointing Runtime / State Backends > Support file merging across checkpoints > --- > > Key: FLINK-32074 > URL: https://issues.apache.org/jira/browse/FLINK-32074 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32075) Delete merged files on checkpoint abort
[ https://issues.apache.org/jira/browse/FLINK-32075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32075: - Component/s: Runtime / Checkpointing Runtime / State Backends > Delete merged files on checkpoint abort > --- > > Key: FLINK-32075 > URL: https://issues.apache.org/jira/browse/FLINK-32075 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32075) Delete merged files on checkpoint abort
[ https://issues.apache.org/jira/browse/FLINK-32075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32075: - Affects Version/s: 1.18.0 > Delete merged files on checkpoint abort > --- > > Key: FLINK-32075 > URL: https://issues.apache.org/jira/browse/FLINK-32075 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32074) Support file merging across checkpoints
[ https://issues.apache.org/jira/browse/FLINK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32074: - Affects Version/s: 1.18.0 > Support file merging across checkpoints > --- > > Key: FLINK-32074 > URL: https://issues.apache.org/jira/browse/FLINK-32074 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32073) Implement file merging in snapshot
[ https://issues.apache.org/jira/browse/FLINK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32073: - Component/s: Runtime / Checkpointing Runtime / State Backends > Implement file merging in snapshot > -- > > Key: FLINK-32073 > URL: https://issues.apache.org/jira/browse/FLINK-32073 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32073) Implement file merging in snapshot
[ https://issues.apache.org/jira/browse/FLINK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32073: - Affects Version/s: 1.18.0 > Implement file merging in snapshot > -- > > Key: FLINK-32073 > URL: https://issues.apache.org/jira/browse/FLINK-32073 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32073) Implement file merging in snapshot
[ https://issues.apache.org/jira/browse/FLINK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32073: - Fix Version/s: 1.18.0 > Implement file merging in snapshot > -- > > Key: FLINK-32073 > URL: https://issues.apache.org/jira/browse/FLINK-32073 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Han Yin >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32072) Create and wire FileMergingSnapshotManager with TaskManagerServices
[ https://issues.apache.org/jira/browse/FLINK-32072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32072: - Component/s: Runtime / Checkpointing Runtime / State Backends > Create and wire FileMergingSnapshotManager with TaskManagerServices > --- > > Key: FLINK-32072 > URL: https://issues.apache.org/jira/browse/FLINK-32072 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32072) Create and wire FileMergingSnapshotManager with TaskManagerServices
[ https://issues.apache.org/jira/browse/FLINK-32072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32072: - Affects Version/s: 1.18.0 > Create and wire FileMergingSnapshotManager with TaskManagerServices > --- > > Key: FLINK-32072 > URL: https://issues.apache.org/jira/browse/FLINK-32072 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32071) Implement the snapshot manager for merged checkpoint files in TM
[ https://issues.apache.org/jira/browse/FLINK-32071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32071: - Component/s: Runtime / Checkpointing Runtime / State Backends > Implement the snapshot manager for merged checkpoint files in TM > > > Key: FLINK-32071 > URL: https://issues.apache.org/jira/browse/FLINK-32071 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32071) Implement the snapshot manager for merged checkpoint files in TM
[ https://issues.apache.org/jira/browse/FLINK-32071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32071: - Affects Version/s: 1.18.0 > Implement the snapshot manager for merged checkpoint files in TM > > > Key: FLINK-32071 > URL: https://issues.apache.org/jira/browse/FLINK-32071 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32070) FLIP-306 Unified File Merging Mechanism for Checkpoints
[ https://issues.apache.org/jira/browse/FLINK-32070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-32070: - Component/s: Runtime / Checkpointing Runtime / State Backends Affects Version/s: 1.18.0 > FLIP-306 Unified File Merging Mechanism for Checkpoints > --- > > Key: FLINK-32070 > URL: https://issues.apache.org/jira/browse/FLINK-32070 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.18.0 > > > The FLIP: > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints] > > The creation of multiple checkpoint files can lead to a 'file flood' problem, > in which a large number of files are written to the checkpoint storage in a > short amount of time. This can cause issues in large clusters with high > workloads, such as the creation and deletion of many files increasing the > amount of file meta modification on DFS, leading to single-machine hotspot > issues for meta maintainers (e.g. NameNode in HDFS). Additionally, the > performance of object storage (e.g. Amazon S3 and Alibaba OSS) can > significantly decrease when listing objects, which is necessary for object > name de-duplication before creating an object, further affecting the > performance of directory manipulation in the file system's perspective of > view (See [hadoop-aws module > documentation|https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#:~:text=an%20intermediate%20state.-,Warning%20%232%3A%20Directories%20are%20mimicked,-The%20S3A%20clients], > section 'Warning #2: Directories are mimicked'). > While many solutions have been proposed for individual types of state files > (e.g. FLINK-11937 for keyed state (RocksDB) and FLINK-26803 for channel > state), the file flood problems from each type of checkpoint file are similar > and lack systematic view and solution. Therefore, the goal of this FLIP is to > establish a unified file merging mechanism to address the file flood problem > during checkpoint creation for all types of state files, including keyed, > non-keyed, channel, and changelog state. This will significantly improve the > system stability and availability of fault tolerance in Flink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30729) AggregateITCase generates stacktraces of IllegalStateExceptions in Maven output but doesn't fail
[
https://issues.apache.org/jira/browse/FLINK-30729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuan Mei resolved FLINK-30729.
--
Resolution: Fixed
> AggregateITCase generates stacktraces of IllegalStateExceptions in Maven
> output but doesn't fail
>
>
> Key: FLINK-30729
> URL: https://issues.apache.org/jira/browse/FLINK-30729
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends, Table SQL / API
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.17.1
>
>
> We're seeing some weird IllegalStateException stacktraces in CI for
> {{o.a.f.table.planner.runtime.stream.table.AggregateITCase}} which,
> interestingly, doesn't cause the test to fail. That's something we should
> investigate, I guess:
> {code}
> java.lang.IllegalStateException: LogWriter is closed
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> at
> org.apache.flink.runtime.state.changelog.inmemory.InMemoryStateChangelogWriter.appendMeta(InMemoryStateChangelogWriter.java:64)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.logMetaIfNeeded(AbstractStateChangeLogger.java:156)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.log(AbstractStateChangeLogger.java:140)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.valueCleared(AbstractStateChangeLogger.java:104)
> at
> org.apache.flink.state.changelog.ChangelogListState.clear(ChangelogListState.java:113)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.cleanupIfEmpty(TtlListState.java:193)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.hasNext(TtlListState.java:186)
> at
> org.apache.flink.table.planner.runtime.utils.JavaUserDefinedAggFunctions$DataViewTestAgg.getValue(JavaUserDefinedAggFunctions.java:355)
> at GroupAggsHandler$10675.getValue(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:146)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:43)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:236)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:835)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:784)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:945)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:924)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:738)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:556)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44987=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=13847
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30729) AggregateITCase generates stacktraces of IllegalStateExceptions in Maven output but doesn't fail
[
https://issues.apache.org/jira/browse/FLINK-30729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715819#comment-17715819
]
Yuan Mei commented on FLINK-30729:
--
merged commit
[{{4ca3604}}|https://github.com/apache/flink/commit/4ca3604a6afb7dace03464fafe7c5f14ab95fc5b]
into apache:master
> AggregateITCase generates stacktraces of IllegalStateExceptions in Maven
> output but doesn't fail
>
>
> Key: FLINK-30729
> URL: https://issues.apache.org/jira/browse/FLINK-30729
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends, Table SQL / API
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.17.1
>
>
> We're seeing some weird IllegalStateException stacktraces in CI for
> {{o.a.f.table.planner.runtime.stream.table.AggregateITCase}} which,
> interestingly, doesn't cause the test to fail. That's something we should
> investigate, I guess:
> {code}
> java.lang.IllegalStateException: LogWriter is closed
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> at
> org.apache.flink.runtime.state.changelog.inmemory.InMemoryStateChangelogWriter.appendMeta(InMemoryStateChangelogWriter.java:64)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.logMetaIfNeeded(AbstractStateChangeLogger.java:156)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.log(AbstractStateChangeLogger.java:140)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.valueCleared(AbstractStateChangeLogger.java:104)
> at
> org.apache.flink.state.changelog.ChangelogListState.clear(ChangelogListState.java:113)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.cleanupIfEmpty(TtlListState.java:193)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.hasNext(TtlListState.java:186)
> at
> org.apache.flink.table.planner.runtime.utils.JavaUserDefinedAggFunctions$DataViewTestAgg.getValue(JavaUserDefinedAggFunctions.java:355)
> at GroupAggsHandler$10675.getValue(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:146)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:43)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:236)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:835)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:784)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:945)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:924)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:738)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:556)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44987=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=13847
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30729) AggregateITCase generates stacktraces of IllegalStateExceptions in Maven output but doesn't fail
[
https://issues.apache.org/jira/browse/FLINK-30729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuan Mei updated FLINK-30729:
-
Fix Version/s: 1.17.1
> AggregateITCase generates stacktraces of IllegalStateExceptions in Maven
> output but doesn't fail
>
>
> Key: FLINK-30729
> URL: https://issues.apache.org/jira/browse/FLINK-30729
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends, Table SQL / API
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.17.1
>
>
> We're seeing some weird IllegalStateException stacktraces in CI for
> {{o.a.f.table.planner.runtime.stream.table.AggregateITCase}} which,
> interestingly, doesn't cause the test to fail. That's something we should
> investigate, I guess:
> {code}
> java.lang.IllegalStateException: LogWriter is closed
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> at
> org.apache.flink.runtime.state.changelog.inmemory.InMemoryStateChangelogWriter.appendMeta(InMemoryStateChangelogWriter.java:64)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.logMetaIfNeeded(AbstractStateChangeLogger.java:156)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.log(AbstractStateChangeLogger.java:140)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.valueCleared(AbstractStateChangeLogger.java:104)
> at
> org.apache.flink.state.changelog.ChangelogListState.clear(ChangelogListState.java:113)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.cleanupIfEmpty(TtlListState.java:193)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.hasNext(TtlListState.java:186)
> at
> org.apache.flink.table.planner.runtime.utils.JavaUserDefinedAggFunctions$DataViewTestAgg.getValue(JavaUserDefinedAggFunctions.java:355)
> at GroupAggsHandler$10675.getValue(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:146)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:43)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:236)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:835)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:784)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:945)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:924)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:738)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:556)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44987=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=13847
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30729) AggregateITCase generates stacktraces of IllegalStateExceptions in Maven output but doesn't fail
[
https://issues.apache.org/jira/browse/FLINK-30729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuan Mei updated FLINK-30729:
-
Issue Type: Improvement (was: Bug)
> AggregateITCase generates stacktraces of IllegalStateExceptions in Maven
> output but doesn't fail
>
>
> Key: FLINK-30729
> URL: https://issues.apache.org/jira/browse/FLINK-30729
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends, Table SQL / API
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.17.1
>
>
> We're seeing some weird IllegalStateException stacktraces in CI for
> {{o.a.f.table.planner.runtime.stream.table.AggregateITCase}} which,
> interestingly, doesn't cause the test to fail. That's something we should
> investigate, I guess:
> {code}
> java.lang.IllegalStateException: LogWriter is closed
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> at
> org.apache.flink.runtime.state.changelog.inmemory.InMemoryStateChangelogWriter.appendMeta(InMemoryStateChangelogWriter.java:64)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.logMetaIfNeeded(AbstractStateChangeLogger.java:156)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.log(AbstractStateChangeLogger.java:140)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.valueCleared(AbstractStateChangeLogger.java:104)
> at
> org.apache.flink.state.changelog.ChangelogListState.clear(ChangelogListState.java:113)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.cleanupIfEmpty(TtlListState.java:193)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.hasNext(TtlListState.java:186)
> at
> org.apache.flink.table.planner.runtime.utils.JavaUserDefinedAggFunctions$DataViewTestAgg.getValue(JavaUserDefinedAggFunctions.java:355)
> at GroupAggsHandler$10675.getValue(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:146)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:43)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:236)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:835)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:784)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:945)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:924)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:738)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:556)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44987=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=13847
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-30729) AggregateITCase generates stacktraces of IllegalStateExceptions in Maven output but doesn't fail
[
https://issues.apache.org/jira/browse/FLINK-30729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuan Mei reassigned FLINK-30729:
Assignee: Yanfei Lei
> AggregateITCase generates stacktraces of IllegalStateExceptions in Maven
> output but doesn't fail
>
>
> Key: FLINK-30729
> URL: https://issues.apache.org/jira/browse/FLINK-30729
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends, Table SQL / API
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
>
> We're seeing some weird IllegalStateException stacktraces in CI for
> {{o.a.f.table.planner.runtime.stream.table.AggregateITCase}} which,
> interestingly, doesn't cause the test to fail. That's something we should
> investigate, I guess:
> {code}
> java.lang.IllegalStateException: LogWriter is closed
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> at
> org.apache.flink.runtime.state.changelog.inmemory.InMemoryStateChangelogWriter.appendMeta(InMemoryStateChangelogWriter.java:64)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.logMetaIfNeeded(AbstractStateChangeLogger.java:156)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.log(AbstractStateChangeLogger.java:140)
> at
> org.apache.flink.state.changelog.AbstractStateChangeLogger.valueCleared(AbstractStateChangeLogger.java:104)
> at
> org.apache.flink.state.changelog.ChangelogListState.clear(ChangelogListState.java:113)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.cleanupIfEmpty(TtlListState.java:193)
> at
> org.apache.flink.runtime.state.ttl.TtlListState$IteratorWithCleanup.hasNext(TtlListState.java:186)
> at
> org.apache.flink.table.planner.runtime.utils.JavaUserDefinedAggFunctions$DataViewTestAgg.getValue(JavaUserDefinedAggFunctions.java:355)
> at GroupAggsHandler$10675.getValue(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:146)
> at
> org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:43)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:236)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:835)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:784)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:945)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:924)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:738)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:556)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44987=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=13847
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-23411) Expose Flink checkpoint details metrics
[ https://issues.apache.org/jira/browse/FLINK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-23411: - Fix Version/s: 1.18.0 > Expose Flink checkpoint details metrics > --- > > Key: FLINK-23411 > URL: https://issues.apache.org/jira/browse/FLINK-23411 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.13.1, 1.12.4 >Reporter: Jun Qin >Assignee: Hangxiang Yu >Priority: Major > Fix For: 1.18.0 > > > The checkpoint metrics as shown in the Flink Web UI like the > sync/async/alignment/start delay are not exposed to the metrics system. This > makes problem investigation harder when Web UI is not enabled: those numbers > can not get in the DEBUG logs. I think we should see how we can expose > metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30916) RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp failed with assertion
[
https://issues.apache.org/jira/browse/FLINK-30916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuan Mei resolved FLINK-30916.
--
Resolution: Duplicate
> RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp failed with assertion
>
>
> Key: FLINK-30916
> URL: https://issues.apache.org/jira/browse/FLINK-30916
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Rui Fan
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {{RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp}} failed due to an
> assertion:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45730=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=50bf7a25-bdc4-5e56-5478-c7b4511dde53=10328
> {code}
> Feb 06 02:54:01 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 0.427 s <<< FAILURE! - in
> org.apache.flink.contrib.streaming.state.RocksDBStateUploaderTest
> Feb 06 02:54:01 [ERROR]
> org.apache.flink.contrib.streaming.state.RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp
> Time elapsed: 0.115 s <<< FAILURE!
> Feb 06 02:54:01 java.lang.AssertionError:
> Feb 06 02:54:01
> Feb 06 02:54:01 Expecting empty but was:
> ["379065d4-dd29-4455-b30f-4dbc53336ea2"]
> Feb 06 02:54:01 at
> org.apache.flink.contrib.streaming.state.RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp(RocksDBStateUploaderTest.java:166)
> Feb 06 02:54:01 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30916) RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp failed with assertion
[
https://issues.apache.org/jira/browse/FLINK-30916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685033#comment-17685033
]
Yuan Mei commented on FLINK-30916:
--
Resolved in FLINK-30461
> RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp failed with assertion
>
>
> Key: FLINK-30916
> URL: https://issues.apache.org/jira/browse/FLINK-30916
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Rui Fan
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {{RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp}} failed due to an
> assertion:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45730=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=50bf7a25-bdc4-5e56-5478-c7b4511dde53=10328
> {code}
> Feb 06 02:54:01 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 0.427 s <<< FAILURE! - in
> org.apache.flink.contrib.streaming.state.RocksDBStateUploaderTest
> Feb 06 02:54:01 [ERROR]
> org.apache.flink.contrib.streaming.state.RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp
> Time elapsed: 0.115 s <<< FAILURE!
> Feb 06 02:54:01 java.lang.AssertionError:
> Feb 06 02:54:01
> Feb 06 02:54:01 Expecting empty but was:
> ["379065d4-dd29-4455-b30f-4dbc53336ea2"]
> Feb 06 02:54:01 at
> org.apache.flink.contrib.streaming.state.RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp(RocksDBStateUploaderTest.java:166)
> Feb 06 02:54:01 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
