[jira] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
[ https://issues.apache.org/jira/browse/FLINK-33463 ] RocMarshal deleted comment on FLINK-33463: was (Author: rocmarshal): The ticket is mainly to do the three items: 1. Support the implementation of dynamic source/factories tables based on the new source 2. Mark the old APIs about dynamic table source or factories as Deprecated. 3. Supplement the docs about the usage of stream semantic table or other extended feature if needed. > Support the implementation of dynamic source tables based on the new source > --- > > Key: FLINK-33463 > URL: https://issues.apache.org/jira/browse/FLINK-33463 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / JDBC >Reporter: RocMarshal >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35176) Support property authentication connection for JDBC catalog & dynamic table
[ https://issues.apache.org/jira/browse/FLINK-35176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-35176: --- Parent: FLINK-25420 Issue Type: Sub-task (was: Improvement) > Support property authentication connection for JDBC catalog & dynamic table > --- > > Key: FLINK-35176 > URL: https://issues.apache.org/jira/browse/FLINK-35176 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35176) Support property authentication connection for JDBC catalog & dynamic table
RocMarshal created FLINK-35176: -- Summary: Support property authentication connection for JDBC catalog & dynamic table Key: FLINK-35176 URL: https://issues.apache.org/jira/browse/FLINK-35176 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33460) Support property authentication connection.
[ https://issues.apache.org/jira/browse/FLINK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33460: --- Summary: Support property authentication connection. (was: Support more authentication connection types such as the secret.) > Support property authentication connection. > --- > > Key: FLINK-33460 > URL: https://issues.apache.org/jira/browse/FLINK-33460 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798856#comment-17798856 ] RocMarshal edited comment on FLINK-29050 at 4/11/24 10:56 AM: -- Based on [https://github.com/apache/flink/pull/20990#discussion_r1292783464] We'd like to do the following sub-tasks for the current jira. - Rename AbstractTestBase, JavaProgramTestBase MultipleProgramsTestBase to JUnit4, - Use jUnit5 to re-write the implementations for the above classes & tag JUnit4 classes as deprecated - Use junit5 implementation classes to migrate the Module: flink-hadoop-compatibility - Use junit5 implementation to make adaption for the sub-classes of JUnit4 (Maybe this part of the work needs to be recorded and promoted in other jiras) was (Author: rocmarshal): Based on [https://github.com/apache/flink/pull/20990#discussion_r1292783464] We'd like to do the following sub-tasks for the current jira. - Rename AbstractTestBase, JavaProgramTestBase MultipleProgramsTestBase to JUnit4, - Use jUnit5 to re-write the implementations for the above classes & tag JUnit4 classes as deprecated - Use junit5 implementation classes to migrate the Module: flink-hadoop-compatibility - Use junit5 implementation to make adaption for the sub-classes of JUnit4 > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available, stale-assigned, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33461) Support streaming related semantics for the new jdbc source
[ https://issues.apache.org/jira/browse/FLINK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33461: --- Summary: Support streaming related semantics for the new jdbc source (was: Support stream related semantics for the new jdbc source) > Support streaming related semantics for the new jdbc source > --- > > Key: FLINK-33461 > URL: https://issues.apache.org/jira/browse/FLINK-33461 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34564) Unstable test case TableSourceITCase#testTableHintWithLogicalTableScanReuse
[ https://issues.apache.org/jira/browse/FLINK-34564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822758#comment-17822758 ] RocMarshal commented on FLINK-34564: Thx a lot for [~lincoln.86xy] your comments, I'll check the details via the pr mentioned above~ :) > Unstable test case TableSourceITCase#testTableHintWithLogicalTableScanReuse > --- > > Key: FLINK-34564 > URL: https://issues.apache.org/jira/browse/FLINK-34564 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 2.0.0, 1.19.0 >Reporter: RocMarshal >Priority: Minor > Attachments: image-2024-03-02-11-01-12-718.png, > image-2024-03-02-11-01-44-431.png > > > * branch 1.19 & master > * java version 1.8 > * how to re-produce > ** Add '@RepeatedTest' for > TableSourceITCase#testTableHintWithLogicalTableScanReuse > ** then run it > ** !image-2024-03-02-11-01-12-718.png! > ** !image-2024-03-02-11-01-44-431.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34564) Unstable test case TableSourceITCase#testTableHintWithLogicalTableScanReuse
[ https://issues.apache.org/jira/browse/FLINK-34564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822744#comment-17822744 ] RocMarshal commented on FLINK-34564: I discovered this issue while assisting [~fanrui] in fixing a bug in try https://github.com/apache/flink/pull/24407#issuecomment-1970101117. https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57951=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=ae4f8708-9994-57d3-c2d7-b892156e7812 > Unstable test case TableSourceITCase#testTableHintWithLogicalTableScanReuse > --- > > Key: FLINK-34564 > URL: https://issues.apache.org/jira/browse/FLINK-34564 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 2.0.0, 1.19.0 >Reporter: RocMarshal >Priority: Minor > Attachments: image-2024-03-02-11-01-12-718.png, > image-2024-03-02-11-01-44-431.png > > > * branch 1.19 & master > * java version 1.8 > * how to re-produce > ** Add '@RepeatedTest' for > TableSourceITCase#testTableHintWithLogicalTableScanReuse > ** then run it > ** !image-2024-03-02-11-01-12-718.png! > ** !image-2024-03-02-11-01-44-431.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34564) Unstable test case TableSourceITCase#testTableHintWithLogicalTableScanReuse
RocMarshal created FLINK-34564: -- Summary: Unstable test case TableSourceITCase#testTableHintWithLogicalTableScanReuse Key: FLINK-34564 URL: https://issues.apache.org/jira/browse/FLINK-34564 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 2.0.0, 1.19.0 Reporter: RocMarshal Attachments: image-2024-03-02-11-01-12-718.png, image-2024-03-02-11-01-44-431.png * branch 1.19 & master * java version 1.8 * how to re-produce ** Add '@RepeatedTest' for TableSourceITCase#testTableHintWithLogicalTableScanReuse ** then run it ** !image-2024-03-02-11-01-12-718.png! ** !image-2024-03-02-11-01-44-431.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34249) Remove DefaultSlotTracker related logic.
[ https://issues.apache.org/jira/browse/FLINK-34249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-34249: --- Description: pre step: https://issues.apache.org/jira/browse/FLINK-34174 The main reason for initiating this ticket is https://issues.apache.org/jira/browse/FLINK-31449 & https://issues.apache.org/jira/browse/FLINK-34174 (IIUC) as the current related logic is no longer being used. > Remove DefaultSlotTracker related logic. > > > Key: FLINK-34249 > URL: https://issues.apache.org/jira/browse/FLINK-34249 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Task >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > Labels: pull-request-available > Fix For: 1.19.0 > > > pre step: https://issues.apache.org/jira/browse/FLINK-34174 > The main reason for initiating this ticket is > https://issues.apache.org/jira/browse/FLINK-31449 & > https://issues.apache.org/jira/browse/FLINK-34174 > (IIUC) as the current related logic is no longer being used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34249) Remove DefaultSlotTracker related logic.
[ https://issues.apache.org/jira/browse/FLINK-34249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811463#comment-17811463 ] RocMarshal commented on FLINK-34249: pre step: https://issues.apache.org/jira/browse/FLINK-34174 The main reason for initiating this ticket is https://issues.apache.org/jira/browse/FLINK-31449 & https://issues.apache.org/jira/browse/FLINK-34174 (IIUC) as the current related logic is no longer being used. > Remove DefaultSlotTracker related logic. > > > Key: FLINK-34249 > URL: https://issues.apache.org/jira/browse/FLINK-34249 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Task >Reporter: RocMarshal >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34249) Remove DefaultSlotTracker related logic.
RocMarshal created FLINK-34249: -- Summary: Remove DefaultSlotTracker related logic. Key: FLINK-34249 URL: https://issues.apache.org/jira/browse/FLINK-34249 Project: Flink Issue Type: Technical Debt Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34174) Remove SlotMatchingStrategy related logic
[ https://issues.apache.org/jira/browse/FLINK-34174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809232#comment-17809232 ] RocMarshal commented on FLINK-34174: The main reason for initiating this ticket is https://issues.apache.org/jira/browse/FLINK-31449 (IIUC) as the current related logic is no longer being used. > Remove SlotMatchingStrategy related logic > - > > Key: FLINK-34174 > URL: https://issues.apache.org/jira/browse/FLINK-34174 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Task >Reporter: RocMarshal >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34174) Remove SlotMatchingStrategy related logic
RocMarshal created FLINK-34174: -- Summary: Remove SlotMatchingStrategy related logic Key: FLINK-34174 URL: https://issues.apache.org/jira/browse/FLINK-34174 Project: Flink Issue Type: Technical Debt Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34102) Invalid configuration when using 'env.log.max' on yarn application mode
RocMarshal created FLINK-34102: -- Summary: Invalid configuration when using 'env.log.max' on yarn application mode Key: FLINK-34102 URL: https://issues.apache.org/jira/browse/FLINK-34102 Project: Flink Issue Type: Bug Components: Runtime / Configuration Affects Versions: 1.19.0 Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33988) Invalid configuration when using initialized root logger level on yarn application mode
[ https://issues.apache.org/jira/browse/FLINK-33988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33988: --- Description: relevant https://issues.apache.org/jira/browse/FLINK-33166 When I set env. log. level=DEBUG and start the flink job by yarn application mode, the logs of TM and JM are still INFO. Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link is not complete enough. So I used the following configuration: containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG When starting the job by yarn application mode, TM and JM can output DEBUG level logs. Repair ideas: Fill the value of *env. log. level* into the Flink configuration by *containerized. xxx. env. ROOT_ LOG_ LEVEL* before obtaining the environment variable for the container was: from https://issues.apache.org/jira/browse/FLINK-33166 When I set env. log. level=DEBUG and start the flink job by yarn application mode, the logs of TM and JM are still INFO. Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link is not complete enough. So I used the following configuration: containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG When starting the job by yarn application mode, TM and JM can output DEBUG level logs. Repair ideas: Fill the value of *env. log. level* into the Flink configuration by *containerized. xxx. env. ROOT_ LOG_ LEVEL* before obtaining the environment variable for the container > Invalid configuration when using initialized root logger level on yarn > application mode > --- > > Key: FLINK-33988 > URL: https://issues.apache.org/jira/browse/FLINK-33988 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.19.0 >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Fix For: 1.19.0 > > > relevant https://issues.apache.org/jira/browse/FLINK-33166 > When I set env. log. level=DEBUG and start the flink job by yarn application > mode, the logs of TM and JM are still INFO. > Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link > is not complete enough. > So I used the following configuration: > containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG > containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG > > When starting the job by yarn application mode, TM and JM can output DEBUG > level logs. > > Repair ideas: > Fill the value of *env. log. level* into the Flink configuration by > *containerized. xxx. env. ROOT_ LOG_ LEVEL* before obtaining the environment > variable for the container -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33988) Invalid configuration when using initialized root logger level on yarn application mode
[ https://issues.apache.org/jira/browse/FLINK-33988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33988: --- Description: from https://issues.apache.org/jira/browse/FLINK-33166 When I set env. log. level=DEBUG and start the flink job by yarn application mode, the logs of TM and JM are still INFO. Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link is not complete enough. So I used the following configuration: containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG When starting the job by yarn application mode, TM and JM can output DEBUG level logs. Repair ideas: Fill the value of *env. log. level* into the Flink configuration by *containerized. xxx. env. ROOT_ LOG_ LEVEL* before obtaining the environment variable for the container was: from https://issues.apache.org/jira/browse/FLINK-33166 When I set env. log. level=DEBUG and start the flink job by yarn application mode, the logs of TM and JM are still INFO. Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link is not complete enough. So I used the following configuration: containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG When starting the job by yarn application mode, TM and JM can output DEBUG level logs. Repair ideas: Fill the value of *env. log. level* into the Flink configuration before obtaining the environment variable for the container, with the new key being *ROOT_ LOG_ LEVEL* > Invalid configuration when using initialized root logger level on yarn > application mode > --- > > Key: FLINK-33988 > URL: https://issues.apache.org/jira/browse/FLINK-33988 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.18.0, 1.17.2 >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > > from https://issues.apache.org/jira/browse/FLINK-33166 > When I set env. log. level=DEBUG and start the flink job by yarn application > mode, the logs of TM and JM are still INFO. > Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link > is not complete enough. > So I used the following configuration: > containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG > containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG > > When starting the job by yarn application mode, TM and JM can output DEBUG > level logs. > > Repair ideas: > Fill the value of *env. log. level* into the Flink configuration by > *containerized. xxx. env. ROOT_ LOG_ LEVEL* before obtaining the environment > variable for the container -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33988) Invalid configuration when using initialized root logger level on yarn application mode
[ https://issues.apache.org/jira/browse/FLINK-33988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33988: --- Description: from https://issues.apache.org/jira/browse/FLINK-33166 When I set env. log. level=DEBUG and start the flink job by yarn application mode, the logs of TM and JM are still INFO. Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link is not complete enough. So I used the following configuration: containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG When starting the job by yarn application mode, TM and JM can output DEBUG level logs. Repair ideas: Fill the value of *env. log. level* into the Flink configuration before obtaining the environment variable for the container, with the new key being *ROOT_ LOG_ LEVEL* was:from https://issues.apache.org/jira/browse/FLINK-33166 > Invalid configuration when using initialized root logger level on yarn > application mode > --- > > Key: FLINK-33988 > URL: https://issues.apache.org/jira/browse/FLINK-33988 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.18.0, 1.17.2 >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Minor > > from https://issues.apache.org/jira/browse/FLINK-33166 > When I set env. log. level=DEBUG and start the flink job by yarn application > mode, the logs of TM and JM are still INFO. > Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link > is not complete enough. > So I used the following configuration: > containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG > containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG > > When starting the job by yarn application mode, TM and JM can output DEBUG > level logs. > > Repair ideas: > Fill the value of *env. log. level* into the Flink configuration before > obtaining the environment variable for the container, with the new key being > *ROOT_ LOG_ LEVEL* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33988) Invalid configuration when using initialized root logger level on yarn application mode
[ https://issues.apache.org/jira/browse/FLINK-33988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33988: --- Description: from https://issues.apache.org/jira/browse/FLINK-33166 > Invalid configuration when using initialized root logger level on yarn > application mode > --- > > Key: FLINK-33988 > URL: https://issues.apache.org/jira/browse/FLINK-33988 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.18.0, 1.17.2 >Reporter: RocMarshal >Priority: Minor > > from https://issues.apache.org/jira/browse/FLINK-33166 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33988) Invalid configuration when using initialized root logger level on yarn application mode
RocMarshal created FLINK-33988: -- Summary: Invalid configuration when using initialized root logger level on yarn application mode Key: FLINK-33988 URL: https://issues.apache.org/jira/browse/FLINK-33988 Project: Flink Issue Type: Bug Components: Runtime / Configuration Affects Versions: 1.17.2, 1.18.0 Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33959) Unstable test case ChangelogRecoveryITCase.testMaterialization on 1.17 release branch
[
https://issues.apache.org/jira/browse/FLINK-33959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RocMarshal updated FLINK-33959:
---
Summary: Unstable test case ChangelogRecoveryITCase.testMaterialization on
1.17 release branch (was: Unstable test case )
> Unstable test case ChangelogRecoveryITCase.testMaterialization on 1.17
> release branch
> -
>
> Key: FLINK-33959
> URL: https://issues.apache.org/jira/browse/FLINK-33959
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.17.0, 1.17.1, 1.17.2
>Reporter: RocMarshal
>Priority: Blocker
> Attachments: image-2023-12-29-19-51-35-996.png
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7]
>
> Dec 29 07:35:05 [ERROR] ChangelogRecoveryITCase.testMaterialization
> Dec 29 07:35:05 [INFO] Run 1: PASS
> Dec 29 07:35:05 [ERROR] Run 2: org.apache.flink.runtime.JobException:
> Recovery is suppressed by
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2,
> backoffTimeMS=0)
>
> Dec 29 07:35:05 [INFO] Run 3: PASS
>
> {code:java}
> Dec 29 07:31:37 Caused by: java.io.FileNotFoundException:
> /tmp/junit666440189967214661/junit7847292894745938127/7c8c2cdf87500be80d28b1001902edcc/dstl/3cc55f1f-bd4f-4bc6-a41d-e8d91b54b4b0
> (No such file or directory)
> Dec 29 07:31:37 at java.io.FileInputStream.open0(Native Method)
> Dec 29 07:31:37 at
> java.io.FileInputStream.open(FileInputStream.java:195)
> Dec 29 07:31:37 at
> java.io.FileInputStream.(FileInputStream.java:138)
> Dec 29 07:31:37 at
> org.apache.flink.core.fs.local.LocalDataInputStream.(LocalDataInputStream.java:50)
> Dec 29 07:31:37 at
> org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:134)
> Dec 29 07:31:37 at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:87)
> Dec 29 07:31:37 at
> org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
> Dec 29 07:31:37 at
> org.apache.flink.changelog.fs.ChangelogStreamHandleReaderWithCache.openAndSeek(ChangelogStreamHandleReaderWithCache.java:89)
> Dec 29 07:31:37 at
> org.apache.flink.changelog.fs.StateChangeIteratorImpl.read(StateChangeIteratorImpl.java:42)
> Dec 29 07:31:37 at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:85)
> Dec 29 07:31:37 ... 21 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33959) Unstable test case
[ https://issues.apache.org/jira/browse/FLINK-33959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801149#comment-17801149 ] RocMarshal commented on FLINK-33959: Hi, could someone help to check it ? Thank you very much :) > Unstable test case > --- > > Key: FLINK-33959 > URL: https://issues.apache.org/jira/browse/FLINK-33959 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0, 1.17.1, 1.17.2 >Reporter: RocMarshal >Priority: Blocker > Attachments: image-2023-12-29-19-51-35-996.png > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] > > > !image-2023-12-29-19-51-35-996.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33959) Unstable test case
[ https://issues.apache.org/jira/browse/FLINK-33959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33959: --- Attachment: image-2023-12-29-19-51-35-996.png > Unstable test case > --- > > Key: FLINK-33959 > URL: https://issues.apache.org/jira/browse/FLINK-33959 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0, 1.17.1, 1.17.2 >Reporter: RocMarshal >Priority: Blocker > Attachments: image-2023-12-29-19-51-35-996.png > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] > > > > > Dec 29 07:31:37 at java.io.FileInputStream.open(FileInputStream.java:195) > Dec 29 07:31:37 at java.io.FileInputStream.(FileInputStream.java:138) > Dec 29 07:31:37 at > org.apache.flink.core.fs.local.LocalDataInputStream.(LocalDataInputStream.java:50) > > Dec 29 07:31:37 at > org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:134) > Dec 29 07:31:37 at > org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:87) > > Dec 29 07:31:37 at > org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) > > Dec 29 07:31:37 at > org.apache.flink.changelog.fs.ChangelogStreamHandleReaderWithCache.openAndSeek(ChangelogStreamHandleReaderWithCache.java:89) > > Dec 29 07:31:37 at > org.apache.flink.changelog.fs.StateChangeIteratorImpl.read(StateChangeIteratorImpl.java:42) > > Dec 29 07:31:37 at > org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:85) > > Dec 29 07:31:37 ... 21 more > Dec 29 07:31:37 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33959) Unstable test case
[ https://issues.apache.org/jira/browse/FLINK-33959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33959: --- Description: [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] !image-2023-12-29-19-51-35-996.png! was: [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] Dec 29 07:31:37 at java.io.FileInputStream.open(FileInputStream.java:195) Dec 29 07:31:37 at java.io.FileInputStream.(FileInputStream.java:138) Dec 29 07:31:37 at org.apache.flink.core.fs.local.LocalDataInputStream.(LocalDataInputStream.java:50) Dec 29 07:31:37 at org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:134) Dec 29 07:31:37 at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:87) Dec 29 07:31:37 at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) Dec 29 07:31:37 at org.apache.flink.changelog.fs.ChangelogStreamHandleReaderWithCache.openAndSeek(ChangelogStreamHandleReaderWithCache.java:89) Dec 29 07:31:37 at org.apache.flink.changelog.fs.StateChangeIteratorImpl.read(StateChangeIteratorImpl.java:42) Dec 29 07:31:37 at org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:85) Dec 29 07:31:37 ... 21 more Dec 29 07:31:37 > Unstable test case > --- > > Key: FLINK-33959 > URL: https://issues.apache.org/jira/browse/FLINK-33959 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0, 1.17.1, 1.17.2 >Reporter: RocMarshal >Priority: Blocker > Attachments: image-2023-12-29-19-51-35-996.png > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] > > > !image-2023-12-29-19-51-35-996.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33959) Unstable test case
RocMarshal created FLINK-33959: -- Summary: Unstable test case Key: FLINK-33959 URL: https://issues.apache.org/jira/browse/FLINK-33959 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.17.2, 1.17.1, 1.17.0 Reporter: RocMarshal [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33959) Unstable test case
[ https://issues.apache.org/jira/browse/FLINK-33959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33959: --- Description: [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] Dec 29 07:31:37 at java.io.FileInputStream.open(FileInputStream.java:195) Dec 29 07:31:37 at java.io.FileInputStream.(FileInputStream.java:138) Dec 29 07:31:37 at org.apache.flink.core.fs.local.LocalDataInputStream.(LocalDataInputStream.java:50) Dec 29 07:31:37 at org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:134) Dec 29 07:31:37 at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:87) Dec 29 07:31:37 at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) Dec 29 07:31:37 at org.apache.flink.changelog.fs.ChangelogStreamHandleReaderWithCache.openAndSeek(ChangelogStreamHandleReaderWithCache.java:89) Dec 29 07:31:37 at org.apache.flink.changelog.fs.StateChangeIteratorImpl.read(StateChangeIteratorImpl.java:42) Dec 29 07:31:37 at org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:85) Dec 29 07:31:37 ... 21 more Dec 29 07:31:37 was: [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] > Unstable test case > --- > > Key: FLINK-33959 > URL: https://issues.apache.org/jira/browse/FLINK-33959 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0, 1.17.1, 1.17.2 >Reporter: RocMarshal >Priority: Blocker > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55940=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7] > > > > > Dec 29 07:31:37 at java.io.FileInputStream.open(FileInputStream.java:195) > Dec 29 07:31:37 at java.io.FileInputStream.(FileInputStream.java:138) > Dec 29 07:31:37 at > org.apache.flink.core.fs.local.LocalDataInputStream.(LocalDataInputStream.java:50) > > Dec 29 07:31:37 at > org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:134) > Dec 29 07:31:37 at > org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:87) > > Dec 29 07:31:37 at > org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) > > Dec 29 07:31:37 at > org.apache.flink.changelog.fs.ChangelogStreamHandleReaderWithCache.openAndSeek(ChangelogStreamHandleReaderWithCache.java:89) > > Dec 29 07:31:37 at > org.apache.flink.changelog.fs.StateChangeIteratorImpl.read(StateChangeIteratorImpl.java:42) > > Dec 29 07:31:37 at > org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:85) > > Dec 29 07:31:37 ... 21 more > Dec 29 07:31:37 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33947) Fix bugs about prefix in DelegatingConfiguration
[ https://issues.apache.org/jira/browse/FLINK-33947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33947: --- Summary: Fix bugs about prefix in DelegatingConfiguration (was: Fix bugs in DelegatingConfiguration missed the prefix mapping ) > Fix bugs about prefix in DelegatingConfiguration > - > > Key: FLINK-33947 > URL: https://issues.apache.org/jira/browse/FLINK-33947 > Project: Flink > Issue Type: Bug > Components: API / Core >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Fix For: 1.19.0, 1.17.3, 1.18.2 > > > It was resulted from > [https://github.com/apache/flink/pull/23994#issuecomment-1869905090] > - Check and confirm other potential bug points > - Fix the bugs about prefix key mapping when operating. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33947) Fix bugs in DelegatingConfiguration missed the prefix mapping
[ https://issues.apache.org/jira/browse/FLINK-33947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800805#comment-17800805 ] RocMarshal commented on FLINK-33947: > And how about allow prefix is empty? SGTM +1. > Fix bugs in DelegatingConfiguration missed the prefix mapping > -- > > Key: FLINK-33947 > URL: https://issues.apache.org/jira/browse/FLINK-33947 > Project: Flink > Issue Type: Bug > Components: API / Core >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Fix For: 1.19.0, 1.17.3, 1.18.2 > > > It was resulted from > [https://github.com/apache/flink/pull/23994#issuecomment-1869905090] > - Check and confirm other potential bug points > - Fix the bugs about prefix key mapping when operating. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33947) Fix bugs in DelegatingConfiguration missed the prefix mapping
[ https://issues.apache.org/jira/browse/FLINK-33947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800794#comment-17800794 ] RocMarshal commented on FLINK-33947: hi, [~fanrui] thanks for your attention. I want to set the prefix attribute as Nonnull (@Nonnull) There are two benefits to this: - Ensure that potential _*NPE*_ risks can be addressed - Maintain minimal changes (in my limited read, since the initial design allowed prefixes to default to empty strings, I tend to maintain this default strategy to ensure some backward compatibility space is left. Of course, if it is necessary not to allow prefixes to be empty strings, I am happy to make corresponding changes) Please let me know what's your opinion~ > Fix bugs in DelegatingConfiguration missed the prefix mapping > -- > > Key: FLINK-33947 > URL: https://issues.apache.org/jira/browse/FLINK-33947 > Project: Flink > Issue Type: Bug > Components: API / Core >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Fix For: 1.19.0, 1.17.3, 1.18.2 > > > It was resulted from > [https://github.com/apache/flink/pull/23994#issuecomment-1869905090] > - Check and confirm other potential bug points > - Fix the bugs about prefix key mapping when operating. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33947) Fix bugs in DelegatingConfiguration missed the prefix mapping
[
https://issues.apache.org/jira/browse/FLINK-33947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800774#comment-17800774
]
RocMarshal edited comment on FLINK-33947 at 12/27/23 2:09 PM:
--
Before officially starting the repair, there are two issues worth discussing:
- First:
Is the *_prefix_* attribute allowed to be {*}_null_{*}?
What are the risks in using *_prefix_* if it is not allowed to be {*}_null_{*}?
Or will there be any defects?
>From the current effect, if the *_prefix_* is {*}_null_{*}, methods such as
>*_hashCode()_* will result in *_NPE_*
Based on the feedback from the current community regarding the use of this
class, there should be no instances of using a *_null_* {_}*prefix*{_}, unless
there are no calls to methods that may result in _*NPE*_
There are two corresponding alternative solutions here:
1. It is not allowed for _*prefix*_ to be a *_null_* object, so there will be
no corresponding method to cause _*NPE*_
2. Alternatively, allow the _*prefix*_ to be a _*null*_ object, but we need to
design logic to handle _*null*_ situations for methods that may cause _*NPE*_
to avoid the occurrence of _*NPE*_
- Secondly:
Regarding the mapping bug of _*prefix*_ values in various methods,
Only exists in the _*removeKey* *removeConfiguration*_ methods. We just need to
fix and add tests
was (Author: rocmarshal):
Before officially starting the repair, there are two issues worth discussing:
- First:
Is the *_prefix_* attribute allowed to be {*}_null_{*}?
What are the risks in using *_prefix_* if it is not allowed to be {*}_null_{*}?
Or will there be any defects?
>From the current effect, if the *_prefix_* is {*}_null_{*}, methods such as
>*_hashCode()_* will result in *_NPE_*
Based on the feedback from the current community regarding the use of this
class, there should be no instances of using a *_null_* {_}*prefix*{_}, unless
there are no calls to methods that may result in _*NPE*_
There are two corresponding alternative solutions here:
1. It is not allowed for _*prefix*_ to be a *_null_* object, so there will be
no corresponding method to cause _*NPE*_
2. Alternatively, allow the _*prefix*_ to be a _*null*_ object, but we need to
design logic to handle _*null*_ situations for methods that may cause _*NPE*_
to avoid the occurrence of _*NPE*_
By the way, it's best to modify the _*prefix*_ as _*final*_
- Secondly:
Regarding the mapping bug of _*prefix*_ values in various methods,
Only exists in the _*removeKey* *removeConfiguration*_ methods. We just need to
fix and add tests
> Fix bugs in DelegatingConfiguration missed the prefix mapping
> --
>
> Key: FLINK-33947
> URL: https://issues.apache.org/jira/browse/FLINK-33947
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Fix For: 1.19.0, 1.17.3, 1.18.2
>
>
> It was resulted from
> [https://github.com/apache/flink/pull/23994#issuecomment-1869905090]
> - Check and confirm other potential bug points
> - Fix the bugs about prefix key mapping when operating.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33947) Fix bugs in DelegatingConfiguration missed the prefix mapping
[
https://issues.apache.org/jira/browse/FLINK-33947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800774#comment-17800774
]
RocMarshal commented on FLINK-33947:
Before officially starting the repair, there are two issues worth discussing:
- First:
Is the *_prefix_* attribute allowed to be {*}_null_{*}?
What are the risks in using *_prefix_* if it is not allowed to be {*}_null_{*}?
Or will there be any defects?
>From the current effect, if the *_prefix_* is {*}_null_{*}, methods such as
>*_hashCode()_* will result in *_NPE_*
Based on the feedback from the current community regarding the use of this
class, there should be no instances of using a *_null_* {_}*prefix*{_}, unless
there are no calls to methods that may result in _*NPE*_
There are two corresponding alternative solutions here:
1. It is not allowed for _*prefix*_ to be a *_null_* object, so there will be
no corresponding method to cause _*NPE*_
2. Alternatively, allow the _*prefix*_ to be a _*null*_ object, but we need to
design logic to handle _*null*_ situations for methods that may cause _*NPE*_
to avoid the occurrence of _*NPE*_
By the way, it's best to modify the _*prefix*_ as _*final*_
- Secondly:
Regarding the mapping bug of _*prefix*_ values in various methods,
Only exists in the _*removeKey* *removeConfiguration*_ methods. We just need to
fix and add tests
> Fix bugs in DelegatingConfiguration missed the prefix mapping
> --
>
> Key: FLINK-33947
> URL: https://issues.apache.org/jira/browse/FLINK-33947
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Fix For: 1.19.0, 1.17.3, 1.18.2
>
>
> It was resulted from
> [https://github.com/apache/flink/pull/23994#issuecomment-1869905090]
> - Check and confirm other potential bug points
> - Fix the bugs about prefix key mapping when operating.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33947) Fix bugs in DelegatingConfiguration missed the prefix mapping
RocMarshal created FLINK-33947: -- Summary: Fix bugs in DelegatingConfiguration missed the prefix mapping Key: FLINK-33947 URL: https://issues.apache.org/jira/browse/FLINK-33947 Project: Flink Issue Type: Bug Components: API / Core Reporter: RocMarshal It was resulted from [https://github.com/apache/flink/pull/23994#issuecomment-1869905090] - Check and confirm other potential bug points - Fix the bugs about prefix key mapping when operating. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798856#comment-17798856 ] RocMarshal edited comment on FLINK-29050 at 12/20/23 8:08 AM: -- Based on [https://github.com/apache/flink/pull/20990#discussion_r1292783464] We'd like to do the following sub-tasks for the current jira. - Rename AbstractTestBase, JavaProgramTestBase MultipleProgramsTestBase to JUnit4, - Use jUnit5 to re-write the implementations for the above classes & tag JUnit4 classes as deprecated - Use junit5 implementation classes to migrate the Module: flink-hadoop-compatibility - Use junit5 implementation to make adaption for the sub-classes of JUnit4 was (Author: rocmarshal): Based on [https://github.com/apache/flink/pull/20990#discussion_r1292783464] We'd like to do the following sub-tasks for the current jira. - Rename AbstractTestBase, JavaProgramTestBase MultipleProgramsTestBase to JUnit4, Use jUnit5 to re-write the implementations for the above classes & tag JUnit4 classes as deprecated - Use junit5 implementation classes to migrate the Module: flink-hadoop-compatibility - Use junit5 implementation to make adaption for the sub-classes of JUnit4 > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available, stale-assigned, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798856#comment-17798856 ] RocMarshal edited comment on FLINK-29050 at 12/20/23 8:07 AM: -- Based on [https://github.com/apache/flink/pull/20990#discussion_r1292783464] We'd like to do the following sub-tasks for the current jira. - Rename AbstractTestBase, JavaProgramTestBase MultipleProgramsTestBase to JUnit4, Use jUnit5 to re-write the implementations for the above classes & tag JUnit4 classes as deprecated - Use junit5 implementation classes to migrate the Module: flink-hadoop-compatibility - Use junit5 implementation to make adaption for the sub-classes of JUnit4 was (Author: rocmarshal): Based on [https://github.com/apache/flink/pull/20990#discussion_r1292783464] We'd like to do the following sub-tasks for the current jira. - Rename AbstractTestBase, JavaProgramTestBase MultipleProgramsTestBase to JUnit4 - Use jUnit5 to re-write the implementations for the above classes & tag JUnit4 classes as deprecated - Use junit5 implementation classes to migrate the Module: flink-hadoop-compatibility - Use junit5 implementation to make adaption for the sub-classes of JUnit4 > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available, stale-assigned, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798856#comment-17798856 ] RocMarshal commented on FLINK-29050: Based on [https://github.com/apache/flink/pull/20990#discussion_r1292783464] We'd like to do the following sub-tasks for the current jira. - Rename AbstractTestBase, JavaProgramTestBase MultipleProgramsTestBase to JUnit4 - Use jUnit5 to re-write the implementations for the above classes & tag JUnit4 classes as deprecated - Use junit5 implementation classes to migrate the Module: flink-hadoop-compatibility - Use junit5 implementation to make adaption for the sub-classes of JUnit4 > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available, stale-assigned, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33875) Support slots wait mechanism at DeclarativeSlotPoolBridge side for Default Scheduler
RocMarshal created FLINK-33875: -- Summary: Support slots wait mechanism at DeclarativeSlotPoolBridge side for Default Scheduler Key: FLINK-33875 URL: https://issues.apache.org/jira/browse/FLINK-33875 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33874) Support resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33874: --- Summary: Support resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler (was: Introduce resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler) > Support resource request wait mechanism at DefaultDeclarativeSlotPool side > for Default Scheduler > > > Key: FLINK-33874 > URL: https://issues.apache.org/jira/browse/FLINK-33874 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33874) Introduce resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33874: --- Parent: FLINK-31757 Issue Type: Sub-task (was: New Feature) > Introduce resource request wait mechanism at DefaultDeclarativeSlotPool side > for Default Scheduler > -- > > Key: FLINK-33874 > URL: https://issues.apache.org/jira/browse/FLINK-33874 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33874) Introduce resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler
RocMarshal created FLINK-33874: -- Summary: Introduce resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler Key: FLINK-33874 URL: https://issues.apache.org/jira/browse/FLINK-33874 Project: Flink Issue Type: New Feature Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33853) [JUnit5 Migration] Migrate Junit5 for DeclarativeSlotPoolBridge test classes of runtime module
[ https://issues.apache.org/jira/browse/FLINK-33853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33853: --- Summary: [JUnit5 Migration] Migrate Junit5 for DeclarativeSlotPoolBridge test classes of runtime module (was: [JUnit5 Migration] Migrate Junit5 for DefaultDeclarativeSlotPool test classes of runtime module) > [JUnit5 Migration] Migrate Junit5 for DeclarativeSlotPoolBridge test classes > of runtime module > -- > > Key: FLINK-33853 > URL: https://issues.apache.org/jira/browse/FLINK-33853 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: RocMarshal >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33853) Migrate Junit5 for DefaultDeclarativeSlotPool test classes
RocMarshal created FLINK-33853: -- Summary: Migrate Junit5 for DefaultDeclarativeSlotPool test classes Key: FLINK-33853 URL: https://issues.apache.org/jira/browse/FLINK-33853 Project: Flink Issue Type: Improvement Components: Tests Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33853) [JUnit5 Migration] Migrate Junit5 for DefaultDeclarativeSlotPool test classes of runtime module
[ https://issues.apache.org/jira/browse/FLINK-33853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33853: --- Summary: [JUnit5 Migration] Migrate Junit5 for DefaultDeclarativeSlotPool test classes of runtime module (was: Migrate Junit5 for DefaultDeclarativeSlotPool test classes) > [JUnit5 Migration] Migrate Junit5 for DefaultDeclarativeSlotPool test classes > of runtime module > --- > > Key: FLINK-33853 > URL: https://issues.apache.org/jira/browse/FLINK-33853 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: RocMarshal >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33653) Introduce a benchmark for balanced tasks scheduling
RocMarshal created FLINK-33653: -- Summary: Introduce a benchmark for balanced tasks scheduling Key: FLINK-33653 URL: https://issues.apache.org/jira/browse/FLINK-33653 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-25420) Port JDBC Source to new Source API (FLIP-27)
[ https://issues.apache.org/jira/browse/FLINK-25420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782972#comment-17782972 ] RocMarshal edited comment on FLINK-25420 at 11/5/23 10:09 AM: -- Hi, [~martijnvisser] [~eskabetxe] I have divided the work that needs to be done for the current parent ticket based on FLIP as follows. Could you please help take a look? In the current ticket order with my limited read: - Ticket 2 and ticket 3 can be independently advanced. Because both ticket 2 and ticket 3 are partially dependent on ticket 1 - Ticket 4 and ticket 5 are partially dependent on ticket 3, and ticket 5 and ticket 4 could be completed independently, because ticket 5 is transparent to the user. Looking forward to your opinion about the task-splitting or discussion about any sub-ticket~ Thank you~ CC [~jingge] was (Author: rocmarshal): Hi, [~martijnvisser] [~eskabetxe] I have divided the work that needs to be done for the current parent ticket based on FLIP as follows. Could you please help take a look? In the current ticket order with my limited read: - Ticket 2 and ticket 3 can be independently advanced. Because both ticket 2 and ticket 3 are partially dependent on ticket 1 - Ticket 4 and ticket 5 are partially dependent on ticket 3, and ticket 5 and ticket 4 could be completed independently, because ticket 5 is transparent to the user. Looking forward your opinion about the task-splitting or discussion about any sub-ticket~ Thank you~ CC [~jingge] > Port JDBC Source to new Source API (FLIP-27) > > > Key: FLINK-25420 > URL: https://issues.apache.org/jira/browse/FLINK-25420 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: Martijn Visser >Assignee: RocMarshal >Priority: Major > > The current JDBC connector is using the old SourceFunction interface, which > is going to be deprecated. We should port/refactor the JDBC Source to use the > new Source API, based on FLIP-27 > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25420) Port JDBC Source to new Source API (FLIP-27)
[ https://issues.apache.org/jira/browse/FLINK-25420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782972#comment-17782972 ] RocMarshal commented on FLINK-25420: Hi, [~martijnvisser] [~eskabetxe] I have divided the work that needs to be done for the current parent ticket based on FLIP as follows. Could you please help take a look? In the current ticket order with my limited read: - Ticket 2 and ticket 3 can be independently advanced. Because both ticket 2 and ticket 3 are partially dependent on ticket 1 - Ticket 4 and ticket 5 are partially dependent on ticket 3, and ticket 5 and ticket 4 could be completed independently, because ticket 5 is transparent to the user. Looking forward your opinion about the task-splitting or discussion about any sub-ticket~ Thank you~ CC [~jingge] > Port JDBC Source to new Source API (FLIP-27) > > > Key: FLINK-25420 > URL: https://issues.apache.org/jira/browse/FLINK-25420 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: Martijn Visser >Assignee: RocMarshal >Priority: Major > > The current JDBC connector is using the old SourceFunction interface, which > is going to be deprecated. We should port/refactor the JDBC Source to use the > new Source API, based on FLIP-27 > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
[ https://issues.apache.org/jira/browse/FLINK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782969#comment-17782969 ] RocMarshal edited comment on FLINK-33463 at 11/5/23 9:36 AM: - The ticket is mainly to do the three items: 1. Support the implementation of dynamic source/factories tables based on the new source 2. Mark the old APIs about dynamic table source or factories as Deprecated. 3. Supplement the docs about the usage of stream semantic table or other extended feature if needed. was (Author: rocmarshal): The ticket is mainly to do the two items: 1. Support the implementation of dynamic source/factories tables based on the new source 2. Mark the old APIs about dynamic table source or factories as Deprecated. > Support the implementation of dynamic source tables based on the new source > --- > > Key: FLINK-33463 > URL: https://issues.apache.org/jira/browse/FLINK-33463 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
[ https://issues.apache.org/jira/browse/FLINK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782969#comment-17782969 ] RocMarshal commented on FLINK-33463: The ticket is mainly to do the two items: 1. Support the implementation of dynamic source/factories tables based on the new source 2. Mark the old APIs about dynamic table source or factories as Deprecated. > Support the implementation of dynamic source tables based on the new source > --- > > Key: FLINK-33463 > URL: https://issues.apache.org/jira/browse/FLINK-33463 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33459) Support the new source that keeps the same functionality as the original JDBC input format
[ https://issues.apache.org/jira/browse/FLINK-33459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782968#comment-17782968 ] RocMarshal commented on FLINK-33459: The ticket is mainly to do the two items: 1. Support the new source that keeps the same functionality as the original JDBC input format 2. Mark the old APIs as Deprecated. > Support the new source that keeps the same functionality as the original JDBC > input format > -- > > Key: FLINK-33459 > URL: https://issues.apache.org/jira/browse/FLINK-33459 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33462) Sort out the document page about the new Jdbc source.
RocMarshal created FLINK-33462: -- Summary: Sort out the document page about the new Jdbc source. Key: FLINK-33462 URL: https://issues.apache.org/jira/browse/FLINK-33462 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33459) Support the new source that keeps the same functionality as the original JDBC input format
[ https://issues.apache.org/jira/browse/FLINK-33459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33459: --- Summary: Support the new source that keeps the same functionality as the original JDBC input format (was: Support the new source that supports the same functionality as the original JDBC input format) > Support the new source that keeps the same functionality as the original JDBC > input format > -- > > Key: FLINK-33459 > URL: https://issues.apache.org/jira/browse/FLINK-33459 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
RocMarshal created FLINK-33463: -- Summary: Support the implementation of dynamic source tables based on the new source Key: FLINK-33463 URL: https://issues.apache.org/jira/browse/FLINK-33463 Project: Flink Issue Type: Sub-task Components: Table SQL / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33461) Support stream related semantics for the new jdbc source
RocMarshal created FLINK-33461: -- Summary: Support stream related semantics for the new jdbc source Key: FLINK-33461 URL: https://issues.apache.org/jira/browse/FLINK-33461 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33459) Support the new source that supports the same functionality as the original JDBC input format
[ https://issues.apache.org/jira/browse/FLINK-33459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33459: --- Summary: Support the new source that supports the same functionality as the original JDBC input format (was: Port the new source that supports the same functionality as the original JDBC input format) > Support the new source that supports the same functionality as the original > JDBC input format > - > > Key: FLINK-33459 > URL: https://issues.apache.org/jira/browse/FLINK-33459 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33459) Port the new source that supports the same functionality as the original JDBC input format
[ https://issues.apache.org/jira/browse/FLINK-33459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33459: --- Summary: Port the new source that supports the same functionality as the original JDBC input format (was: Introduce the new source that supports the same functionality as the original JDBC input format) > Port the new source that supports the same functionality as the original JDBC > input format > -- > > Key: FLINK-33459 > URL: https://issues.apache.org/jira/browse/FLINK-33459 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33460) Support more authentication connection types such as the secret.
RocMarshal created FLINK-33460: -- Summary: Support more authentication connection types such as the secret. Key: FLINK-33460 URL: https://issues.apache.org/jira/browse/FLINK-33460 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33459) Introduce the new source that supports the same functionality as the original JDBC input format
RocMarshal created FLINK-33459: -- Summary: Introduce the new source that supports the same functionality as the original JDBC input format Key: FLINK-33459 URL: https://issues.apache.org/jira/browse/FLINK-33459 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33448) Introduce a new configuration item 'taskmanager.load-balance.mode'
[ https://issues.apache.org/jira/browse/FLINK-33448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782503#comment-17782503 ] RocMarshal commented on FLINK-33448: Thanks for the reply~ :) > Introduce a new configuration item 'taskmanager.load-balance.mode' > -- > > Key: FLINK-33448 > URL: https://issues.apache.org/jira/browse/FLINK-33448 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > > Introduce a new configuration item 'taskmanager.load-balance.mode' and make > it compatible with "cluster.evenly-spread-out-slots" > The ticket is mainly to do three items: > - Introduce a new configuration item 'taskmanager.load-balance.mode' > - Make it compatible with "cluster.evenly-spread-out-slots" > - Mark "cluster.evenly-spread-out-slots" as Deprecated -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33448) Introduce a new configuration item 'taskmanager.load-balance.mode'
[ https://issues.apache.org/jira/browse/FLINK-33448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33448: --- Description: Introduce a new configuration item 'taskmanager.load-balance.mode' and make it compatible with "cluster.evenly-spread-out-slots" The ticket is mainly to do three items: - Introduce a new configuration item 'taskmanager.load-balance.mode' - Make it compatible with "cluster.evenly-spread-out-slots" - Mark "cluster.evenly-spread-out-slots" as Deprecated was: Introduce a new configuration item 'taskmanager.load-balance.mode' and make it compatible with "cluster.evenly-spread-out-slots" The ticket is mainly to do three items: - Introduce a new configuration item 'taskmanager.load-balance.mode' - Make it compatible with "cluster.evenly-spread-out-slots" - Marked "cluster.evenly-spread-out-slots" as Deprecated > Introduce a new configuration item 'taskmanager.load-balance.mode' > -- > > Key: FLINK-33448 > URL: https://issues.apache.org/jira/browse/FLINK-33448 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > > Introduce a new configuration item 'taskmanager.load-balance.mode' and make > it compatible with "cluster.evenly-spread-out-slots" > The ticket is mainly to do three items: > - Introduce a new configuration item 'taskmanager.load-balance.mode' > - Make it compatible with "cluster.evenly-spread-out-slots" > - Mark "cluster.evenly-spread-out-slots" as Deprecated -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33448) Introduce a new configuration item 'taskmanager.load-balance.mode'
[ https://issues.apache.org/jira/browse/FLINK-33448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782487#comment-17782487 ] RocMarshal commented on FLINK-33448: The Jira ticket resulted from [https://github.com/apache/flink/pull/23635#discussion_r1381212518] Hi, [~fanrui] would you help to take a look ? Thanks a lot~ > Introduce a new configuration item 'taskmanager.load-balance.mode' > -- > > Key: FLINK-33448 > URL: https://issues.apache.org/jira/browse/FLINK-33448 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > > Introduce a new configuration item 'taskmanager.load-balance.mode' and make > it compatible with "cluster.evenly-spread-out-slots" > The ticket is mainly to do three items: > - Introduce a new configuration item 'taskmanager.load-balance.mode' > - Make it compatible with "cluster.evenly-spread-out-slots" > - Marked "cluster.evenly-spread-out-slots" as Deprecated -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33448) Introduce a new configuration item 'taskmanager.load-balance.mode'
RocMarshal created FLINK-33448: -- Summary: Introduce a new configuration item 'taskmanager.load-balance.mode' Key: FLINK-33448 URL: https://issues.apache.org/jira/browse/FLINK-33448 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal Introduce a new configuration item 'taskmanager.load-balance.mode' and make it compatible with "cluster.evenly-spread-out-slots" The ticket is mainly to do three items: - Introduce a new configuration item 'taskmanager.load-balance.mode' - Make it compatible with "cluster.evenly-spread-out-slots" - Marked "cluster.evenly-spread-out-slots" as Deprecated -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33389) Support tasks balancing at slot level for Adaptive Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33389: --- Summary: Support tasks balancing at slot level for Adaptive Scheduler (was: Introduce the assigner for Adaptive Scheduler to pursuit task balancing based slots level) > Support tasks balancing at slot level for Adaptive Scheduler > > > Key: FLINK-33389 > URL: https://issues.apache.org/jira/browse/FLINK-33389 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Assignee: Rui Fan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33386) Support tasks balancing at slot level for Default Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33386: --- Summary: Support tasks balancing at slot level for Default Scheduler (was: Introduce the strategy for Default Scheduler to pursue tasks balancing based on slots level.) > Support tasks balancing at slot level for Default Scheduler > --- > > Key: FLINK-33386 > URL: https://issues.apache.org/jira/browse/FLINK-33386 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33388) Support tasks balancing at TM level for Default Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33388: --- Summary: Support tasks balancing at TM level for Default Scheduler (was: Implement tasks to taskmanagers balancing for the Default Scheduler) > Support tasks balancing at TM level for Default Scheduler > - > > Key: FLINK-33388 > URL: https://issues.apache.org/jira/browse/FLINK-33388 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33392) Add the documentation page for balanced tasks scheduling
RocMarshal created FLINK-33392: -- Summary: Add the documentation page for balanced tasks scheduling Key: FLINK-33392 URL: https://issues.apache.org/jira/browse/FLINK-33392 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33391) Support tasks balancing at TM level for Adaptive Scheduler
RocMarshal created FLINK-33391: -- Summary: Support tasks balancing at TM level for Adaptive Scheduler Key: FLINK-33391 URL: https://issues.apache.org/jira/browse/FLINK-33391 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33390) Support slot balancing at TM level for Adaptive Scheduler
RocMarshal created FLINK-33390: -- Summary: Support slot balancing at TM level for Adaptive Scheduler Key: FLINK-33390 URL: https://issues.apache.org/jira/browse/FLINK-33390 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33388) Implement tasks to taskmanagers balancing for the Default Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33388: --- Summary: Implement tasks to taskmanagers balancing for the Default Scheduler (was: Implement slots to taskmanagers balancing for the Default Scheduler) > Implement tasks to taskmanagers balancing for the Default Scheduler > --- > > Key: FLINK-33388 > URL: https://issues.apache.org/jira/browse/FLINK-33388 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33389) Introduce the assigner for Adaptive Scheduler to pursuit task balancing based slots level
RocMarshal created FLINK-33389: -- Summary: Introduce the assigner for Adaptive Scheduler to pursuit task balancing based slots level Key: FLINK-33389 URL: https://issues.apache.org/jira/browse/FLINK-33389 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33388) Implement slots to taskmanagers balancing for the Default Scheduler
RocMarshal created FLINK-33388: -- Summary: Implement slots to taskmanagers balancing for the Default Scheduler Key: FLINK-33388 URL: https://issues.apache.org/jira/browse/FLINK-33388 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33387) Introduce the abstraction and the interface about loading
RocMarshal created FLINK-33387: -- Summary: Introduce the abstraction and the interface about loading Key: FLINK-33387 URL: https://issues.apache.org/jira/browse/FLINK-33387 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33386) Introduce the strategy for Default Scheduler to pursue tasks balancing based on slots level.
RocMarshal created FLINK-33386: -- Summary: Introduce the strategy for Default Scheduler to pursue tasks balancing based on slots level. Key: FLINK-33386 URL: https://issues.apache.org/jira/browse/FLINK-33386 Project: Flink Issue Type: Sub-task Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33320) Support Dynamic Logger Level Adjustment
[ https://issues.apache.org/jira/browse/FLINK-33320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1522#comment-1522 ] RocMarshal commented on FLINK-33320: Hi, [~fanrui] Thank you for the assign~ I'm willing to contribute to this ticket. Although this change is relatively minor, I still want to confirm if there is any other discussion needed before starting the contribution. Such as discussion or FLIP. Thank you very much~ > Support Dynamic Logger Level Adjustment > --- > > Key: FLINK-33320 > URL: https://issues.apache.org/jira/browse/FLINK-33320 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: REST_API, runtime > > During the process of routine program debugging or troubleshooting, analyzing > system logs is a common approach. > Comprehensive and detailed system logs contribute to improved visibility of > internal system execution information and also enhance the efficiency of > program debugging or issue troubleshooting.However, comprehensive and > detailed log settings can lead to the following issues: > # A sharp increase in log volume, accelerating disk occupancy. > # Potential risks of system performance degradation due to a large volume of > log printing. > # The need to simplify log configuration subsequently. > Therefore, introducing a mechanism to dynamically adjust the online log > output level in the event of diagnosing online issues or debugging programs > could be meaningful. > This mechanism should ideally provide the following two basic capabilities: > # Dynamically adjust log levels. > # Query the current log levels of the JM/TM in the cluster. > > The proposal doc: > https://docs.google.com/document/d/1s2XQzet_8oPhMs3WyDhP_pPhAE3d1Gdw_qR4W0nKtlY/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33320) Support Dynamic Logger Level Adjustment
RocMarshal created FLINK-33320: -- Summary: Support Dynamic Logger Level Adjustment Key: FLINK-33320 URL: https://issues.apache.org/jira/browse/FLINK-33320 Project: Flink Issue Type: Improvement Components: Runtime / REST Reporter: RocMarshal During the process of routine program debugging or troubleshooting, analyzing system logs is a common approach. Comprehensive and detailed system logs contribute to improved visibility of internal system execution information and also enhance the efficiency of program debugging or issue troubleshooting.However, comprehensive and detailed log settings can lead to the following issues: # A sharp increase in log volume, accelerating disk occupancy. # Potential risks of system performance degradation due to a large volume of log printing. # The need to simplify log configuration subsequently. Therefore, introducing a mechanism to dynamically adjust the online log output level in the event of diagnosing online issues or debugging programs could be meaningful. This mechanism should ideally provide the following two basic capabilities: # Dynamically adjust log levels. # Query the current log levels of the JM/TM in the cluster. The proposal doc: https://docs.google.com/document/d/1s2XQzet_8oPhMs3WyDhP_pPhAE3d1Gdw_qR4W0nKtlY/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25420) Port JDBC Source to new Source API (FLIP-27)
[ https://issues.apache.org/jira/browse/FLINK-25420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774850#comment-17774850 ] RocMarshal commented on FLINK-25420: Hi, [~eskabetxe] [~martijnvisser] Thank you very much for excellent work and excellent and discussion from other contributors on FLIP May I get this ticket? I would do some task split to facilitate better collaboration and development If I would get the ticket. Thank you~ :) > Port JDBC Source to new Source API (FLIP-27) > > > Key: FLINK-25420 > URL: https://issues.apache.org/jira/browse/FLINK-25420 > Project: Flink > Issue Type: Improvement >Reporter: Martijn Visser >Priority: Major > > The current JDBC connector is using the old SourceFunction interface, which > is going to be deprecated. We should port/refactor the JDBC Source to use the > new Source API, based on FLIP-27 > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31757) Support Balanced Tasks Scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RocMarshal updated FLINK-31757:
---
Summary: Support Balanced Tasks Scheduling (was: Optimize Flink
un-balanced tasks scheduling)
> Support Balanced Tasks Scheduling
> -
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-04-13-08-04-04-667.png
>
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RocMarshal updated FLINK-31757:
---
Labels: pull-request-available (was: pull-request-available stale-assigned)
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-04-13-08-04-04-667.png
>
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750078#comment-17750078
]
RocMarshal edited comment on FLINK-31757 at 8/2/23 3:54 AM:
Hi, [~heigebupahei] Thanks for your attention. we've updated the new edition
design docs in
[https://docs.google.com/document/d/14WhrSNGBdcsRl3IK7CZO-RaZ5KXU2X1dWqxPEFr3iS8/edit?usp=sharing]
Would you mind having a look on the document ? Any suggestion is appreciated~
was (Author: rocmarshal):
Hi, [~heigebupahei] Thanks for your attention. we've updated the new edition
design docs in
[https://docs.google.com/document/d/14WhrSNGBdcsRl3IK7CZO-RaZ5KXU2X1dWqxPEFr3iS8/edit?usp=sharing]
Would you mind having a look on the doc ? Any suggestion is appreciated~
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-04-13-08-04-04-667.png
>
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750078#comment-17750078
]
RocMarshal commented on FLINK-31757:
Hi, [~heigebupahei] Thanks for your attention. we've updated the new edition
design docs in
[https://docs.google.com/document/d/14WhrSNGBdcsRl3IK7CZO-RaZ5KXU2X1dWqxPEFr3iS8/edit?usp=sharing]
Would you mind having a look on the doc ? Any suggestion is appreciated~
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-04-13-08-04-04-667.png
>
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739030#comment-17739030 ] RocMarshal commented on FLINK-29050: Hello, sorry for the late update. Could someone help to review it ? I'd appreciated it with your help. :D > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Priority: Major > Labels: pull-request-available, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735249#comment-17735249 ] RocMarshal commented on FLINK-20628: [~martijnvisser] Yes,I'd like to do it. There's nothing better. Thank you~:) > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: RocMarshal >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17734964#comment-17734964 ] RocMarshal commented on FLINK-20628: [~martijnvisser] Thank you for your reply. Yes, I'd like to continue on the open PR. May I take the ticket ? > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733990#comment-17733990 ] RocMarshal commented on FLINK-20628: Hi, [pscls|https://github.com/pscls] Thank you very much for the contribution. I notice that this ticket has not been updated for a long time. Would you like to continue advancing it ? After the PR completed, FLINK-25380 will be introduced. Looking forward to your opinion. Thanks. CC [~martijnvisser] [~monster#12] > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.12.0 > > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727041#comment-17727041
]
RocMarshal commented on FLINK-31757:
I have compiled a draft
[https://docs.google.com/document/d/14WhrSNGBdcsRl3IK7CZO-RaZ5KXU2X1dWqxPEFr3iS8/edit?usp=sharing]
Looking forward your discussion. [~fanrui] [~huwh] [~Weijie Guo] [~chesnay]
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Attachments: image-2023-04-13-08-04-04-667.png
>
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727041#comment-17727041
]
RocMarshal edited comment on FLINK-31757 at 5/29/23 8:25 AM:
-
I have compiled a draft
[https://docs.google.com/document/d/14WhrSNGBdcsRl3IK7CZO-RaZ5KXU2X1dWqxPEFr3iS8/edit?usp=sharing]
Looking forward your discussion. [~fanrui] [~huwh] [~Weijie Guo] [~chesnay]
Thank you.
was (Author: rocmarshal):
I have compiled a draft
[https://docs.google.com/document/d/14WhrSNGBdcsRl3IK7CZO-RaZ5KXU2X1dWqxPEFr3iS8/edit?usp=sharing]
Looking forward your discussion. [~fanrui] [~huwh] [~Weijie Guo] [~chesnay]
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Attachments: image-2023-04-13-08-04-04-667.png
>
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715580#comment-17715580
]
RocMarshal commented on FLINK-31757:
I am still working on this jira, and due to the Labor Day holiday, I will
provide a design draft as soon as possible after it.
Thanks a lot.
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Attachments: image-2023-04-13-08-04-04-667.png
>
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711897#comment-17711897
]
RocMarshal commented on FLINK-31757:
Thanks for your [~Weijie Guo] sorted concise and precise description. There's
nothing better.(y)
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711895#comment-17711895
]
RocMarshal commented on FLINK-31757:
Thank you [~fanrui] & [~huwh] very much.
I'll prepare a design document for discussion.
> Optimize Flink un-balanced tasks scheduling
> ---
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
>
> Supposed we have a Job with 21 {{JobVertex}}. The parallelism of vertex A is
> 100, and the others are 5. If each {{TaskManager}} only have one slot, then
> we need 100 TMs.
> There will be 5 slots with 21 sub-tasks, and the others will only have one
> sub-task of A. Does this mean we have to make a trade-off between wasted
> resources and insufficient resources?
> From a resource utilization point of view, we expect all subtasks to be
> evenly distributed on each TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[ https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711875#comment-17711875 ] RocMarshal commented on FLINK-31757: Based on this problem, we has achieved balanced task distribution on TaskManager. I would very much like to be able to contribute it. Would you [~huwh] like to contribute it together ? > Optimize Flink un-balanced tasks scheduling > --- > > Key: FLINK-31757 > URL: https://issues.apache.org/jira/browse/FLINK-31757 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[ https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal reopened FLINK-31757: > Optimize Flink un-balanced tasks scheduling > --- > > Key: FLINK-31757 > URL: https://issues.apache.org/jira/browse/FLINK-31757 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[ https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711776#comment-17711776 ] RocMarshal edited comment on FLINK-31757 at 4/13/23 9:24 AM: - h1. Problem description and impact h1. The case Supposed a Job has 21 tasks: * Task A has the parallelism of 100, * The every remained task has the parallelism of 5. Each TM slot = 1, so the tasks in the job need to apply for 100 TMs. h2. Problem Description Assuming that the TM number is 0-99, from the perspective of Task, the actual result after scheduling is: After the job deployed. There are 5 TMs loading with 21 sub-tasks, while other TMs only load a sub-task. h2. Influence If the user allocates resources to TM: All TM resources are applied according to the 5 TMs (loading 21-subtasks), then subsequent TM resources will be wasted. If apply the resources based on other TM(only loading a subtask), the 5 TMs resources are insufficient, tasks running on its may have lag. >From the perspective of resource usage, we expect all subtasks to be evenly >distributed on each TM. was (Author: rocmarshal): h1. Problem description and impact Supposed a Job has 21 tasks: * Task A has the parallelism of 100, * The every remained task has the parallelism of 5. Each TM slot = 1, so the tasks in the job need to apply for 100 TMs. h2. Problem Description Assuming that the TM number is 0-99, from the perspective of Task, the actual result after scheduling is: After the job deployed. There are 5 TMs loading with 21 sub-tasks, while other TMs only load a sub-task. h2. Influence If the user allocates resources to TM: All TM resources are applied according to the 5 TMs (loading 21-subtasks), then subsequent TM resources will be wasted. If apply the resources based on other TM(only loading a subtask), the 5 TMs resources are insufficient, tasks running on its may have lag. >From the perspective of resource usage, we expect all subtasks to be evenly >distributed on each TM. > Optimize Flink un-balanced tasks scheduling > --- > > Key: FLINK-31757 > URL: https://issues.apache.org/jira/browse/FLINK-31757 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[ https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711776#comment-17711776 ] RocMarshal commented on FLINK-31757: h1. Problem description and impact Supposed a Job has 21 tasks: * Task A has the parallelism of 100, * The every remained task has the parallelism of 5. Each TM slot = 1, so the tasks in the job need to apply for 100 TMs. h2. Problem Description Assuming that the TM number is 0-99, from the perspective of Task, the actual result after scheduling is: After the job deployed. There are 5 TMs loading with 21 sub-tasks, while other TMs only load a sub-task. h2. Influence If the user allocates resources to TM: All TM resources are applied according to the 5 TMs (loading 21-subtasks), then subsequent TM resources will be wasted. If apply the resources based on other TM(only loading a subtask), the 5 TMs resources are insufficient, tasks running on its may have lag. >From the perspective of resource usage, we expect all subtasks to be evenly >distributed on each TM. > Optimize Flink un-balanced tasks scheduling > --- > > Key: FLINK-31757 > URL: https://issues.apache.org/jira/browse/FLINK-31757 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29672) Support oracle catalog
[ https://issues.apache.org/jira/browse/FLINK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710104#comment-17710104 ] RocMarshal commented on FLINK-29672: It looks like a duplicated Jira as https://issues.apache.org/jira/browse/FLINK-17508 > Support oracle catalog > --- > > Key: FLINK-29672 > URL: https://issues.apache.org/jira/browse/FLINK-29672 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.16.0 >Reporter: waywtdcc >Priority: Major > > Support oracle catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[ https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710022#comment-17710022 ] RocMarshal commented on FLINK-31757: [~Weijie Guo] Glad to get your attention and reminding~:) I'll add the background and cases description later. Looking forward your discussion after that~ Thanks a lot. > Optimize Flink un-balanced tasks scheduling > --- > > Key: FLINK-31757 > URL: https://issues.apache.org/jira/browse/FLINK-31757 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31757) Optimize Flink un-balanced tasks scheduling
[ https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-31757: --- Summary: Optimize Flink un-balanced tasks scheduling (was: Optimize Flink tasks un-balanced scheduling) > Optimize Flink un-balanced tasks scheduling > --- > > Key: FLINK-31757 > URL: https://issues.apache.org/jira/browse/FLINK-31757 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31757) Optimize Flink tasks un-balanced scheduling
RocMarshal created FLINK-31757: -- Summary: Optimize Flink tasks un-balanced scheduling Key: FLINK-31757 URL: https://issues.apache.org/jira/browse/FLINK-31757 Project: Flink Issue Type: Improvement Components: Runtime / Task Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29816) Userfunction exception in ProcessWindowFunction was called before invoke during restore state(subtask was in INITIALIZING state), but SteamTask skip handle Exception
[
https://issues.apache.org/jira/browse/FLINK-29816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692057#comment-17692057
]
RocMarshal commented on FLINK-29816:
As described in the historical comments and description text,
The exceptions happen to `StreamTask` during the `restore()` was ignored by
`asyncExceptionHandler`.
At the `Execution` side, it is possible to enter the \{@code FAILED} state from
any other state described at `ExecutionState` class. However, here's no
`isInitializing` flag or `Initializing` state in StreamTask.
We can deal the issue with the state rule of `ExecutionState`.
- Introduce is `isInitializing` flag for `StreamTask` in order to help
`asyncExceptionHandler` judge handle branch. It is worth noting that such an
approach would result in two adjacent states where it is unsafe to change the
value of the flags, and we can only rely on overlapping boundary conditions to
ensure that exceptions can be handled
!image-2023-02-22-17-26-06-200.png!
* Or we can introduce a State Enum for `StreamTask` like `ExecutionState`, If
so, we should ensure that the state introduced is simple and overrides the
current StreamTask state transition as a basic standard, and the security of
state transitions(thread-safe).
Please let me know what's your opinon. Thanks so much~
CC [~xieyi] [~Weijie Guo] [~kevin.cyj]
> Userfunction exception in ProcessWindowFunction was called before invoke
> during restore state(subtask was in INITIALIZING state), but SteamTask skip
> handle Exception
> -
>
> Key: FLINK-29816
> URL: https://issues.apache.org/jira/browse/FLINK-29816
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0, 1.16.0, 1.15.3
>Reporter: Xie Yi
>Assignee: RocMarshal
>Priority: Major
> Attachments: image-2022-10-31-19-49-52-432.png,
> image-2022-10-31-19-54-12-546.png, image-2022-11-02-10-42-21-099.png,
> image-2022-11-02-10-57-08-064.png, image-2022-11-02-11-06-37-925.png,
> image-2022-11-02-11-10-25-508.png, image-2023-02-22-17-26-06-200.png
>
>
> h4. 1. How to repeat
> ProcessWindowFunction, and make some exception in process()
> test code
> {code:java}
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> env.enableCheckpointing(60 * 1000);
>
> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> env.getCheckpointConfig().setCheckpointTimeout(6);
> KafkaSource kafkaConsumer = KafkaSource.builder()
> .setBootstrapServers("")
> .setTopics("")
> .setGroupId("")
> .setValueOnlyDeserializer(new SimpleStringSchema())
> .setStartingOffsets(OffsetsInitializer.earliest())
> .build();
> DataStreamSource kafkaSource = env.fromSource(kafkaConsumer,
> WatermarkStrategy.noWatermarks(), "Kafka Source");
> SingleOutputStreamOperator mapSourse = kafkaSource.keyBy(s ->
> s).window(TumblingProcessingTimeWindows.of(Time.seconds(15)))
> .process(new ProcessWindowFunction TimeWindow>() {
> @Override
> public void process(String s,
> ProcessWindowFunction.Context context,
> Iterable iterable, Collector collector) throws Exception {
> //when process event:"abc" .It causes
> java.lang.NumberFormatException
> Integer intS = Integer.valueOf(s);
> collector.collect(s);
> }
> })
> .name("name-process").uid("uid-process");
> mapSourse.print();
> env.execute();
> }
> {code}
> kafka input event
> {code:java}
> >1
> >1
> >2
> >2
> >3
> >3
> >abc
> >abc
> >
> {code}
> h4. 2. fault phenomena
> when job process the event:"abc",It will cause
> java.lang.NumberFormatException and failover ,Then attempt and failover
> continuously.
> However, it only failover 2 times(attempt 0, attempt 1) and when attempt for
> third time, It work normally, and no exception
> !image-2022-10-31-19-54-12-546.png!
> checkpoint 1 complete in attempt 1,before failover exception 1
> {code:java}
> 2022-10-31 16:59:53,644 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 1 (type=CHECKPOINT) @ 1667206793605 for job
> 7bca78a75b089d447bb4c99efcfd6527.2022-10-31 16:59:54,010 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
> checkpoint 1 for job
[jira] [Updated] (FLINK-29816) Userfunction exception in ProcessWindowFunction was called before invoke during restore state(subtask was in INITIALIZING state), but SteamTask skip handle Exception
[
https://issues.apache.org/jira/browse/FLINK-29816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RocMarshal updated FLINK-29816:
---
Attachment: image-2023-02-22-17-26-06-200.png
> Userfunction exception in ProcessWindowFunction was called before invoke
> during restore state(subtask was in INITIALIZING state), but SteamTask skip
> handle Exception
> -
>
> Key: FLINK-29816
> URL: https://issues.apache.org/jira/browse/FLINK-29816
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0, 1.16.0, 1.15.3
>Reporter: Xie Yi
>Assignee: RocMarshal
>Priority: Major
> Attachments: image-2022-10-31-19-49-52-432.png,
> image-2022-10-31-19-54-12-546.png, image-2022-11-02-10-42-21-099.png,
> image-2022-11-02-10-57-08-064.png, image-2022-11-02-11-06-37-925.png,
> image-2022-11-02-11-10-25-508.png, image-2023-02-22-17-26-06-200.png
>
>
> h4. 1. How to repeat
> ProcessWindowFunction, and make some exception in process()
> test code
> {code:java}
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> env.enableCheckpointing(60 * 1000);
>
> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> env.getCheckpointConfig().setCheckpointTimeout(6);
> KafkaSource kafkaConsumer = KafkaSource.builder()
> .setBootstrapServers("")
> .setTopics("")
> .setGroupId("")
> .setValueOnlyDeserializer(new SimpleStringSchema())
> .setStartingOffsets(OffsetsInitializer.earliest())
> .build();
> DataStreamSource kafkaSource = env.fromSource(kafkaConsumer,
> WatermarkStrategy.noWatermarks(), "Kafka Source");
> SingleOutputStreamOperator mapSourse = kafkaSource.keyBy(s ->
> s).window(TumblingProcessingTimeWindows.of(Time.seconds(15)))
> .process(new ProcessWindowFunction TimeWindow>() {
> @Override
> public void process(String s,
> ProcessWindowFunction.Context context,
> Iterable iterable, Collector collector) throws Exception {
> //when process event:"abc" .It causes
> java.lang.NumberFormatException
> Integer intS = Integer.valueOf(s);
> collector.collect(s);
> }
> })
> .name("name-process").uid("uid-process");
> mapSourse.print();
> env.execute();
> }
> {code}
> kafka input event
> {code:java}
> >1
> >1
> >2
> >2
> >3
> >3
> >abc
> >abc
> >
> {code}
> h4. 2. fault phenomena
> when job process the event:"abc",It will cause
> java.lang.NumberFormatException and failover ,Then attempt and failover
> continuously.
> However, it only failover 2 times(attempt 0, attempt 1) and when attempt for
> third time, It work normally, and no exception
> !image-2022-10-31-19-54-12-546.png!
> checkpoint 1 complete in attempt 1,before failover exception 1
> {code:java}
> 2022-10-31 16:59:53,644 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 1 (type=CHECKPOINT) @ 1667206793605 for job
> 7bca78a75b089d447bb4c99efcfd6527.2022-10-31 16:59:54,010 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
> checkpoint 1 for job 7bca78a75b089d447bb4c99efcfd6527 (21630 bytes,
> checkpointDuration=333 ms, finalizationTime=72 ms). {code}
>
> attempt 2 was restore from checkpoint
> {code:java}
> 2022-10-31 17:00:30,033 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring
> job 7bca78a75b089d447bb4c99efcfd6527 from Checkpoint 1 @ 1667206793605 for
> 7bca78a75b089d447bb4c99efcfd6527 located at
> hdfs://eadhadoop/user/sloth/sloth-fs-checkpoints/meta/1_7/7bca78a75b089d447bb4c99efcfd6527/chk-1.
> {code}
>
>
> h4. 3. possible reasons
> during attempt 2 , task restore from checkpoint, userfunction in
> ProcessWindowFunction was called in SteamTask.restore and produce
> "java.lang.NumberFormatException", However, SteamTask catch exception and
> didn't handle exception because subtask is not in RUNNING state.
> *the stack trace in attempt 2*
> user function was called in SteamTask.restore(subtask state is INITIALIZING)
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> com.youdao.analysis.KafkaCheckpointWindowProcessTest$1.process(KafkaCheckpointWindowProcessTest.java:45)
>
[jira] [Commented] (FLINK-29816) Userfunction exception in ProcessWindowFunction was called before invoke during restore state(subtask was in INITIALIZING state), but SteamTask skip handle Exception
[
https://issues.apache.org/jira/browse/FLINK-29816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691339#comment-17691339
]
RocMarshal commented on FLINK-29816:
[~Weijie Guo] thanks for the reply.
I'm interested in it. May I get the ticket ?
IMO, before starting the process, we need sort out the `handleAsyncException`
mechanism & state-switch of `SteamTask`
> Userfunction exception in ProcessWindowFunction was called before invoke
> during restore state(subtask was in INITIALIZING state), but SteamTask skip
> handle Exception
> -
>
> Key: FLINK-29816
> URL: https://issues.apache.org/jira/browse/FLINK-29816
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0, 1.16.0, 1.15.3
>Reporter: Xie Yi
>Assignee: Weijie Guo
>Priority: Major
> Attachments: image-2022-10-31-19-49-52-432.png,
> image-2022-10-31-19-54-12-546.png, image-2022-11-02-10-42-21-099.png,
> image-2022-11-02-10-57-08-064.png, image-2022-11-02-11-06-37-925.png,
> image-2022-11-02-11-10-25-508.png
>
>
> h4. 1. How to repeat
> ProcessWindowFunction, and make some exception in process()
> test code
> {code:java}
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> env.enableCheckpointing(60 * 1000);
>
> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> env.getCheckpointConfig().setCheckpointTimeout(6);
> KafkaSource kafkaConsumer = KafkaSource.builder()
> .setBootstrapServers("")
> .setTopics("")
> .setGroupId("")
> .setValueOnlyDeserializer(new SimpleStringSchema())
> .setStartingOffsets(OffsetsInitializer.earliest())
> .build();
> DataStreamSource kafkaSource = env.fromSource(kafkaConsumer,
> WatermarkStrategy.noWatermarks(), "Kafka Source");
> SingleOutputStreamOperator mapSourse = kafkaSource.keyBy(s ->
> s).window(TumblingProcessingTimeWindows.of(Time.seconds(15)))
> .process(new ProcessWindowFunction TimeWindow>() {
> @Override
> public void process(String s,
> ProcessWindowFunction.Context context,
> Iterable iterable, Collector collector) throws Exception {
> //when process event:"abc" .It causes
> java.lang.NumberFormatException
> Integer intS = Integer.valueOf(s);
> collector.collect(s);
> }
> })
> .name("name-process").uid("uid-process");
> mapSourse.print();
> env.execute();
> }
> {code}
> kafka input event
> {code:java}
> >1
> >1
> >2
> >2
> >3
> >3
> >abc
> >abc
> >
> {code}
> h4. 2. fault phenomena
> when job process the event:"abc",It will cause
> java.lang.NumberFormatException and failover ,Then attempt and failover
> continuously.
> However, it only failover 2 times(attempt 0, attempt 1) and when attempt for
> third time, It work normally, and no exception
> !image-2022-10-31-19-54-12-546.png!
> checkpoint 1 complete in attempt 1,before failover exception 1
> {code:java}
> 2022-10-31 16:59:53,644 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 1 (type=CHECKPOINT) @ 1667206793605 for job
> 7bca78a75b089d447bb4c99efcfd6527.2022-10-31 16:59:54,010 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
> checkpoint 1 for job 7bca78a75b089d447bb4c99efcfd6527 (21630 bytes,
> checkpointDuration=333 ms, finalizationTime=72 ms). {code}
>
> attempt 2 was restore from checkpoint
> {code:java}
> 2022-10-31 17:00:30,033 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring
> job 7bca78a75b089d447bb4c99efcfd6527 from Checkpoint 1 @ 1667206793605 for
> 7bca78a75b089d447bb4c99efcfd6527 located at
> hdfs://eadhadoop/user/sloth/sloth-fs-checkpoints/meta/1_7/7bca78a75b089d447bb4c99efcfd6527/chk-1.
> {code}
>
>
> h4. 3. possible reasons
> during attempt 2 , task restore from checkpoint, userfunction in
> ProcessWindowFunction was called in SteamTask.restore and produce
> "java.lang.NumberFormatException", However, SteamTask catch exception and
> didn't handle exception because subtask is not in RUNNING state.
> *the stack trace in attempt 2*
> user function was called in SteamTask.restore(subtask state is INITIALIZING)
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
>
[jira] [Commented] (FLINK-29816) Userfunction exception in ProcessWindowFunction was called before invoke during restore state(subtask was in INITIALIZING state), but SteamTask skip handle Exception
[
https://issues.apache.org/jira/browse/FLINK-29816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691132#comment-17691132
]
RocMarshal commented on FLINK-29816:
hi, [~Weijie Guo] Any process on this issue ?
> Userfunction exception in ProcessWindowFunction was called before invoke
> during restore state(subtask was in INITIALIZING state), but SteamTask skip
> handle Exception
> -
>
> Key: FLINK-29816
> URL: https://issues.apache.org/jira/browse/FLINK-29816
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0, 1.16.0, 1.15.3
>Reporter: Xie Yi
>Assignee: Weijie Guo
>Priority: Major
> Attachments: image-2022-10-31-19-49-52-432.png,
> image-2022-10-31-19-54-12-546.png, image-2022-11-02-10-42-21-099.png,
> image-2022-11-02-10-57-08-064.png, image-2022-11-02-11-06-37-925.png,
> image-2022-11-02-11-10-25-508.png
>
>
> h4. 1. How to repeat
> ProcessWindowFunction, and make some exception in process()
> test code
> {code:java}
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> env.enableCheckpointing(60 * 1000);
>
> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> env.getCheckpointConfig().setCheckpointTimeout(6);
> KafkaSource kafkaConsumer = KafkaSource.builder()
> .setBootstrapServers("")
> .setTopics("")
> .setGroupId("")
> .setValueOnlyDeserializer(new SimpleStringSchema())
> .setStartingOffsets(OffsetsInitializer.earliest())
> .build();
> DataStreamSource kafkaSource = env.fromSource(kafkaConsumer,
> WatermarkStrategy.noWatermarks(), "Kafka Source");
> SingleOutputStreamOperator mapSourse = kafkaSource.keyBy(s ->
> s).window(TumblingProcessingTimeWindows.of(Time.seconds(15)))
> .process(new ProcessWindowFunction TimeWindow>() {
> @Override
> public void process(String s,
> ProcessWindowFunction.Context context,
> Iterable iterable, Collector collector) throws Exception {
> //when process event:"abc" .It causes
> java.lang.NumberFormatException
> Integer intS = Integer.valueOf(s);
> collector.collect(s);
> }
> })
> .name("name-process").uid("uid-process");
> mapSourse.print();
> env.execute();
> }
> {code}
> kafka input event
> {code:java}
> >1
> >1
> >2
> >2
> >3
> >3
> >abc
> >abc
> >
> {code}
> h4. 2. fault phenomena
> when job process the event:"abc",It will cause
> java.lang.NumberFormatException and failover ,Then attempt and failover
> continuously.
> However, it only failover 2 times(attempt 0, attempt 1) and when attempt for
> third time, It work normally, and no exception
> !image-2022-10-31-19-54-12-546.png!
> checkpoint 1 complete in attempt 1,before failover exception 1
> {code:java}
> 2022-10-31 16:59:53,644 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 1 (type=CHECKPOINT) @ 1667206793605 for job
> 7bca78a75b089d447bb4c99efcfd6527.2022-10-31 16:59:54,010 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
> checkpoint 1 for job 7bca78a75b089d447bb4c99efcfd6527 (21630 bytes,
> checkpointDuration=333 ms, finalizationTime=72 ms). {code}
>
> attempt 2 was restore from checkpoint
> {code:java}
> 2022-10-31 17:00:30,033 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring
> job 7bca78a75b089d447bb4c99efcfd6527 from Checkpoint 1 @ 1667206793605 for
> 7bca78a75b089d447bb4c99efcfd6527 located at
> hdfs://eadhadoop/user/sloth/sloth-fs-checkpoints/meta/1_7/7bca78a75b089d447bb4c99efcfd6527/chk-1.
> {code}
>
>
> h4. 3. possible reasons
> during attempt 2 , task restore from checkpoint, userfunction in
> ProcessWindowFunction was called in SteamTask.restore and produce
> "java.lang.NumberFormatException", However, SteamTask catch exception and
> didn't handle exception because subtask is not in RUNNING state.
> *the stack trace in attempt 2*
> user function was called in SteamTask.restore(subtask state is INITIALIZING)
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> com.youdao.analysis.KafkaCheckpointWindowProcessTest$1.process(KafkaCheckpointWindowProcessTest.java:45)
>
[jira] [Commented] (FLINK-30511) Ignore the Exception in user-timer Triggerble when recover form state.
[
https://issues.apache.org/jira/browse/FLINK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654895#comment-17654895
]
RocMarshal commented on FLINK-30511:
Thank you [~Weijie Guo] [~kevin.cyj] . Looking forward to the fix.
> Ignore the Exception in user-timer Triggerble when recover form state.
> --
>
> Key: FLINK-30511
> URL: https://issues.apache.org/jira/browse/FLINK-30511
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.16.0
> Environment: Flink 1.16.0
> java8
> deployment Mode: miniCluster in IDC; standalone, yarn-application.
>Reporter: RocMarshal
>Priority: Minor
> Attachments: 截屏2022-12-27 18.51.12.png, 截屏2022-12-27 19.20.00.png
>
>
> * Code segment:
> {code:java}
> public class OnTimerDemo {
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.setString("taskmanager.numberOfTaskSlots", "4");
> conf.setString("state.checkpoint-storage", "filesystem");
> conf.setString("state.checkpoints.dir", "file:///tmp/flinkjob");
> conf.setString("execution.checkpointing.interval", "30s");
> //conf.setString("execution.savepoint.path",
> "file:///tmp/flinkjob/159561b8c97c9e0b4f9eeb649086796a/chk-1"); // Anchor-A:
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
> env.setParallelism(1);
> EnvironmentSettings envSetting = EnvironmentSettings
> .newInstance()
> .inStreamingMode()
> .build();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env,
> envSetting);
> String sourceDDL = "CREATE TABLE orders (\n" +
> " id INT,\n" +
> " app INT,\n" +
> " user_id STRING" +
> ") WITH (\n" +
> " 'connector' = 'datagen',\n" +
> " 'rows-per-second'='1',\n" +
> " 'fields.app.min'='1',\n" +
> " 'fields.app.max'='10',\n" +
> " 'fields.user_id.length'='10'\n" +
> ")";
> tableEnv.executeSql(sourceDDL);
> Table query = tableEnv.sqlQuery("select * from orders");
> DataStream rowDataStream = tableEnv.toAppendStream(query,
> Row.class);
> TypeInformation[] returnTypes = new TypeInformation[4];
> returnTypes[0] = Types.INT;
> returnTypes[1] = Types.INT; // Anchor-B:
> returnTypes[2] = Types.INT;
> returnTypes[3] = Types.INT;
> rowDataStream.keyBy(new KeySelector() {
> @Override
> public String getKey(Row value) throws Exception {
> return value.getFieldAs(2);
> }
> }).process(new KeyedProcessFunction() {
> private Row firstRow;
> @Override
> public void processElement(Row value, Context ctx,
> Collector out) throws Exception {
> if (firstRow == null) {
> firstRow = value;
> }
>
> ctx.timerService().registerProcessingTimeTimer(System.currentTimeMillis() +
> 3000);
> }
> @Override
> public void onTimer(long timestamp, OnTimerContext ctx,
> Collector out) throws Exception {
> Row colRow = new Row(4);
> colRow.setField(0, 0);
> colRow.setField(1, 1);
> colRow.setField(2, 2);
> colRow.setField(3, 3);
> out.collect(colRow); // Anchor-C
> }
> }).name("TargetTestUDF")
> .returns(new RowTypeInfo(returnTypes))
> .print();
> env.execute(OnTimerDemo.class.getSimpleName());
> }
> }
> {code}
> * Recurrence steps
> ** Run the job without state.
> ** Collect the latest available checkpoint path as 'checkpoint-path-a'
> ** Stop the job.
> ** Fill the real value of 'checkpoint-path-a' into 'Anchor-A' line and
> un-comment the line.
> ** Set 'returnTypes[1] = Types.INT;' -> 'returnTypes[1] = Types.LONG;' at
> the 'Anchor-B' line.
> ** Then add break-point at 'StreamTask#handleAsyncException' method.
> ** Run the job. The 'java.lang.ClassCastException: java.lang.Integer cannot
> be cast to java.lang.Long' exception caused at the 'Anchor-C' line will
> ignore at 'StreamTask#handleAsyncException' method.
> ** So, The framework can't catch the same exception in the case.
> *
