[jira] [Updated] (FLINK-35448) Translate pod templates documentation into Chinese
[ https://issues.apache.org/jira/browse/FLINK-35448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35448: --- Labels: pull-request-available (was: ) > Translate pod templates documentation into Chinese > -- > > Key: FLINK-35448 > URL: https://issues.apache.org/jira/browse/FLINK-35448 > Project: Flink > Issue Type: Sub-task >Reporter: Caican Cai >Priority: Minor > Labels: pull-request-available > > Translate pod templates documentation into Chinese > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35447) Flink CDC Document document file had removed but website can access
[ https://issues.apache.org/jira/browse/FLINK-35447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35447: --- Labels: pull-request-available (was: ) > Flink CDC Document document file had removed but website can access > --- > > Key: FLINK-35447 > URL: https://issues.apache.org/jira/browse/FLINK-35447 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > > https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/connectors/overview/ > the link should not appeared. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35435) [FLIP-451] Introduce timeout configuration to AsyncSink
[ https://issues.apache.org/jira/browse/FLINK-35435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35435: --- Labels: pull-request-available (was: ) > [FLIP-451] Introduce timeout configuration to AsyncSink > --- > > Key: FLINK-35435 > URL: https://issues.apache.org/jira/browse/FLINK-35435 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Reporter: Ahmed Hamdy >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: Screenshot 2024-05-24 at 11.06.30.png, Screenshot > 2024-05-24 at 12.06.20.png > > > Implementation Ticket for: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34545) Add OceanBase pipeline connector to Flink CDC
[ https://issues.apache.org/jira/browse/FLINK-34545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34545: --- Labels: pull-request-available (was: ) > Add OceanBase pipeline connector to Flink CDC > - > > Key: FLINK-34545 > URL: https://issues.apache.org/jira/browse/FLINK-34545 > Project: Flink > Issue Type: New Feature > Components: Flink CDC >Reporter: He Wang >Priority: Major > Labels: pull-request-available > > Flink CDC supports end-to-end jobs starting from version 3.0.0, but now only > supports sink to Doris and StarRocks. OceanBase has made a lot of > improvements on AP recently, so supporting writing to OceanBase via Flink CDC > would be helpful to users. > OceanBase maintains a sink connector that supports multi-table sink and ddl > sink. We can use it to build the pipeline connector. > [https://github.com/oceanbase/flink-connector-oceanbase] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35430) ZoneId is not passed to DebeziumJsonSerializationSchema
[ https://issues.apache.org/jira/browse/FLINK-35430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35430: --- Labels: pull-request-available (was: ) > ZoneId is not passed to DebeziumJsonSerializationSchema > --- > > Key: FLINK-35430 > URL: https://issues.apache.org/jira/browse/FLINK-35430 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.2.0 >Reporter: LvYanquan >Priority: Not a Priority > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > ZoneId is used to convert TIMESTAMP_WITH_LOCAL_TIME_ZONE type to specific > time zone, but DebeziumJsonSerializationSchema does not used the user-defined > zoneid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35440) unable to connect tableau to jdbc flink url using flink sql driver
[
https://issues.apache.org/jira/browse/FLINK-35440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35440:
---
Labels: pull-request-available (was: )
> unable to connect tableau to jdbc flink url using flink sql driver
> --
>
> Key: FLINK-35440
> URL: https://issues.apache.org/jira/browse/FLINK-35440
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / JDBC
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Zach
>Priority: Minor
> Labels: pull-request-available
>
> Tableau 2023.1 using
> [https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-jdbc-driver-bundle/]
> version 1.19.0 yields the following error when a connection is established
> to a local flink sql cluster using the uri
> {{{}jdbc:{}}}{{{}[flink://localhost:8083]{}}}
> {{{"ts":"2024-05-23T14:21:05.858","pid":12172,"tid":"6a70","sev":"error","req":"-","sess":"-","site":"-","user":"-","k":"jdbc-error","e":\{"excp-error-code":"0xFAB9A2C5","excp-source":"NeedsClassification","excp-status-code":"UNKNOWN"},"v":\{"context":"GrpcProtocolProxy::IsConnected
>
> (D:\\tc\\work\\t231\\g_pc\\modules\\connectors\\tabmixins\\main\\db\\GrpcProtocolProxy.cpp:456)","driver-name":"org.apache.flink.table.jdbc.FlinkDriver","driver-version":"1.19.0","error-code":"0","error-messages":["FlinkConnection#isValid
> is not supported
> yet."],"grpc-status-code":"2","protocol-id":3,"sql-state":"0"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35326) Implement lineage interface for hive connector
[ https://issues.apache.org/jira/browse/FLINK-35326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35326: --- Labels: pull-request-available (was: ) > Implement lineage interface for hive connector > -- > > Key: FLINK-35326 > URL: https://issues.apache.org/jira/browse/FLINK-35326 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.20.0 >Reporter: Zhenqiu Huang >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34877) [Feature][Pipeline] Flink CDC pipeline transform supports type conversion
[ https://issues.apache.org/jira/browse/FLINK-34877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34877: --- Labels: github-import pull-request-available (was: github-import) > [Feature][Pipeline] Flink CDC pipeline transform supports type conversion > - > > Key: FLINK-34877 > URL: https://issues.apache.org/jira/browse/FLINK-34877 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Flink CDC Issue Import >Priority: Major > Labels: github-import, pull-request-available > > ### Search before asking > - [X] I searched in the > [issues|https://github.com/ververica/flink-cdc-connectors/issues] and found > nothing similar. > ### Motivation > To be supplemented. > ### Solution > To be supplemented. > ### Alternatives > None. > ### Anything else? > To be supplemented. > ### Are you willing to submit a PR? > - [X] I'm willing to submit a PR! > Imported from GitHub > Url: https://github.com/apache/flink-cdc/issues/3078 > Created by: [aiwenmo|https://github.com/aiwenmo] > Labels: enhancement, > Created at: Mon Feb 26 23:47:03 CST 2024 > State: open -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35437) BlockStatementGrouper uses lots of memory
[
https://issues.apache.org/jira/browse/FLINK-35437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35437:
---
Labels: pull-request-available (was: )
> BlockStatementGrouper uses lots of memory
> -
>
> Key: FLINK-35437
> URL: https://issues.apache.org/jira/browse/FLINK-35437
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.19.0
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> For deeply nested {{if else}} statements {{BlockStatementGrouper}} uses loads
> of memory and fails with OOM quickly.
> When running JMs with around 400mb a query like:
> {code}
> select case when orderid = 0 then 1 when orderid = 1 then 2 when orderid
> = 2 then 3 when orderid = 3 then 4 when orderid = 4 then 5 when orderid =
> 5 then
> 6 when orderid = 6 then 7 when orderid = 7 then 8 when orderid = 8 then 9
> when
> orderid = 9 then 10 when orderid = 10 then 11 when orderid = 11 then 12
> when orderid
> = 12 then 13 when orderid = 13 then 14 when orderid = 14 then 15 when
> orderid
> = 15 then 16 when orderid = 16 then 17 when orderid = 17 then 18 when
> orderid
> = 18 then 19 when orderid = 19 then 20 when orderid = 20 then 21 when
> orderid
> = 21 then 22 when orderid = 22 then 23 when orderid = 23 then 24 when
> orderid
> = 24 then 25 when orderid = 25 then 26 when orderid = 26 then 27 when
> orderid
> = 27 then 28 when orderid = 28 then 29 when orderid = 29 then 30 when
> orderid
> = 30 then 31 when orderid = 31 then 32 when orderid = 32 then 33 when
> orderid
> = 33 then 34 when orderid = 34 then 35 when orderid = 35 then 36 when
> orderid
> = 36 then 37 when orderid = 37 then 38 when orderid = 38 then 39 when
> orderid
> = 39 then 40 when orderid = 40 then 41 when orderid = 41 then 42 when
> orderid
> = 42 then 43 when orderid = 43 then 44 when orderid = 44 then 45 when
> orderid
> = 45 then 46 when orderid = 46 then 47 when orderid = 47 then 48 when
> orderid
> = 48 then 49 when orderid = 49 then 50 when orderid = 50 then 51 when
> orderid
> = 51 then 52 when orderid = 52 then 53 when orderid = 53 then 54 when
> orderid
> = 54 then 55 when orderid = 55 then 56 when orderid = 56 then 57 when
> orderid
> = 57 then 58 when orderid = 58 then 59 when orderid = 59 then 60 when
> orderid
> = 60 then 61 when orderid = 61 then 62 when orderid = 62 then 63 when
> orderid
> = 63 then 64 when orderid = 64 then 65 when orderid = 65 then 66 when
> orderid
> = 66 then 67 when orderid = 67 then 68 when orderid = 68 then 69 when
> orderid
> = 69 then 70 when orderid = 70 then 71 when orderid = 71 then 72 when
> orderid
> = 72 then 73 when orderid = 73 then 74 when orderid = 74 then 75 when
> orderid
> = 75 then 76 when orderid = 76 then 77 when orderid = 77 then 78 when
> orderid
> = 78 then 79 when orderid = 79 then 80 when orderid = 80 then 81 when
> orderid
> = 81 then 82 when orderid = 82 then 83 when orderid = 83 then 84 when
> orderid
> = 84 then 85 when orderid = 85 then 86 when orderid = 86 then 87 when
> orderid
> = 87 then 88 when orderid = 88 then 89 when orderid = 89 then 90 when
> orderid
> = 90 then 91 when orderid = 91 then 92 when orderid = 92 then 93 when
> orderid
> = 93 then 94 when orderid = 94 then 95 when orderid = 95 then 96 when
> orderid
> = 96 then 97 when orderid = 97 then 98 when orderid = 98 then 99 when
> orderid
> = 99 then 100 when orderid = 100 then 101 when orderid = 101 then 102
> when orderid
> = 102 then 103 when orderid = 103 then 104 when orderid = 104 then 105
> when orderid
> = 105 then 106 when orderid = 106 then 107 when orderid = 107 then 108
> when orderid
> = 108 then 109 when orderid = 109 then 110 when orderid = 110 then 111
> when orderid
> = 111 then 112 when orderid = 112 then 113 when orderid = 113 then 114
> when orderid
> = 114 then 115 when orderid = 115 then 116 when orderid = 116 then 117
> when orderid
> = 117 then 118 when orderid = 118 then 119 when orderid = 119 then 120
> when orderid
> = 120 then 121 when orderid = 121 then 122 when orderid = 122 then 123
> when orderid
> = 123 then 124 when orderid = 124 then 125 when orderid = 125 then 126
> when orderid
> = 126 then 127 when orderid = 127 then 128 when orderid = 128 then 129
> when orderid
> = 129 then 130 when orderid = 130 then 131 when orderid = 131 then 132
> when orderid
> = 132 then 133 when orderid = 133 then 134 when orderid = 134 then 135
> when orderid
> = 135 then 136 when orderid = 136 then 137 when orderid
[jira] [Updated] (FLINK-35436) Job can't launch when setting the option schema.change.behavior to IGNORE
[
https://issues.apache.org/jira/browse/FLINK-35436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35436:
---
Labels: pull-request-available (was: )
> Job can't launch when setting the option schema.change.behavior to IGNORE
> -
>
> Key: FLINK-35436
> URL: https://issues.apache.org/jira/browse/FLINK-35436
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: hk__lrzy
>Priority: Major
> Labels: pull-request-available
>
> Now in the 3.0 pipeline, *SchemaOperator* already was necessary operator in
> the Flink DAG, both *PrePartitionOperator* and *DataSinkWriterOperator* have
> connection with the *SchemaRegister* according the *schemaEvolutionClient,*
> but when we set the option schema.change.behavior to ignore or exception, the
> pipeline will add a filter operator instead of the *Schema Operator,* final
> cause the job fail.
> I think we still need keep the option for the schema.change.behavior to meet
> the difference cases, so i advice to move schema.change.behavior to the
> *SchemaRegister* to let *SchemaOperator* will be always in the DAG, and let
> to *SchemaRegister* decided to apply the schema change or not.{*}{*}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35429) We don't need introduce getFlinkConfigurationOptions for SqlGatewayRestEndpointFactory#Context
[ https://issues.apache.org/jira/browse/FLINK-35429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35429: --- Labels: pull-request-available (was: ) > We don't need introduce getFlinkConfigurationOptions for > SqlGatewayRestEndpointFactory#Context > -- > > Key: FLINK-35429 > URL: https://issues.apache.org/jira/browse/FLINK-35429 > Project: Flink > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Major > Labels: pull-request-available > > We don't need this method, as ReadableConfig has a toMap method now. > This will fix the compile error in 1.19 > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59754=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=13638. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35434) Support pass exception in StateExecutor to runtime
[ https://issues.apache.org/jira/browse/FLINK-35434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35434: --- Labels: pull-request-available (was: ) > Support pass exception in StateExecutor to runtime > -- > > Key: FLINK-35434 > URL: https://issues.apache.org/jira/browse/FLINK-35434 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Feifan Wang >Priority: Major > Labels: pull-request-available > > Exception may thrown when _StateExecutor_ execute the state request , such as > a IOException. We should pass the exception to runtime then failed the job in > this situation. > > _InternalStateFuture#completeExceptionally()_ will be added as [discussion > here|https://github.com/apache/flink/pull/24739#discussion_r1590633134]. > And then, _ForStWriteBatchOperation_ and _ForStGeneralMultiGetOperation_ > will call this method when exception occurred. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35412) Batch execution of async state request callback
[ https://issues.apache.org/jira/browse/FLINK-35412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35412: --- Labels: pull-request-available (was: ) > Batch execution of async state request callback > --- > > Key: FLINK-35412 > URL: https://issues.apache.org/jira/browse/FLINK-35412 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends, Runtime / Task >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > > There is one mail for each callback when async state result returns. One > possible optimization is to encapsulate multiple callbacks into one mail. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35432) Support catch modify-event for mysql-cdc
[
https://issues.apache.org/jira/browse/FLINK-35432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35432:
---
Labels: pull-request-available (was: )
> Support catch modify-event for mysql-cdc
>
>
> Key: FLINK-35432
> URL: https://issues.apache.org/jira/browse/FLINK-35432
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: hk__lrzy
>Priority: Major
> Labels: pull-request-available
>
> Now users common use sql like to modify the column type in MySQL.
> {code:java}
> Alter table MODIFY COLUMN `new_name` new_type{code}
>
> Flink-CDC use *CustomAlterTableParserListener* to parse the ddl and wrap it
> as ChangeEevent now. But i noticed that *CustomAlterTableParserListener* not
> implement the method *enterAlterByModifyColumn* and
> {*}exitAlterByModifyColumn{*}, and it will cause we can't received the
> *AlterColumnTypeEvent* now
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35431) Migrate references in Flink CDC documentation from Debezium 1.9 to 2.0
[ https://issues.apache.org/jira/browse/FLINK-35431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35431: --- Labels: pull-request-available (was: ) > Migrate references in Flink CDC documentation from Debezium 1.9 to 2.0 > -- > > Key: FLINK-35431 > URL: https://issues.apache.org/jira/browse/FLINK-35431 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.2.0 >Reporter: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > > Backgroud: > The website has taken down the documentation for versions before 2.0. And > flink cdc document ci failed. > Discuss link: https://lists.apache.org/thread/twdpd4bvoxtmjd0tykk7hs73hx4rm6yv -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35410) Avoid sync waiting in coordinator thread of ForSt executor
[ https://issues.apache.org/jira/browse/FLINK-35410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35410: --- Labels: pull-request-available (was: ) > Avoid sync waiting in coordinator thread of ForSt executor > -- > > Key: FLINK-35410 > URL: https://issues.apache.org/jira/browse/FLINK-35410 > Project: Flink > Issue Type: Sub-task >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > > Currently, the coordinator thread of ForSt executor will sync wait the state > access result, which can be optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35426) Change the distribution of DynamicFilteringDataCollector to Broadcast
[ https://issues.apache.org/jira/browse/FLINK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35426: --- Labels: pull-request-available (was: ) > Change the distribution of DynamicFilteringDataCollector to Broadcast > - > > Key: FLINK-35426 > URL: https://issues.apache.org/jira/browse/FLINK-35426 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.20.0 >Reporter: xingbe >Assignee: xingbe >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > Currently, the DynamicFilteringDataCollector is utilized in the dynamic > partition pruning feature of batch jobs to collect the partition information > dynamically filtered by the source. Its current data distribution method is > rebalance, and it also acts as an upstream vertex to the probe side Source. > Presently, when the Scheduler dynamically infers the parallelism for vertices > that are both downstream and Source, it considers factors from both sides, > which can lead to an overestimation of parallelism due to > DynamicFilteringDataCollector being an upstream of the Source. We aim to > change the distribution method of the DynamicFilteringDataCollector to > broadcast to prevent the dynamic overestimation of Source parallelism. > Furthermore, given that the DynamicFilteringDataCollector transmits data > through the OperatorCoordinator rather than through normal data distribution, > this change will not affect the DPP (Dynamic Partition Pruning) functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35421) Schema Operator blocking forever when Akka Rpc timeout
[ https://issues.apache.org/jira/browse/FLINK-35421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35421: --- Labels: pull-request-available (was: ) > Schema Operator blocking forever when Akka Rpc timeout > -- > > Key: FLINK-35421 > URL: https://issues.apache.org/jira/browse/FLINK-35421 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: hk__lrzy >Priority: Major > Labels: pull-request-available > > When SchemaOperator restart and there have no checkpoint before, > SchemaOperator will not send `RefreshPendingListsRequest` to coordinator, and > if coordinator have pending schema events and SchemaOperator will block > forever. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35129) Postgres source commits the offset after every multiple checkpoint cycles.
[ https://issues.apache.org/jira/browse/FLINK-35129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35129: --- Labels: pull-request-available (was: ) > Postgres source commits the offset after every multiple checkpoint cycles. > -- > > Key: FLINK-35129 > URL: https://issues.apache.org/jira/browse/FLINK-35129 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Hongshun Wang >Assignee: Muhammet Orazov >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > After entering the Stream phase, the offset consumed by the global slot is > committed upon the completion of each checkpoint, preventing log files from > being unable to be recycled continuously, which could lead to insufficient > disk space. > However, the job can only restart from the latest checkpoint or savepoint. if > restored from an earlier state, WAL may already have been recycled. > > The way to solve it is to commit the offset after every multiple checkpoint > cycles. The number of checkpoint cycles is determine by connector option, and > the default value is 3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35423) ARRAY_EXCEPT should support set semantics
[
https://issues.apache.org/jira/browse/FLINK-35423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35423:
---
Labels: pull-request-available (was: )
> ARRAY_EXCEPT should support set semantics
> -
>
> Key: FLINK-35423
> URL: https://issues.apache.org/jira/browse/FLINK-35423
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Sergey Nuyanzin
>Priority: Blocker
> Labels: pull-request-available
>
> After a number of discussions e.g. here [1]
> It was decided to follow set semantics for {{ARRAY_EXCEPT}} and
> {{ARRAY_INTERSECT}}.
> It is marked as a blocker since {{ARRAY_EXCEPT}} was added in 1.20 only and
> has not been released yet, so the change should be done before 1.20.0 release
> to avoid inconsistencies.
> [1] https://github.com/apache/flink/pull/24526
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35420) WordCountMapredITCase fails to compile in IntelliJ
[
https://issues.apache.org/jira/browse/FLINK-35420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35420:
---
Labels: pull-request-available (was: )
> WordCountMapredITCase fails to compile in IntelliJ
> --
>
> Key: FLINK-35420
> URL: https://issues.apache.org/jira/browse/FLINK-35420
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hadoop Compatibility
>Affects Versions: 1.20.0
>Reporter: Piotr Nowojski
>Priority: Major
> Labels: pull-request-available
>
> {noformat}
> flink-connectors/flink-hadoop-compatibility/src/test/scala/org/apache/flink/api/hadoopcompatibility/scala/WordCountMapredITCase.scala:42:8
> value isFalse is not a member of ?0
> possible cause: maybe a semicolon is missing before `value isFalse'?
> .isFalse()
> {noformat}
> Might be caused by:
> https://youtrack.jetbrains.com/issue/SCL-20679
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35415) CDC Fails to create sink with Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35415: --- Labels: pull-request-available (was: ) > CDC Fails to create sink with Flink 1.19 > > > Key: FLINK-35415 > URL: https://issues.apache.org/jira/browse/FLINK-35415 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: yux >Assignee: yux >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > Currently, Flink CDC doesn't work with Flink 1.19 with the following > exception: > Exception in thread "main" java.lang.NoSuchMethodError: 'void > org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory.(org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink, > boolean, boolean)' > The reason is Flink CDC uses Flink @Internal API and it was changed in 1.19 > update. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34673) SessionRelatedITCase#testTouchSession failure on GitHub Actions
[
https://issues.apache.org/jira/browse/FLINK-34673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-34673:
---
Labels: pull-request-available starter test-stability (was: starter
test-stability)
> SessionRelatedITCase#testTouchSession failure on GitHub Actions
> ---
>
> Key: FLINK-34673
> URL: https://issues.apache.org/jira/browse/FLINK-34673
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.19.0
>Reporter: Ryan Skraba
>Priority: Major
> Labels: pull-request-available, starter, test-stability
>
> [https://github.com/apache/flink/actions/runs/8258416388/job/22590907051#step:10:12155]
> {code:java}
> Error: 03:08:21 03:08:21.304 [ERROR]
> org.apache.flink.table.gateway.rest.SessionRelatedITCase.testTouchSession --
> Time elapsed: 0.015 s <<< FAILURE!
> Mar 13 03:08:21 java.lang.AssertionError:
> Mar 13 03:08:21
> Mar 13 03:08:21 Expecting actual:
> Mar 13 03:08:21 1710299301198L
> Mar 13 03:08:21 to be greater than:
> Mar 13 03:08:21 1710299301198L
> Mar 13 03:08:21
> Mar 13 03:08:21 at
> org.apache.flink.table.gateway.rest.SessionRelatedITCase.testTouchSession(SessionRelatedITCase.java:175)
> Mar 13 03:08:21 at
> java.base/java.lang.reflect.Method.invoke(Method.java:580)
> Mar 13 03:08:21 at
> java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
> Mar 13 03:08:21 at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> Mar 13 03:08:21 at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> Mar 13 03:08:21 at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> Mar 13 03:08:21 at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> Mar 13 03:08:21 at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35143) Expose newly added tables capture in mysql pipeline connector
[ https://issues.apache.org/jira/browse/FLINK-35143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35143: --- Labels: pull-request-available (was: ) > Expose newly added tables capture in mysql pipeline connector > - > > Key: FLINK-35143 > URL: https://issues.apache.org/jira/browse/FLINK-35143 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Hongshun Wang >Assignee: Hongshun Wang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > Currently, mysql pipeline connector still don't allowed to capture newly > added tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35416) Weekly CI for ElasticSearch connector failed to compile
[ https://issues.apache.org/jira/browse/FLINK-35416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35416: --- Labels: pull-request-available (was: ) > Weekly CI for ElasticSearch connector failed to compile > --- > > Key: FLINK-35416 > URL: https://issues.apache.org/jira/browse/FLINK-35416 > Project: Flink > Issue Type: Bug > Components: Connectors / ElasticSearch >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Major > Labels: pull-request-available > > ElasticsearchSinkBaseITCase.java:[31,65] package > org.apache.flink.shaded.guava30.com.google.common.collect does not exist -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33461) Support streaming related semantics for the new JDBC source
[ https://issues.apache.org/jira/browse/FLINK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33461: --- Labels: pull-request-available (was: ) > Support streaming related semantics for the new JDBC source > --- > > Key: FLINK-33461 > URL: https://issues.apache.org/jira/browse/FLINK-33461 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35347) Implement InMemory workflow scheduler service and plugin to support materialized table
[ https://issues.apache.org/jira/browse/FLINK-35347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35347: --- Labels: pull-request-available (was: ) > Implement InMemory workflow scheduler service and plugin to support > materialized table > -- > > Key: FLINK-35347 > URL: https://issues.apache.org/jira/browse/FLINK-35347 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Gateway >Affects Versions: 1.20.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20402) Migrate test_tpch.sh
[ https://issues.apache.org/jira/browse/FLINK-20402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20402: --- Labels: pull-request-available (was: ) > Migrate test_tpch.sh > > > Key: FLINK-20402 > URL: https://issues.apache.org/jira/browse/FLINK-20402 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Ecosystem, Tests >Reporter: Jark Wu >Assignee: Muhammet Orazov >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35409) Request more splits if all splits are filtered from addSplits method
[ https://issues.apache.org/jira/browse/FLINK-35409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35409: --- Labels: pull-request-available (was: ) > Request more splits if all splits are filtered from addSplits method > > > Key: FLINK-35409 > URL: https://issues.apache.org/jira/browse/FLINK-35409 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xiao Huang >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35408) Add 30 min tolerance value when validating the time-zone setting
[ https://issues.apache.org/jira/browse/FLINK-35408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35408: --- Labels: pull-request-available (was: ) > Add 30 min tolerance value when validating the time-zone setting > > > Key: FLINK-35408 > URL: https://issues.apache.org/jira/browse/FLINK-35408 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xiao Huang >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35242) Add fine-grained schema evolution strategy
[ https://issues.apache.org/jira/browse/FLINK-35242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35242: --- Labels: pull-request-available (was: ) > Add fine-grained schema evolution strategy > -- > > Key: FLINK-35242 > URL: https://issues.apache.org/jira/browse/FLINK-35242 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: yux >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > Original Estimate: 336h > Remaining Estimate: 336h > > Currently, Flink CDC allows three SE strategy: evolving it, ignoring it, or > throwing an exception. However such configuration strategy doesn't cover all > user cases and requires want more fine-grained strategy configuration. > This ticket suggests adding one more strategy "try_evolve" or > "evolve_when_available". It's basically like "evolving" option, but doesn't > throw an exception if such operation fails, which provides more flexibility. > Also, this ticket suggests allowing user to configure per-schema-event > strategy, so users could evolve some types of event (like rename column) and > reject some dangerous events (like truncate table, remove column). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35406) Use inner serializer when casting RAW type to BINARY or STRING in cast rules
[ https://issues.apache.org/jira/browse/FLINK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35406: --- Labels: pull-request-available (was: ) > Use inner serializer when casting RAW type to BINARY or STRING in cast rules > > > Key: FLINK-35406 > URL: https://issues.apache.org/jira/browse/FLINK-35406 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Zhenghua Gao >Priority: Major > Labels: pull-request-available > > The generated code in RawToStringCastRule and RawToBinaryCastRule use > BinaryRawValueData::toBytes and BinaryRawValueData::toObject to convert > RawValueData(to java object or byte array), which should use inner serializer > instead of RawValueDataSerializer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35400) Rebuild FileMergingSnapshotManager in failover
[
https://issues.apache.org/jira/browse/FLINK-35400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35400:
---
Labels: pull-request-available (was: )
> Rebuild FileMergingSnapshotManager in failover
> --
>
> Key: FLINK-35400
> URL: https://issues.apache.org/jira/browse/FLINK-35400
> Project: Flink
> Issue Type: Sub-task
>Reporter: Zakelly Lan
>Priority: Major
> Labels: pull-request-available
>
> Currently, the {{FileMergingSnapshotManager}} is released within
> {{{}releaseJobResources{}}}, which will not be invoked during failover and
> restore. However, the manager should be created again to clear all internal
> states in a new job attempt.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35354: --- Labels: pull-request-available (was: ) > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode in tikv, we > need to do network mapping, such as `spark.tispark.host_mapping` in > [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I > think we need support `host_mapping` in our Flink tikv cdc connector. > > Add param: > tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35359) General Improvement to Configuration for Flink 2.0
[ https://issues.apache.org/jira/browse/FLINK-35359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35359: --- Labels: pull-request-available (was: ) > General Improvement to Configuration for Flink 2.0 > -- > > Key: FLINK-35359 > URL: https://issues.apache.org/jira/browse/FLINK-35359 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Xuannan Su >Priority: Major > Labels: pull-request-available > > As Flink moves toward version 2.0, we want to provide users with a better > experience with the existing configuration. In this FLIP, we outline several > general improvements to the current configuration: > * Ensure all the ConfigOptions are properly annotated > * Ensure all user-facing configurations are included in the documentation > generation process > * Make the existing ConfigOptions use the proper type > * Mark all internally used ConfigOptions with the @Internal annotation > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35403) FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding
[ https://issues.apache.org/jira/browse/FLINK-35403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35403: --- Labels: pull-request-available (was: ) > FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding > > > Key: FLINK-35403 > URL: https://issues.apache.org/jira/browse/FLINK-35403 > Project: Flink > Issue Type: Improvement >Reporter: Alan Sheinberg >Assignee: Alan Sheinberg >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35402) [flink-operator][Deployment] add labels to metadata
[ https://issues.apache.org/jira/browse/FLINK-35402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35402: --- Labels: pull-request-available (was: ) > [flink-operator][Deployment] add labels to metadata > --- > > Key: FLINK-35402 > URL: https://issues.apache.org/jira/browse/FLINK-35402 > Project: Flink > Issue Type: Improvement > Components: flink-contrib >Reporter: Luís Costa >Priority: Minor > Labels: pull-request-available > > Greetings dear team, > I would like to add labels to flink-operator Deployment metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35382) ChangelogCompatibilityITCase.testRestore fails with an NPE
[
https://issues.apache.org/jira/browse/FLINK-35382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35382:
---
Labels: pull-request-available test-stability (was: test-stability)
> ChangelogCompatibilityITCase.testRestore fails with an NPE
> --
>
> Key: FLINK-35382
> URL: https://issues.apache.org/jira/browse/FLINK-35382
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> * 1.20 Java 8 / Test (module: tests)
> https://github.com/apache/flink/actions/runs/9110398985/job/25045798401#step:10:8192
> It looks like there can be a [NullPointerException at this
> line|https://github.com/apache/flink/blob/9a5a99b1a30054268bbde36d565cbb1b81018890/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/FileMergingSnapshotManagerBase.java#L666]
> causing a test failure:
> {code}
> Error: 10:36:23 10:36:23.312 [ERROR] Tests run: 9, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 19.31 s <<< FAILURE! -- in
> org.apache.flink.test.state.ChangelogCompatibilityITCase
> Error: 10:36:23 10:36:23.313 [ERROR]
> org.apache.flink.test.state.ChangelogCompatibilityITCase.testRestore[startWithChangelog=false,
> restoreWithChangelog=true, restoreFrom=CHECKPOINT, allowStore=true,
> allowRestore=true] -- Time elapsed: 1.492 s <<< ERROR!
> May 16 10:36:23 java.lang.RuntimeException:
> org.opentest4j.AssertionFailedError: Graph is in globally terminal state
> (FAILED)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.tryRun(ChangelogCompatibilityITCase.java:204)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.restoreAndValidate(ChangelogCompatibilityITCase.java:190)
> May 16 10:36:23 at java.util.Optional.ifPresent(Optional.java:159)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.testRestore(ChangelogCompatibilityITCase.java:118)
> May 16 10:36:23 at java.lang.reflect.Method.invoke(Method.java:498)
> May 16 10:36:23 Caused by: org.opentest4j.AssertionFailedError: Graph is in
> globally terminal state (FAILED)
> May 16 10:36:23 at
> org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:42)
> May 16 10:36:23 at
> org.junit.jupiter.api.Assertions.fail(Assertions.java:150)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.lambda$waitForAllTaskRunning$3(CommonTestUtils.java:214)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:151)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:209)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:182)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.submit(ChangelogCompatibilityITCase.java:284)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.tryRun(ChangelogCompatibilityITCase.java:197)
> May 16 10:36:23 ... 4 more
> May 16 10:36:23 Caused by: org.apache.flink.runtime.JobException:
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> May 16 10:36:23 at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:219)
> May 16 10:36:23 at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailureAndReport(ExecutionFailureHandler.java:166)
> May 16 10:36:23 at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:121)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:279)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:270)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:263)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:788)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:765)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
> May 16 10:36:23 at
>
[jira] [Updated] (FLINK-35391) Bump dependency of Paimon Pipeline connector to 0.8.0
[ https://issues.apache.org/jira/browse/FLINK-35391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35391: --- Labels: pull-request-available (was: ) > Bump dependency of Paimon Pipeline connector to 0.8.0 > - > > Key: FLINK-35391 > URL: https://issues.apache.org/jira/browse/FLINK-35391 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.2.0 >Reporter: LvYanquan >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > Apache Paimon has [released > 0.8|https://paimon.apache.org/releases/release-0.8/] recently, We can update > dependencies to use new features. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35049) Implement Map Async State API for ForStStateBackend
[ https://issues.apache.org/jira/browse/FLINK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35049: --- Labels: pull-request-available (was: ) > Implement Map Async State API for ForStStateBackend > --- > > Key: FLINK-35049 > URL: https://issues.apache.org/jira/browse/FLINK-35049 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Hangxiang Yu >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35355) Async aggregating state
[ https://issues.apache.org/jira/browse/FLINK-35355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35355: --- Labels: pull-request-available (was: ) > Async aggregating state > --- > > Key: FLINK-35355 > URL: https://issues.apache.org/jira/browse/FLINK-35355 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Zakelly Lan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35395) Fix KeyedStateStore class annotation error
[ https://issues.apache.org/jira/browse/FLINK-35395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35395: --- Labels: pull-request-available (was: ) > Fix KeyedStateStore class annotation error > -- > > Key: FLINK-35395 > URL: https://issues.apache.org/jira/browse/FLINK-35395 > Project: Flink > Issue Type: Improvement > Components: API / Core >Reporter: bradley >Priority: Major > Labels: pull-request-available > > The KeyedStateStore class is annotated incorrectly, and the examples there > are obviously wrong and will mislead users. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32088) Re-uploading in state file-merging for space amplification control
[ https://issues.apache.org/jira/browse/FLINK-32088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32088: --- Labels: pull-request-available (was: ) > Re-uploading in state file-merging for space amplification control > -- > > Key: FLINK-32088 > URL: https://issues.apache.org/jira/browse/FLINK-32088 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35379) File merging manager is not properly notified about checkpoint
[
https://issues.apache.org/jira/browse/FLINK-35379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35379:
---
Labels: pull-request-available (was: )
> File merging manager is not properly notified about checkpoint
> --
>
> Key: FLINK-35379
> URL: https://issues.apache.org/jira/browse/FLINK-35379
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Zakelly Lan
>Assignee: Zakelly Lan
>Priority: Major
> Labels: pull-request-available
>
> Currently, the \{{FileMergingSnapshotManager}} from checkpoint file merging
> mechanism is not properly notified about checkpoint, and it does not handle
> the notifications properly as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35386) Build Flink CDC 3.1 Documentation and mark it as stable
[ https://issues.apache.org/jira/browse/FLINK-35386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35386: --- Labels: pull-request-available (was: ) > Build Flink CDC 3.1 Documentation and mark it as stable > --- > > Key: FLINK-35386 > URL: https://issues.apache.org/jira/browse/FLINK-35386 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > This is part of Flink CDC 3.1.0 release. We need to update the Github Action > configuration to build release-3.1 doc and mark it as stable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35378) [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc
[ https://issues.apache.org/jira/browse/FLINK-35378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35378: --- Labels: pull-request-available (was: ) > [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc > --- > > Key: FLINK-35378 > URL: https://issues.apache.org/jira/browse/FLINK-35378 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > > https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=303794871=af4ace88-98b7-4a53-aece-cd67d2f91a15; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35383) Update compatibility matrix to include 1.19 release
[ https://issues.apache.org/jira/browse/FLINK-35383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35383: --- Labels: pull-request-available (was: ) > Update compatibility matrix to include 1.19 release > --- > > Key: FLINK-35383 > URL: https://issues.apache.org/jira/browse/FLINK-35383 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Aleksandr Pilipenko >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0, 1.19.1 > > > Update compatibility matrix in documentation to include Flink 1.19 release: > https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/ops/upgrading/#compatibility-table > https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/ops/upgrading/#%E5%85%BC%E5%AE%B9%E6%80%A7%E9%80%9F%E6%9F%A5%E8%A1%A8 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35331) Download links for binary releases are displayed as source releases on website
[ https://issues.apache.org/jira/browse/FLINK-35331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35331: --- Labels: pull-request-available (was: ) > Download links for binary releases are displayed as source releases on website > -- > > Key: FLINK-35331 > URL: https://issues.apache.org/jira/browse/FLINK-35331 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Xintong Song >Priority: Major > Labels: pull-request-available > > Take Pre-bundled Hadoop as examples. The content for downloading are binary > releases, while the link is displayed as "Pre-bundled Hadoop 2.x.y Source > Release (asc, sha512)". The problem is caused by misusing > `source_release_[url|asc_url|sha512_url]` for binary contents in the > corresponding [yaml > file.|https://github.com/apache/flink-web/blob/asf-site/docs/data/additional_components.yml] > There are many similar cases in the webpage. > And a relevant issues is that, some source releases are displayed as "XXX > Source Release Source Release", due to including "Source Release" in the > `name` field of the corresponding yaml file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35289) Incorrect timestamp of stream elements collected from onTimer in batch mode
[ https://issues.apache.org/jira/browse/FLINK-35289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35289: --- Labels: pull-request-available (was: ) > Incorrect timestamp of stream elements collected from onTimer in batch mode > --- > > Key: FLINK-35289 > URL: https://issues.apache.org/jira/browse/FLINK-35289 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.18.1 >Reporter: Kanthi Vaidya >Priority: Major > Labels: pull-request-available > > In batch mode all registered timers will fire at the _end of time. Given > this, if a user registers a timer for Long.MAX_VALUE, the timestamp assigned > to the elements that are collected from the onTimer context ends up being > Long.MAX_VALUE. Ideally this should be the time when the batch actually > executed the onTimer function._ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35342) MaterializedTableStatementITCase test can check for wrong status
[
https://issues.apache.org/jira/browse/FLINK-35342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35342:
---
Labels: pull-request-available test-stability (was: test-stability)
> MaterializedTableStatementITCase test can check for wrong status
>
>
> Key: FLINK-35342
> URL: https://issues.apache.org/jira/browse/FLINK-35342
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> * 1.20 AdaptiveScheduler / Test (module: table)
> https://github.com/apache/flink/actions/runs/9056197319/job/24879135605#step:10:12490
>
> It looks like
> {{MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume}}
> can be flaky, where the expected status is not yet RUNNING:
> {code}
> Error: 03:24:03 03:24:03.902 [ERROR] Tests run: 6, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 26.78 s <<< FAILURE! -- in
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase
> Error: 03:24:03 03:24:03.902 [ERROR]
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume(Path,
> RestClusterClient) -- Time elapsed: 3.850 s <<< FAILURE!
> May 13 03:24:03 org.opentest4j.AssertionFailedError:
> May 13 03:24:03
> May 13 03:24:03 expected: "RUNNING"
> May 13 03:24:03 but was: "CREATED"
> May 13 03:24:03 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> May 13 03:24:03 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> May 13 03:24:03 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> May 13 03:24:03 at
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume(MaterializedTableStatementITCase.java:650)
> May 13 03:24:03 at java.lang.reflect.Method.invoke(Method.java:498)
> May 13 03:24:03 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> May 13 03:24:03
> May 13 03:24:04 03:24:04.270 [INFO]
> May 13 03:24:04 03:24:04.270 [INFO] Results:
> May 13 03:24:04 03:24:04.270 [INFO]
> Error: 03:24:04 03:24:04.270 [ERROR] Failures:
> Error: 03:24:04 03:24:04.271 [ERROR]
> MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume:650
>
> May 13 03:24:04 expected: "RUNNING"
> May 13 03:24:04 but was: "CREATED"
> May 13 03:24:04 03:24:04.271 [INFO]
> Error: 03:24:04 03:24:04.271 [ERROR] Tests run: 82, Failures: 1, Errors: 0,
> Skipped: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35373) Simplify Flink CDC Paimon Sink web ui
[ https://issues.apache.org/jira/browse/FLINK-35373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35373: --- Labels: pull-request-available (was: ) > Simplify Flink CDC Paimon Sink web ui > -- > > Key: FLINK-35373 > URL: https://issues.apache.org/jira/browse/FLINK-35373 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.2.0 >Reporter: JunboWang >Priority: Not a Priority > Labels: pull-request-available > Attachments: image-2024-05-16-15-43-26-979.png > > > !image-2024-05-16-15-43-26-979.png! > Modify > "org.apache.flink.cdc.connectors.paimon.sink.v2.MultiTableCommittableChannelComputer" > to "shuffle by database & table", simplify web UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35356) Async reducing state
[ https://issues.apache.org/jira/browse/FLINK-35356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35356: --- Labels: pull-request-available (was: ) > Async reducing state > > > Key: FLINK-35356 > URL: https://issues.apache.org/jira/browse/FLINK-35356 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Zakelly Lan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35325) Paimon connector miss the position of AddColumnEvent
[ https://issues.apache.org/jira/browse/FLINK-35325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35325: --- Labels: pull-request-available (was: ) > Paimon connector miss the position of AddColumnEvent > > > Key: FLINK-35325 > URL: https://issues.apache.org/jira/browse/FLINK-35325 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.2.0 >Reporter: LvYanquan >Priority: Minor > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > Currently, new columns are always added in the last position, however some > newly add columns had a specific before and after relationship with other > column. > Source code: > [https://github.com/apache/flink-cdc/blob/fa6e7ea51258dcd90f06036196618224156df367/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplier.java#L137] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35226) Deprecate execution-mode in ExecutionConfigInfo related rest api
[ https://issues.apache.org/jira/browse/FLINK-35226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35226: --- Labels: pull-request-available (was: ) > Deprecate execution-mode in ExecutionConfigInfo related rest api > > > Key: FLINK-35226 > URL: https://issues.apache.org/jira/browse/FLINK-35226 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35225) Remove Execution mode in Flink WebUI
[ https://issues.apache.org/jira/browse/FLINK-35225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35225: --- Labels: pull-request-available (was: ) > Remove Execution mode in Flink WebUI > > > Key: FLINK-35225 > URL: https://issues.apache.org/jira/browse/FLINK-35225 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32828) Partition aware watermark not handled correctly shortly after job start up from checkpoint or savepoint
[
https://issues.apache.org/jira/browse/FLINK-32828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-32828:
---
Labels: pull-request-available (was: )
> Partition aware watermark not handled correctly shortly after job start up
> from checkpoint or savepoint
> ---
>
> Key: FLINK-32828
> URL: https://issues.apache.org/jira/browse/FLINK-32828
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Kafka
>Affects Versions: 1.17.1, 1.19.0, 1.18.1
> Environment: Affected environments:
> * Local MiniCluster + Confluent Kafka run in docker
> ** See attached files
> * Flink job run in Kubernetes using Flink Operator 1.15 + 3 node Kafka
> cluster run in Kubernetes cluster
>Reporter: Grzegorz Liter
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available
> Attachments: docker-compose.yml, test-job.java
>
>
> When using KafkaSource with partition aware watermarks. Watermarks are being
> emitted even when only one partition has some events just after job startup
> from savepoint/checkpoint. After it has some events on other partitions the
> watermark behaviour is correct and watermark is emited as a minimum watarmark
> from all partition.
>
> Steps to reproduce:
> # Setup a Kafka cluster with a topic that has 2 or more partitions. (see
> attached docker-compose.yml)
> #
> ## {{./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic
> test-2 --partitions 4}}
> # Create a job that (see attached `test-job.java`):
> ## uses a KafkaSource with
> `WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10L)`
> ## has parallelism lower than number of partitions
> ## stores checkpoint/savepoint
> # Start job
> # Send events only on single partition
> ## {{./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic
> test-2 --property "parse.key=true" --property "key.separator=:"}}
>
> {{14:51:19,883 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:18.849Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:32,484 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:31.475Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:35,914 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:34.909Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> Expected: Watermark does not progress. Actual: Watermark does not progress.
> 5. Stop the job
> 6. Startup job from last checkpoint/savepoint
> 7. Send events only on single partitions
> {{14:53:41,693 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:40.662Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:53:46,088 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:45.078Z, watermark
> 2023-08-10T12:53:30.661Z}}
> {{14:53:49,520 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:48.511Z, watermark
> 2023-08-10T12:53:35.077Z}}
> Expected: Watermark does not progress. {color:#ff}*Actual: Watermark has
> progress*{color}
>
> {color:#172b4d}To add bit more of context:{color}
> {color:#172b4d}8. Send events on other partitions and then send events only
> on single partitions{color}
> {{{color:#172b4d}14:54:55,112 WARN com.example.TestJob6$InputSink2
> [] - == Received: test-2/0: 2 -> a, timestamp
> 2023-08-10T12:54:54.104Z, watermark 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/1: 4 -> a, timestamp 2023-08-10T12:54:56.665Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/2: 5 -> a, timestamp 2023-08-10T12:54:57.554Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:55:12,821 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:11.814Z, watermark
> 2023-08-10T12:54:44.103Z
> 14:55:16,099 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:15.091Z, watermark
> 2023-08-10T12:54:44.103Z
> 14:55:19,122 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:18.114Z, watermark
> 2023-08-10T12:54:44.103Z{color}}}
> {color:#172b4d}Expected: Watermark
[jira] [Updated] (FLINK-35353) Translate "Profiler" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-35353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35353: --- Labels: pull-request-available (was: ) > Translate "Profiler" page into Chinese > --- > > Key: FLINK-35353 > URL: https://issues.apache.org/jira/browse/FLINK-35353 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.19.0 >Reporter: Juan Zifeng >Assignee: Juan Zifeng >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > The links are > https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/ops/debugging/profiler/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35358) Breaking change when loading artifacts
[
https://issues.apache.org/jira/browse/FLINK-35358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35358:
---
Labels: pull-request-available (was: )
> Breaking change when loading artifacts
> --
>
> Key: FLINK-35358
> URL: https://issues.apache.org/jira/browse/FLINK-35358
> Project: Flink
> Issue Type: New Feature
> Components: Client / Job Submission, flink-docker
>Affects Versions: 1.19.0
>Reporter: Rasmus Thygesen
>Priority: Not a Priority
> Labels: pull-request-available
>
> We have been using the following code snippet in our Dockerfiles for running
> a Flink job in application mode
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar
> /opt/flink/usrlib/artifacts/my-job.jar
> USER flink {code}
>
> Which has been working since at least around Flink 1.14, but the 1.19 update
> has broken our Dockerfiles. The fix is to put the jar file a step further out
> so the code snippet becomes
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
> USER flink {code}
>
> We have not spent too much time looking into what the cause is, but we get
> the stack trace
>
> {code:java}
> myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load
> the provided entrypoint class.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
> [flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The program's
> entry point class 'my.company.job.MyJob' was not found in the jar file.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | ... 4 more
> myjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
> my.company.job.MyJob
> myjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown
> Source) ~[?:?]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.Class.forName0(Native Method) ~[?:?]
> myjob-jobmanager-1 | at java.lang.Class.forName(Unknown Source) ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:479)
> ~[flink-dist-1.19.0.jar:1.19.0]
>
[jira] [Updated] (FLINK-35361) Delete Flinkhistory files that failed to write to the local directory
[ https://issues.apache.org/jira/browse/FLINK-35361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35361: --- Labels: pull-request-available (was: ) > Delete Flinkhistory files that failed to write to the local directory > - > > Key: FLINK-35361 > URL: https://issues.apache.org/jira/browse/FLINK-35361 > Project: Flink > Issue Type: Bug >Affects Versions: 1.17.0, 1.18.0, 1.19.0, 1.20.0 >Reporter: dengxaing >Priority: Major > Labels: pull-request-available > Attachments: image-2024-05-15-21-15-54-973.png > > > I found a bug in > org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandler > !image-2024-05-15-21-15-54-973.png! > > When the local directory is full, the above code will create an empty or > incomplete file in local. At this point, the Flink History webui page cannot > be open or display abnormally. > However, when the local directory is expanded, the Flink History webui page > will not return to normal because new files will not be regenerated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32092) Integrate snapshot file-merging with existing IT cases
[ https://issues.apache.org/jira/browse/FLINK-32092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32092: --- Labels: pull-request-available (was: ) > Integrate snapshot file-merging with existing IT cases > -- > > Key: FLINK-32092 > URL: https://issues.apache.org/jira/browse/FLINK-32092 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35346) Introduce pluggable workflow scheduler interface for materialized table
[ https://issues.apache.org/jira/browse/FLINK-35346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35346: --- Labels: pull-request-available (was: ) > Introduce pluggable workflow scheduler interface for materialized table > --- > > Key: FLINK-35346 > URL: https://issues.apache.org/jira/browse/FLINK-35346 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.20.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35318) incorrect timezone handling for TIMESTAMP_WITH_LOCAL_TIME_ZONE type during predicate pushdown
[
https://issues.apache.org/jira/browse/FLINK-35318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35318:
---
Labels: pull-request-available (was: )
> incorrect timezone handling for TIMESTAMP_WITH_LOCAL_TIME_ZONE type during
> predicate pushdown
> -
>
> Key: FLINK-35318
> URL: https://issues.apache.org/jira/browse/FLINK-35318
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.18.1
> Environment: flink version 1.18.1
> iceberg version 1.15.1
>Reporter: linshangquan
>Assignee: linshangquan
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-05-09-14-06-58-007.png,
> image-2024-05-09-14-09-38-453.png, image-2024-05-09-14-11-38-476.png,
> image-2024-05-09-14-22-14-417.png, image-2024-05-09-14-22-59-370.png,
> image-2024-05-09-18-52-03-741.png, image-2024-05-09-18-52-28-584.png
>
>
> In our scenario, we have an Iceberg table that contains a column named 'time'
> of the {{timestamptz}} data type. This column has 10 rows of data where the
> 'time' value is {{'2024-04-30 07:00:00'}} expressed in the "Asia/Shanghai"
> timezone.
> !image-2024-05-09-14-06-58-007.png!
>
> We encountered a strange phenomenon when accessing the table using
> Iceberg-flink.
> When the {{WHERE}} clause includes the {{time}} column, the results are
> incorrect.
> ZoneId.{_}systemDefault{_}() = "Asia/Shanghai"
> !image-2024-05-09-18-52-03-741.png!
> When there is no {{WHERE}} clause, the results are correct.
> !image-2024-05-09-18-52-28-584.png!
> During debugging, we found that when a {{WHERE}} clause is present, a
> {{FilterPushDownSpec}} is generated, and this {{FilterPushDownSpec}} utilizes
> {{RexNodeToExpressionConverter}} for translation.
> !image-2024-05-09-14-11-38-476.png!
> !image-2024-05-09-14-22-59-370.png!
> When {{RexNodeToExpressionConverter#visitLiteral}} encounters a
> {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type, it uses the specified timezone
> "Asia/Shanghai" to convert the {{TimestampString}} type to an {{Instant}}
> type. However, the upstream {{TimestampString}} data has already been
> processed in UTC timezone. By applying the local timezone processing here, an
> error occurs due to the mismatch in timezones.
> Whether the handling of {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type of data in
> {{RexNodeToExpressionConverter#visitLiteral}} is a bug, and whether it should
> process the data in UTC timezone.
>
> Please help confirm if this is the issue, and if so, we can submit a patch to
> fix it.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35333) JdbcXaSinkTestBase fails in weekly Flink JDBC Connector tests
[
https://issues.apache.org/jira/browse/FLINK-35333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35333:
---
Labels: pull-request-available test-stability (was: test-stability)
> JdbcXaSinkTestBase fails in weekly Flink JDBC Connector tests
> -
>
> Key: FLINK-35333
> URL: https://issues.apache.org/jira/browse/FLINK-35333
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: jdbc-3.2.0
>Reporter: Martijn Visser
>Priority: Blocker
> Labels: pull-request-available, test-stability
>
> https://github.com/apache/flink-connector-jdbc/actions/runs/9047366679/job/24859224407#step:15:147
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile
> (default-testCompile) on project flink-connector-jdbc: Compilation failure
> Error:
> /home/runner/work/flink-connector-jdbc/flink-connector-jdbc/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/xa/JdbcXaSinkTestBase.java:[164,37]
> is not
> abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.common.functions.RuntimeContext
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24298) Refactor Google PubSub sink to use Unified Sink API
[ https://issues.apache.org/jira/browse/FLINK-24298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-24298: --- Labels: pull-request-available (was: ) > Refactor Google PubSub sink to use Unified Sink API > --- > > Key: FLINK-24298 > URL: https://issues.apache.org/jira/browse/FLINK-24298 > Project: Flink > Issue Type: Sub-task >Reporter: Martijn Visser >Priority: Major > Labels: pull-request-available > > Refactor Google PubSub source to use Unified Sink API > [FLIP-143|https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34931) Update Kudu connector DataStream Sink implementation
[ https://issues.apache.org/jira/browse/FLINK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34931: --- Labels: pull-request-available (was: ) > Update Kudu connector DataStream Sink implementation > > > Key: FLINK-34931 > URL: https://issues.apache.org/jira/browse/FLINK-34931 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kudu >Reporter: Ferenc Csaky >Assignee: Ferenc Csaky >Priority: Major > Labels: pull-request-available > > Update the DataSource API classes to use the current interfaces. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35109) Add support for Flink 1.20-SNAPSHOT in Flink Kafka connector and drop support for 1.17 and 1.18
[
https://issues.apache.org/jira/browse/FLINK-35109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35109:
---
Labels: pull-request-available (was: )
> Add support for Flink 1.20-SNAPSHOT in Flink Kafka connector and drop support
> for 1.17 and 1.18
> ---
>
> Key: FLINK-35109
> URL: https://issues.apache.org/jira/browse/FLINK-35109
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Kafka
>Reporter: Martijn Visser
>Assignee: Hang Ruan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: kafka-4.0.0
>
>
> The Flink Kafka connector currently can't compile against Flink
> 1.20-SNAPSHOT. An example failure can be found at
> https://github.com/apache/flink-connector-kafka/actions/runs/8659822490/job/23746484721#step:15:169
> The {code:java} TypeSerializerUpgradeTestBase{code} has had issues before,
> see FLINK-32455. See also specifically the comment in
> https://issues.apache.org/jira/browse/FLINK-32455?focusedCommentId=17739785=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17739785
> Next to that, there's also FLINK-25509 which can only be supported with Flink
> 1.19 and higher.
> So we should:
> * Drop support for 1.17 and 1.18
> * Refactor the Flink Kafka connector to use the new
> {code:java}MigrationTest{code}
> We will support the Flink Kafka connector for Flink 1.18 via the v3.1 branch;
> this change will be a new v4.0 version with support for Flink 1.19 and the
> upcoming Flink 1.20
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35351) Restore from unaligned checkpoints with a custom partitioner fails.
[
https://issues.apache.org/jira/browse/FLINK-35351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35351:
---
Labels: pull-request-available (was: )
> Restore from unaligned checkpoints with a custom partitioner fails.
> ---
>
> Key: FLINK-35351
> URL: https://issues.apache.org/jira/browse/FLINK-35351
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Reporter: Dmitriy Linevich
>Assignee: Dmitriy Linevich
>Priority: Major
> Labels: pull-request-available
>
> We encountered a problem when using a custom partitioner with unaligned
> checkpoints. The bug reproduces under the following steps:
> # Run a job with graph: Source[2]->Sink[3], the custom partitioner applied
> after the Source task.
> # Make a checkpoint.
> # Restore from the checkpoint with a different source parallelism:
> Source[1]->Sink[3].
> # An exception is thrown.
> This issue does not occur when restoring with the same parallelism or when
> changing the Sink parallelism. The exception only occurs when the parallelism
> of the Source is changed while the Sink parallelism remains the same.
> See the exception below and the test code at the end.
> {code:java}
> [db13789c52b80aad852c53a0afa26247] Task [Sink: sink (3/3)#0] WARN Sink: sink
> (3/3)#0
> (be1d158c2e77fc9ed9e3e5d9a8431dc2_0a448493b4782967b150582570326227_2_0)
> switched from RUNNING to FAILED with failure cause:
> java.io.IOException: Can't get next record for channel
> InputChannelInfo{gateIdx=0, inputChannelIdx=0}
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:106)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:600)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:930)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:879)
> ~[classes/:?]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:960)
> ~[classes/:?]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939)
> [classes/:?]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:753)
> [classes/:?]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
> [classes/:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
> Caused by: java.io.IOException: Corrupt stream, found tag: -1
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:222)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:44)
> ~[classes/:?]
> at
> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
> ~[classes/:?]
> at
> org.apache.flink.runtime.io.network.api.serialization.NonSpanningWrapper.readInto(NonSpanningWrapper.java:337)
> ~[classes/:?]
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNonSpanningRecord(SpillingAdaptiveSpanningRecordDeserializer.java:128)
> ~[classes/:?]
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:103)
> ~[classes/:?]
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:93)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer$VirtualChannel.getNextRecord(DemultiplexingRecordDeserializer.java:79)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer.getNextRecord(DemultiplexingRecordDeserializer.java:154)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer.getNextRecord(DemultiplexingRecordDeserializer.java:54)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:103)
> ~[classes/:?]
> ... 10 more {code}
> We discovered that this issue occurs due to an optimization in the
>
[jira] [Updated] (FLINK-33983) Introduce JobEvent and JobEventStore for Batch Job Recovery
[ https://issues.apache.org/jira/browse/FLINK-33983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33983: --- Labels: pull-request-available (was: ) > Introduce JobEvent and JobEventStore for Batch Job Recovery > --- > > Key: FLINK-33983 > URL: https://issues.apache.org/jira/browse/FLINK-33983 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35153) Internal Async State Implementation and StateDescriptor for Map/List State
[ https://issues.apache.org/jira/browse/FLINK-35153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35153: --- Labels: pull-request-available (was: ) > Internal Async State Implementation and StateDescriptor for Map/List State > -- > > Key: FLINK-35153 > URL: https://issues.apache.org/jira/browse/FLINK-35153 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Zakelly Lan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35344) Move same code from multiple subclasses to JdbcSourceChunkSplitter
[ https://issues.apache.org/jira/browse/FLINK-35344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35344: --- Labels: pull-request-available (was: ) > Move same code from multiple subclasses to JdbcSourceChunkSplitter > -- > > Key: FLINK-35344 > URL: https://issues.apache.org/jira/browse/FLINK-35344 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Hongshun Wang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > Current, subclasses of JdbcSourceChunkSplitter almost share same code, but > each have one copy. It's hard for later maintenance. > Thus, this Jira aim to move same code from multiple subclasses to > JdbcSourceChunkSplitter, just like what have done in > AbstractScanFetchTask(https://github.com/apache/flink-cdc/issues/2690) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35253) Translate "State Processor API "page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35253: --- Labels: pull-request-available (was: ) > Translate "State Processor API "page into Chinese > - > > Key: FLINK-35253 > URL: https://issues.apache.org/jira/browse/FLINK-35253 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.19.0 >Reporter: Juan Zifeng >Assignee: Juan Zifeng >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > The links are > https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/libs/state_processor_api/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-5193) Recovering all jobs fails completely if a single recovery fails
[
https://issues.apache.org/jira/browse/FLINK-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-5193:
--
Labels: pull-request-available (was: )
> Recovering all jobs fails completely if a single recovery fails
> ---
>
> Key: FLINK-5193
> URL: https://issues.apache.org/jira/browse/FLINK-5193
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.1.3, 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.4, 1.2.0
>
>
> In HA case where the {{JobManager}} tries to recover all submitted job
> graphs, e.g. when regaining leadership, it can happen that none of the
> submitted jobs are recovered if a single recovery fails. Instead of failing
> the complete recovery procedure, the {{JobManager}} should still try to
> recover the remaining (non-failing) jobs and print a proper error message for
> the failed recoveries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35337) Keep up with the latest version of tikv client
[ https://issues.apache.org/jira/browse/FLINK-35337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35337: --- Labels: pull-request-available (was: ) > Keep up with the latest version of tikv client > -- > > Key: FLINK-35337 > URL: https://issues.apache.org/jira/browse/FLINK-35337 > Project: Flink > Issue Type: Improvement >Reporter: ouyangwulin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20400) Migrate test_streaming_sql.sh
[ https://issues.apache.org/jira/browse/FLINK-20400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20400: --- Labels: pull-request-available (was: ) > Migrate test_streaming_sql.sh > - > > Key: FLINK-20400 > URL: https://issues.apache.org/jira/browse/FLINK-20400 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Tests >Reporter: Jark Wu >Assignee: Muhammet Orazov >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35332) Manually setting rest.bind-address does not work for Flink running on Hadoop Yarn dual network environment
[
https://issues.apache.org/jira/browse/FLINK-35332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35332:
---
Labels: pull-request-available (was: )
> Manually setting rest.bind-address does not work for Flink running on Hadoop
> Yarn dual network environment
> --
>
> Key: FLINK-35332
> URL: https://issues.apache.org/jira/browse/FLINK-35332
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.15.4
>Reporter: Yao Zhang
>Priority: Major
> Labels: pull-request-available
>
> Given the Hadoop Yarn cluster with dual networks:
> * 192.168.x.x: For data transfer. Speed: 10Gbps.
> * 10.x.x.x: For management only. Speed: 1Gbps.
>
> A client outside the Hadoop Yarn cluster is configured, with management
> network only(10.x.x.x) and data transfer high speed network not accessible.
> To reproduce, we sumbit a Flink job from this client(Batch word count for
> example), the job can be successfully submitted but the result cannot be
> retrieved, with the exception: Connection refused:
> \{jobmanager_hostname}:\{jm_port}. The root cause is the job manager rest
> address is bind to its actual address (192.168.x.x) rather than 0.0.0.0.
> Manually setting rest.bind-address does not work.
>
> One of the changes in Flink-24474 in YarnEntrypointUtils overwrites
> RestOptions.BIND_ADDRESS to the node's actual address. This change should be
> reverted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33986) Extend shuffleMaster to support batch snapshot.
[ https://issues.apache.org/jira/browse/FLINK-33986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33986: --- Labels: pull-request-available (was: ) > Extend shuffleMaster to support batch snapshot. > --- > > Key: FLINK-33986 > URL: https://issues.apache.org/jira/browse/FLINK-33986 > Project: Flink > Issue Type: Sub-task >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > > Extend shuffleMaster to support batch snapshot as follows: > # Add method supportsBatchSnapshot to identify whether the shuffle master > supports taking snapshot in batch scenarios > # Add method snapshotState and restoreState to snapshot and restore the > shuffle master's state. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34108) Add URL_ENCODE and URL_DECODE function
[ https://issues.apache.org/jira/browse/FLINK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34108: --- Labels: pull-request-available (was: ) > Add URL_ENCODE and URL_DECODE function > -- > > Key: FLINK-34108 > URL: https://issues.apache.org/jira/browse/FLINK-34108 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Priority: Major > Labels: pull-request-available > > Add URL_ENCODE and URL_DECODE function > URL_ENCODE(str) - Translates a string into > 'application/x-www-form-urlencoded' format using a specific encoding scheme. > URL_DECODE(str) - Decodes a string in 'application/x-www-form-urlencoded' > format using a specific encoding scheme. > Related ticket from Calcite: CALCITE-5825 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33892) FLIP-383: Support Job Recovery from JobMaster Failures for Batch Jobs
[ https://issues.apache.org/jira/browse/FLINK-33892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33892: --- Labels: pull-request-available (was: ) > FLIP-383: Support Job Recovery from JobMaster Failures for Batch Jobs > - > > Key: FLINK-33892 > URL: https://issues.apache.org/jira/browse/FLINK-33892 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Reporter: Lijie Wang >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > > This is the umbrella ticket for > [FLIP-383|https://cwiki.apache.org/confluence/display/FLINK/FLIP-383%3A+Support+Job+Recovery+for+Batch+Jobs] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35305) FLIP-438: Amazon SQS Sink Connector
[ https://issues.apache.org/jira/browse/FLINK-35305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35305: --- Labels: pull-request-available (was: ) > FLIP-438: Amazon SQS Sink Connector > --- > > Key: FLINK-35305 > URL: https://issues.apache.org/jira/browse/FLINK-35305 > Project: Flink > Issue Type: New Feature > Components: Connectors / AWS >Reporter: Priya Dhingra >Priority: Major > Labels: pull-request-available > > This is an umbrella task for FLIP-438. FLIP-438: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35323) Only the schema of the first hit table is recorded when the source-table of the transformer hits multiple tables
[
https://issues.apache.org/jira/browse/FLINK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35323:
---
Labels: pull-request-available (was: )
> Only the schema of the first hit table is recorded when the source-table of
> the transformer hits multiple tables
>
>
> Key: FLINK-35323
> URL: https://issues.apache.org/jira/browse/FLINK-35323
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: 3.1.0
>Reporter: Wenkai Qi
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> {code:java}
> transform:
> - source-table: mydb.web_\.*
> projection: \*, localtimestamp as new_timestamp
> description: project fields from source table {code}
> Table mydb.web_order: col1, col2, col3
> Table mydb.web_info: col1, col4
> If transform data operator processes `mydb.web_info` first and then
> `mydb.web_order`, its schema will always be `col1, col4`.
> Cause by: TransformDataOperator.java
> {code:java}
> private transient Map
> transformProjectionProcessorMap;
> private transient Map
> transformFilterProcessorMap; {code}
> The relationship of `TableId` is missing here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35041) IncrementalRemoteKeyedStateHandleTest.testSharedStateReRegistration failed
[
https://issues.apache.org/jira/browse/FLINK-35041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35041:
---
Labels: pull-request-available (was: )
> IncrementalRemoteKeyedStateHandleTest.testSharedStateReRegistration failed
> --
>
> Key: FLINK-35041
> URL: https://issues.apache.org/jira/browse/FLINK-35041
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Affects Versions: 1.20.0
>Reporter: Weijie Guo
>Assignee: Feifan Wang
>Priority: Blocker
> Labels: pull-request-available
>
> {code:java}
> Apr 08 03:22:45 03:22:45.450 [ERROR]
> org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandleTest.testSharedStateReRegistration
> -- Time elapsed: 0.034 s <<< FAILURE!
> Apr 08 03:22:45 org.opentest4j.AssertionFailedError:
> Apr 08 03:22:45
> Apr 08 03:22:45 expected: false
> Apr 08 03:22:45 but was: true
> Apr 08 03:22:45 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Apr 08 03:22:45 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Apr 08 03:22:45 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(K.java:45)
> Apr 08 03:22:45 at
> org.apache.flink.runtime.state.DiscardRecordedStateObject.verifyDiscard(DiscardRecordedStateObject.java:34)
> Apr 08 03:22:45 at
> org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandleTest.testSharedStateReRegistration(IncrementalRemoteKeyedStateHandleTest.java:211)
> Apr 08 03:22:45 at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 08 03:22:45 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Apr 08 03:22:45 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Apr 08 03:22:45 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Apr 08 03:22:45 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Apr 08 03:22:45 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58782=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=9238]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35322) PubSub Connector Weekly build fails
[ https://issues.apache.org/jira/browse/FLINK-35322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35322: --- Labels: pull-request-available test-stability (was: test-stability) > PubSub Connector Weekly build fails > > > Key: FLINK-35322 > URL: https://issues.apache.org/jira/browse/FLINK-35322 > Project: Flink > Issue Type: Bug > Components: Connectors / Google Cloud PubSub >Affects Versions: 3.1.0 >Reporter: Ahmed Hamdy >Priority: Major > Labels: pull-request-available, test-stability > > Weekly builds for GCP pubSub connector is failing for 1.19 due to compilation > error in tests. > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8768752932/job/24063472769 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8863605354 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8954270618 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34993) Support Model CRUD in parser
[ https://issues.apache.org/jira/browse/FLINK-34993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34993: --- Labels: pull-request-available (was: ) > Support Model CRUD in parser > > > Key: FLINK-34993 > URL: https://issues.apache.org/jira/browse/FLINK-34993 > Project: Flink > Issue Type: Sub-task >Reporter: Hao Li >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35328) AutoScale supports setting the maximum floating parallelism by the number of Pulsar partitions
[ https://issues.apache.org/jira/browse/FLINK-35328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35328: --- Labels: pull-request-available (was: ) > AutoScale supports setting the maximum floating parallelism by the number of > Pulsar partitions > -- > > Key: FLINK-35328 > URL: https://issues.apache.org/jira/browse/FLINK-35328 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Wenbing Shen >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32082) Documentation of checkpoint file-merging
[ https://issues.apache.org/jira/browse/FLINK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32082: --- Labels: pull-request-available (was: ) > Documentation of checkpoint file-merging > > > Key: FLINK-32082 > URL: https://issues.apache.org/jira/browse/FLINK-32082 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35314) Add Flink CDC pipeline transform user document
[ https://issues.apache.org/jira/browse/FLINK-35314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35314: --- Labels: documentation pull-request-available (was: documentation) > Add Flink CDC pipeline transform user document > -- > > Key: FLINK-35314 > URL: https://issues.apache.org/jira/browse/FLINK-35314 > Project: Flink > Issue Type: New Feature > Components: Flink CDC >Reporter: Wenkai Qi >Priority: Major > Labels: documentation, pull-request-available > Original Estimate: 48h > Remaining Estimate: 48h > > The document outline is as follows: > # Definition > # Parameters > # Metadata Fields > # Functions > # Example > # Problem > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35317) Flink CDC CLI Supports submitting multiple YAML job at once
[
https://issues.apache.org/jira/browse/FLINK-35317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35317:
---
Labels: pull-request-available (was: )
> Flink CDC CLI Supports submitting multiple YAML job at once
> ---
>
> Key: FLINK-35317
> URL: https://issues.apache.org/jira/browse/FLINK-35317
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: yux
>Priority: Minor
> Labels: pull-request-available
>
> Currently, Flink CDC CLI only allows submitting one YAML pipeline job each
> time. It would be convenient if users can submit multiple .yml files at once
> like this:
> {{./bin/flink-cdc.sh job1.yml job2.yml --flink-home /opt/flink ...}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35316) Add CDC e2e test case for on Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35316: --- Labels: pull-request-available (was: ) > Add CDC e2e test case for on Flink 1.19 > --- > > Key: FLINK-35316 > URL: https://issues.apache.org/jira/browse/FLINK-35316 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: yux >Priority: Minor > Labels: pull-request-available > > Since Flink 1.19 has been generally available, Flink CDC is expected to be > used with it. E2e test cases should cover this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35197) Support the execution of suspend, resume materialized table in continuous refresh mode
[
https://issues.apache.org/jira/browse/FLINK-35197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35197:
---
Labels: pull-request-available (was: )
> Support the execution of suspend, resume materialized table in continuous
> refresh mode
> --
>
> Key: FLINK-35197
> URL: https://issues.apache.org/jira/browse/FLINK-35197
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Gateway
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: Feng Jin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> In continuous refresh mode, support suspend, resume the background refresh
> job of materialized table.
> {code:SQL}
> // suspend
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name SUSPEND
> // resume
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name RESUME
> [WITH('key1' = 'val1', 'key2' = 'val2')]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35313) Add upsert changelog mode to avoid UPDATE_BEFORE records push down
[
https://issues.apache.org/jira/browse/FLINK-35313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35313:
---
Labels: pull-request-available (was: )
> Add upsert changelog mode to avoid UPDATE_BEFORE records push down
> --
>
> Key: FLINK-35313
> URL: https://issues.apache.org/jira/browse/FLINK-35313
> Project: Flink
> Issue Type: New Feature
> Components: Flink CDC
>Reporter: ude
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-05-09-15-02-33-572.png
>
>
> I try to use flink sql to write mysql cdc-data into redis as a dimension
> table for other business use. When executing {{UPDATE}} DML, the cdc-data
> will be converted into {{-D (UPDATE_BEFORE)}} and {{+I (UPDATE_AFTER)}} two
> records to sink redis. However, delete first will cause other data streams to
> be lost(NULL) when join data, which is unacceptable.
> I think we can add support for [upser changelog
> mode|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables/#table-to-stream-conversion]
> by adding changelogMode option with mandatory primary key
> configuration.Basically, with {{changelogMode=upsert}} we will avoid
> {{UPDATE_BEFORE}} rows and we will require a primary key for the table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35293) FLIP-445: Support dynamic parallelism inference for HiveSource
[ https://issues.apache.org/jira/browse/FLINK-35293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35293: --- Labels: pull-request-available (was: ) > FLIP-445: Support dynamic parallelism inference for HiveSource > -- > > Key: FLINK-35293 > URL: https://issues.apache.org/jira/browse/FLINK-35293 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Affects Versions: 1.20.0 >Reporter: xingbe >Assignee: xingbe >Priority: Major > Labels: pull-request-available > > [FLIP-379|https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs] > introduces dynamic source parallelism inference, which, compared to static > inference, utilizes runtime information to more accurately determine the > source parallelism. The FileSource already possesses the capability for > dynamic parallelism inference. As a follow-up task to FLIP-379, this FLIP > plans to implement the dynamic parallelism inference interface for > HiveSource, and also switches the default static parallelism inference to > dynamic parallelism inference. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35164) Support `ALTER CATALOG RESET` syntax
[ https://issues.apache.org/jira/browse/FLINK-35164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35164: --- Labels: pull-request-available (was: ) > Support `ALTER CATALOG RESET` syntax > > > Key: FLINK-35164 > URL: https://issues.apache.org/jira/browse/FLINK-35164 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Yubin Li >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-18-23-26-59-854.png > > > h3. ALTER CATALOG catalog_name RESET (key1, key2, ...) > Reset one or more properties to its default value in the specified catalog. > !image-2024-04-18-23-26-59-854.png|width=781,height=527! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35310) Replace RBAC verb wildcards with actual verbs
[
https://issues.apache.org/jira/browse/FLINK-35310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35310:
---
Labels: pull-request-available (was: )
> Replace RBAC verb wildcards with actual verbs
> -
>
> Key: FLINK-35310
> URL: https://issues.apache.org/jira/browse/FLINK-35310
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Environment: Running on Kubernetes using the flink-operator version
> 1.8.0
>Reporter: Tim
>Priority: Major
> Labels: pull-request-available
>
> We are deploying the flink operator on a managed Kubernetes cluster which
> utilizes [Kyverno Policy Management|https://kyverno.io/] and all it's default
> rules. Not complying to certain rules, leads to a restriction in deploying.
> As we are using Helm to build the manifest files (which is super useful) I
> recognized that in the RBAC template "wildcards" are being used for all verbs
> ("*").
> This violates the following Kyverno ruleset:
> [https://kyverno.io/policies/other/restrict-wildcard-verbs/restrict-wildcard-verbs/]
> Besides that I think that it would also be cleaner to explicitly list the
> needed verbs instead of just using the star symbol as a wildcard.
> I have already attempted to change this in a fork as a demonstration how it
> could be changed to be conform. Please take a look and I would greatly
> appreciate a change in that direction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35198) Support the execution of refresh materialized table
[
https://issues.apache.org/jira/browse/FLINK-35198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35198:
---
Labels: pull-request-available (was: )
> Support the execution of refresh materialized table
> ---
>
> Key: FLINK-35198
> URL: https://issues.apache.org/jira/browse/FLINK-35198
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: xuyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> {code:SQL}
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name REFRESH
> [PARTITION (key1=val1, key2=val2, ...)]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35309) Enable Notice file ci check and fix Notice
[ https://issues.apache.org/jira/browse/FLINK-35309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35309: --- Labels: pull-request-available (was: ) > Enable Notice file ci check and fix Notice > --- > > Key: FLINK-35309 > URL: https://issues.apache.org/jira/browse/FLINK-35309 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Blocker > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > Changes: > * Add ci to check Notice file > * Fix Notice file issue -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32087) Space amplification statistics of file merging
[ https://issues.apache.org/jira/browse/FLINK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32087: --- Labels: pull-request-available (was: ) > Space amplification statistics of file merging > -- > > Key: FLINK-32087 > URL: https://issues.apache.org/jira/browse/FLINK-32087 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Rui Xia >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35306) Flink cannot compile with jdk17
[ https://issues.apache.org/jira/browse/FLINK-35306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35306: --- Labels: pull-request-available (was: ) > Flink cannot compile with jdk17 > --- > > Key: FLINK-35306 > URL: https://issues.apache.org/jira/browse/FLINK-35306 > Project: Flink > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: image-2024-05-08-11-48-04-161.png > > > Reason: TypeSerializerUpgradeTestBase.UpgradeVerifier update the > schemaCompatibilityMatcher method name to schemaCompatibilityCondition, but > some subclasses didn't change it, such as: > PojoRecordSerializerUpgradeTestSpecifications.PojoToRecordVerifier. > > It belongs to flink-tests-java17 module, and it doesn't compile by default. > > it's caused by > * https://issues.apache.org/jira/browse/FLINK-25537 > * https://github.com/apache/flink/pull/24603 > > !image-2024-05-08-11-48-04-161.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35281) FlinkEnvironmentUtils#addJar add each jar only once
[
https://issues.apache.org/jira/browse/FLINK-35281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35281:
---
Labels: pull-request-available (was: )
> FlinkEnvironmentUtils#addJar add each jar only once
> ---
>
> Key: FLINK-35281
> URL: https://issues.apache.org/jira/browse/FLINK-35281
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.1.0
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: cdc-3.2.0
>
>
> Current org.apache.flink.cdc.composer.flink.FlinkEnvironmentUtils#addJar will
> be invoked for each source and sink.
> {code:java}
> public static void addJar(StreamExecutionEnvironment env, URL jarUrl) {
> try {
> Class envClass =
> StreamExecutionEnvironment.class;
> Field field = envClass.getDeclaredField("configuration");
> field.setAccessible(true);
> Configuration configuration = ((Configuration) field.get(env));
> List jars =
> configuration.getOptional(PipelineOptions.JARS).orElse(new
> ArrayList<>());
> jars.add(jarUrl.toString());
> configuration.set(PipelineOptions.JARS, jars);
> } catch (Exception e) {
> throw new RuntimeException("Failed to add JAR to Flink execution
> environment", e);
> } {code}
> if multiple source or sink share same jar, the par path will be added
> repeatly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35193) Support drop materialized table syntax and execution in continuous refresh mode
[
https://issues.apache.org/jira/browse/FLINK-35193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35193:
---
Labels: pull-request-available (was: )
> Support drop materialized table syntax and execution in continuous refresh
> mode
> ---
>
> Key: FLINK-35193
> URL: https://issues.apache.org/jira/browse/FLINK-35193
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: xuyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> In continuous refresh mode, support drop materialized table and the
> background refresh job.
> {code:SQL}
> DROP MATERIALIZED TABLE [ IF EXISTS ] [catalog_name.][db_name.]table_name
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35299) FlinkKinesisConsumer does not respect StreamInitialPosition for new Kinesis Stream when restoring from snapshot
[ https://issues.apache.org/jira/browse/FLINK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35299: --- Labels: pull-request-available (was: ) > FlinkKinesisConsumer does not respect StreamInitialPosition for new Kinesis > Stream when restoring from snapshot > --- > > Key: FLINK-35299 > URL: https://issues.apache.org/jira/browse/FLINK-35299 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: aws-connector-4.2.0 >Reporter: Hong Liang Teoh >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.4.0 > > > h3. What > The FlinkKinesisConsumer allows users to read from [multiple Kinesis > Streams|https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L224]. > Users can also specify a STREAM_INITIAL_POSITION, which configures if the > consumer starts reading the stream from TRIM_HORIZON / LATEST / AT_TIMESTAMP. > When restoring the Kinesis Consumer from an existing snapshot, users can > configure the consumer to read from additional Kinesis Streams. The expected > behavior would be for the FlinkKinesisConsumer to start reading from the > additional Kinesis Streams respecting the STREAM_INITIAL_POSITION > configuration. However, we find that it currently reads from TRIM_HORIZON. > This is surprising behavior and should be corrected. > h3. Why > Principle of Least Astonishment > h3. How > We recommend that we reconstruct the previously seen streams by iterating > through the [sequenceNumsStateForCheckpoint in > FlinkKinesisConsumer#initializeState()|https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L454]. > h3. Risks > This might increase the state restore time. We can consider adding a feature > flag for users to turn this check off. -- This message was sent by Atlassian Jira (v8.20.10#820010)
