[jira] [Closed] (FLINK-25177) Support "DESCRIBE TABLE EXTENDED" with managed table
[ https://issues.apache.org/jira/browse/FLINK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-25177. Fix Version/s: (was: 1.20.0) Resolution: Invalid > Support "DESCRIBE TABLE EXTENDED" with managed table > > > Key: FLINK-25177 > URL: https://issues.apache.org/jira/browse/FLINK-25177 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Jingsong Lee >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available, stale-assigned > > Expose informations in FLIP: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-188:+Introduce+Built-in+Dynamic+Table+Storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33621) Table Store with PyFlink
[ https://issues.apache.org/jira/browse/FLINK-33621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788950#comment-17788950 ] Jingsong Lee commented on FLINK-33621: -- Hi [~mfatihaktas] First, I think you can take a look to Paimon ([https://paimon.apache.org/),] Table Store has became Paimon, a separate project now. And you can use Table API in Python, and convert Table to DataStream, similar Java code is in: https://paimon.apache.org/docs/master/api/flink-api/ > Table Store with PyFlink > > > Key: FLINK-33621 > URL: https://issues.apache.org/jira/browse/FLINK-33621 > Project: Flink > Issue Type: Technical Debt > Components: Documentation, Table Store >Reporter: Mehmet Aktas >Priority: Major > Labels: documentation > > We are working on a project that requires setting up [Table > Store|https://nightlies.apache.org/flink/flink-table-store-docs-stable/] with > PyFlink. However, we could not find any documentation on (1) if this is > possible, (2) if possible, how to set it up. We could find only the links > through [https://nightlies.apache.org/flink/flink-table-store-docs-stable/,] > which does not seem to address PyFlink. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773075#comment-17773075
]
Jingsong Lee commented on FLINK-33200:
--
CC [~lincoln.86xy]
> ItemAt Expression validation fail in Table API due to type mismatch
> ---
>
> Key: FLINK-33200
> URL: https://issues.apache.org/jira/browse/FLINK-33200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Zhenqiu Huang
>Priority: Minor
> Fix For: 1.8.4
>
>
> The table schema is defined as below:
> public static final DataType DATA_TYPE = DataTypes.ROW(
> DataTypes.FIELD("id", DataTypes.STRING()),
> DataTypes.FIELD("events",
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
> );
> public static final Schema SCHEMA =
> Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
> inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
> The validation fail as "eventType" is inferred as
> BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
> StringDataTypeInfo. The validation fail at
> case mti: MapTypeInfo[_, _] =>
> if (key.resultType == mti.getKeyTypeInfo) {
> ValidationSuccess
> } else {
> ValidationFailure(
> s"Map entry access needs a valid key of type " +
> s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-22128) Window aggregation should have unique keys
[
https://issues.apache.org/jira/browse/FLINK-22128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766177#comment-17766177
]
Jingsong Lee commented on FLINK-22128:
--
[~337361...@qq.com] I think yes, this still should be done. But I don't have
time to research more details. CC [~lincoln.86xy]
> Window aggregation should have unique keys
> --
>
> Key: FLINK-22128
> URL: https://issues.apache.org/jira/browse/FLINK-22128
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jingsong Lee
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned
>
> We should add match method in {{FlinkRelMdUniqueKeys for
> StreamPhysicalWindowAggregate}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32001) SupportsRowLevelUpdate does not support returning only a part of the columns.
[
https://issues.apache.org/jira/browse/FLINK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719601#comment-17719601
]
Jingsong Lee commented on FLINK-32001:
--
CC [~yuxia]
> SupportsRowLevelUpdate does not support returning only a part of the columns.
> -
>
> Key: FLINK-32001
> URL: https://issues.apache.org/jira/browse/FLINK-32001
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0
>Reporter: Ming Li
>Priority: Major
>
> [FLIP-282|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061]
> introduces the new Delete and Update API in Flink SQL. Although it is
> described in the documentation that in case of {{partial-update}} we only
> need to return the primary key columns and the updated columns.
> But in fact, the topology of the job is {{{}source -> cal ->
> constraintEnforcer -> sink{}}}, and the constraint check will be performed in
> the operator of {{{}constraintEnforcer{}}}, which is done according to index,
> not according to column. If only some columns are returned, the constraint
> check is wrong, and it is easy to generate
> {{{}ArrayIndexOutOfBoundsException{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-31335) using sql-gateway to submit job to yarn cluster, sql-gateway don't support kerberos
[ https://issues.apache.org/jira/browse/FLINK-31335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-31335: Assignee: felixzh > using sql-gateway to submit job to yarn cluster, sql-gateway don't support > kerberos > --- > > Key: FLINK-31335 > URL: https://issues.apache.org/jira/browse/FLINK-31335 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Gateway >Affects Versions: 1.16.0 >Reporter: felixzh >Assignee: felixzh >Priority: Major > Labels: pull-request-available > > when submit job to yarn cluster, sql-gateway don't support kerberos. > 1. yarn-per-job mode > -Dexecution.target=yarn-per-job > 2. yarn-session mode > -Dexecution.target=yarn-session -Dyarn.application.id=yarnSessionAppID(eg: > application_1677479737242_0052) > sql-gateway need to use SecurityUtils Modules. > default use flink-conf.yaml(security.kerberos.login.principal and > security.kerberos.login.keytab), also support > -Dsecurity.kerberos.login.keytab and -Dsecurity.kerberos.login.principal (eg: > 1/2) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31429) Support CTAS(create table as) streaming job with schema changes in table store
[ https://issues.apache.org/jira/browse/FLINK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31429. Resolution: Fixed > Support CTAS(create table as) streaming job with schema changes in table store > -- > > Key: FLINK-31429 > URL: https://issues.apache.org/jira/browse/FLINK-31429 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Fix For: table-store-0.4.0 > > > Currently CDC connectors for Flink has the ability to stream out records > changes and schema changes of a database table. Flink Table Store should have > the ability to directly consume these changes, including schema changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31435) Introduce event parser for MySql Debezium JSON format in Table Store
[ https://issues.apache.org/jira/browse/FLINK-31435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31435. Resolution: Fixed > Introduce event parser for MySql Debezium JSON format in Table Store > > > Key: FLINK-31435 > URL: https://issues.apache.org/jira/browse/FLINK-31435 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Fix For: table-store-0.4.0 > > > MySQL is widely used among Flink CDC connector users. We should first support > consuming changes from MySQL. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30419) Allow tuning of transaction timeout
[ https://issues.apache.org/jira/browse/FLINK-30419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30419. Resolution: Fixed > Allow tuning of transaction timeout > --- > > Key: FLINK-30419 > URL: https://issues.apache.org/jira/browse/FLINK-30419 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Vicky Papavasileiou >Assignee: Alex Sorokoumov >Priority: Major > > FTS sets the producer transaction timeout to 1hr. The maximum allowed by a > kafka broker is 15 mins. This causes exceptions to be thrown and the job dies > when kafka log is enabled on a table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30419) Allow tuning of transaction timeout
[ https://issues.apache.org/jira/browse/FLINK-30419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706198#comment-17706198 ] Jingsong Lee commented on FLINK-30419: -- https://github.com/apache/incubator-paimon/issues/754 > Allow tuning of transaction timeout > --- > > Key: FLINK-30419 > URL: https://issues.apache.org/jira/browse/FLINK-30419 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Vicky Papavasileiou >Assignee: Alex Sorokoumov >Priority: Major > > FTS sets the producer transaction timeout to 1hr. The maximum allowed by a > kafka broker is 15 mins. This causes exceptions to be thrown and the job dies > when kafka log is enabled on a table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27958) Compare batch maxKey to reduce comparisons in SortMergeReader
[ https://issues.apache.org/jira/browse/FLINK-27958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-27958. Resolution: Fixed > Compare batch maxKey to reduce comparisons in SortMergeReader > - > > Key: FLINK-27958 > URL: https://issues.apache.org/jira/browse/FLINK-27958 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Minor > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > In SortMergeReader, each sub reader is batched reader. > When adding a new batch to the priority queue, we can look at the maximum key > of the batch, and if its maximum key is smaller than the minimum key of other > batches, then we can just output the whole batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31111) Introduce CatalogTestBase
[ https://issues.apache.org/jira/browse/FLINK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-3. Resolution: Fixed > Introduce CatalogTestBase > - > > Key: FLINK-3 > URL: https://issues.apache.org/jira/browse/FLINK-3 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, only tests for FlinkCatalog or ITCase, we should add cases for > catalogs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31001) Introduce Hive writer
[ https://issues.apache.org/jira/browse/FLINK-31001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31001. Resolution: Fixed > Introduce Hive writer > - > > Key: FLINK-31001 > URL: https://issues.apache.org/jira/browse/FLINK-31001 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27103) Don't store redundant primary key fields
[ https://issues.apache.org/jira/browse/FLINK-27103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-27103. Resolution: Fixed > Don't store redundant primary key fields > > > Key: FLINK-27103 > URL: https://issues.apache.org/jira/browse/FLINK-27103 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Minor > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > We are currently storing the primary key redundantly in the file, we can > directly use the primary key field in the original fields to avoid redundant > storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27002) Optimize batch multiple partitions inserting
[ https://issues.apache.org/jira/browse/FLINK-27002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-27002. Resolution: Fixed > Optimize batch multiple partitions inserting > > > Key: FLINK-27002 > URL: https://issues.apache.org/jira/browse/FLINK-27002 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Minor > Fix For: table-store-0.4.0 > > > By default, batch sink should sort the input by partition and sequence_field > to avoid generating a large number of small files. Too many small files cause > poor performance, especially object storage. > We can not implement `SupportsPartitioning.requiresPartitionGrouping`. we > need sequence.field to sort, otherwise we can't confirm what the last record > is. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31190) Supports Spark call procedure command on Table Store
[ https://issues.apache.org/jira/browse/FLINK-31190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31190. Resolution: Fixed > Supports Spark call procedure command on Table Store > > > Key: FLINK-31190 > URL: https://issues.apache.org/jira/browse/FLINK-31190 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Nicholas Jiang >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > At present Hudi and Iceberg supports the Spark call procedure command to > execute the table service action etc. Flink Table Store could also support > Spark call procedure command to run compaction etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28086) Table Store Catalog supports partition methods
[ https://issues.apache.org/jira/browse/FLINK-28086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28086. Resolution: Fixed > Table Store Catalog supports partition methods > -- > > Key: FLINK-28086 > URL: https://issues.apache.org/jira/browse/FLINK-28086 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Minor > Fix For: table-store-0.4.0 > > > Table Store Catalog can support: > * listPartitions > * listPartitionsByFilter > * getPartition > * partitionExists > * dropPartition -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31194) Introduces savepoint mechanism of Table Store
[ https://issues.apache.org/jira/browse/FLINK-31194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31194. Resolution: Fixed > Introduces savepoint mechanism of Table Store > - > > Key: FLINK-31194 > URL: https://issues.apache.org/jira/browse/FLINK-31194 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Nicholas Jiang >Priority: Major > Fix For: table-store-0.4.0 > > > Disaster Recovery is very much mission critical for any software. Especially > when it comes to data systems, the impact could be very serious leading to > delay in business decisions or even wrong business decisions at times. Flink > Table Store could introduce savepoint mechanism to assist users in recovering > data from a previous state. > As the name suggest, "savepoint" saves the table as of the snapshot, so that > it lets you restore the table to this savepoint at a later point in snapshot > if need be. Care is taken to ensure cleaner will not clean up any files that > are savepointed. On similar lines, savepoint cannot be triggered on a > snapshot that is already cleaned up. In simpler terms, this is synonymous to > taking a backup, just that we don't make a new copy of the table, but just > save the state of the table elegantly so that we can restore it later when in > need. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31195) FullChangelogStoreSinkWrite bucket writer conflicts with rescale
[ https://issues.apache.org/jira/browse/FLINK-31195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31195. Resolution: Fixed > FullChangelogStoreSinkWrite bucket writer conflicts with rescale > > > Key: FLINK-31195 > URL: https://issues.apache.org/jira/browse/FLINK-31195 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > At present, this operator relies on ListState, Flink distributes data > according to round-robin when rescaling, which may be different from the > distribution rules of our bucket after rescaling. > We need to change the mode of UnionListState, broadcast to each node, and > finally decide whether it belongs to the task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30945) FTS does not support multiple writers into the same table and topic
[
https://issues.apache.org/jira/browse/FLINK-30945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30945.
Resolution: Fixed
https://github.com/apache/incubator-paimon/issues/745
> FTS does not support multiple writers into the same table and topic
> ---
>
> Key: FLINK-30945
> URL: https://issues.apache.org/jira/browse/FLINK-30945
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Vicky Papavasileiou
>Priority: Major
>
> When creating two different streaming jobs that INSERT INTO the same table
> and kafka topic, the second job is never able to make progress as the
> transaction gets constantly aborted due to the producer getting fenced.
> FTS should set the transactionalIdPrefix to avoid transactions of different
> jobs clashing.
> {code:java}
> 2023-02-06 17:13:36,088 WARN org.apache.flink.runtime.taskmanager.Task [] -
> Writer -> Global Committer -> Sink: end (1/1)#0
> (8cf4197af9716623c3c19e7fa3d7c071_b5c8d46f3e7b141acf271f12622e752b_0_0)
> switched from RUNNING to FAILED with failure cause:
> org.apache.flink.util.FlinkRuntimeException: Committing one of transactions
> failed, logging first encountered failure at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.notifyCheckpointComplete(TwoPhaseCommitSinkFunction.java:323)
> at
> org.apache.flink.table.store.connector.sink.StoreWriteOperator.notifyCheckpointComplete(StoreWriteOperator.java:175)
> at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:104)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.notifyCheckpointComplete(RegularOperatorChain.java:145)
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpoint(SubtaskCheckpointCoordinatorImpl.java:479)
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:413)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1412)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$15(StreamTask.java:1353)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$18(StreamTask.java:1392)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) at
> java.lang.Thread.run(Thread.java:750) Caused by:
> org.apache.flink.table.store.shaded.org.apache.kafka.common.errors.ProducerFencedException:
> There is a newer producer with the same transactionalId which fences the
> current one. {code}
> Sample queries:
>
>
> {code:java}
> CREATE CATALOG table_store_catalog WITH (
> 'type'='table-store',
> 'warehouse'='s3://my-bucket/table-store'
> );
> USE CATALOG table_store_catalog;
> SET 'execution.checkpointing.interval' = '10 s';
> CREATE TABLE word_count_kafka (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> ) WITH (
> 'log.system' = 'kafka',
> 'kafka.bootstrap.servers' = 'broker:9092',
> 'kafka.topic' = 'word_count_log'
> );
> CREATE TEMPORARY TABLE word_table (
> word STRING
> ) WITH (
> 'connector' = 'datagen',
> 'fields.word.length' = '1'
> );
> {code}
>
> And the two INSERT jobs:
> {code:java}
> INSERT INTO word_count_kafka SELECT word, COUNT(*) FROM word_table GROUP BY
> word;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31076) Supports filter predicate in Parquet format of table store
[ https://issues.apache.org/jira/browse/FLINK-31076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31076. Resolution: Fixed > Supports filter predicate in Parquet format of table store > -- > > Key: FLINK-31076 > URL: https://issues.apache.org/jira/browse/FLINK-31076 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Nicholas Jiang >Assignee: Nicholas Jiang >Priority: Major > Fix For: table-store-0.4.0 > > > Parquet format is the main file format of table store, which doesn't support > filter predicate. Filter predicate should also support in Parquet format, not > only the ORC format. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30723) Introduce filter pushdown for parquet format
[ https://issues.apache.org/jira/browse/FLINK-30723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30723. Resolution: Fixed > Introduce filter pushdown for parquet format > > > Key: FLINK-30723 > URL: https://issues.apache.org/jira/browse/FLINK-30723 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Priority: Major > > Introduce filter pushdown for parquet format -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30572) Make parquet as default data file format
[ https://issues.apache.org/jira/browse/FLINK-30572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30572. Resolution: Fixed > Make parquet as default data file format > > > Key: FLINK-30572 > URL: https://issues.apache.org/jira/browse/FLINK-30572 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > - We have done some tests. Parquet is 30% faster. > - After FLINK-30565, Parquet can support complex types and file systems such > as OSS and s3 (decoupled from hadoop filesystem). > - After FLINK-30569, the table can switch formats at will. > Therefore, if detailed and comprehensive tests have been carried out here, we > can use Parquet as the default format. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31224) Add metrics for flink table store
[
https://issues.apache.org/jira/browse/FLINK-31224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31224.
Resolution: Fixed
> Add metrics for flink table store
> -
>
> Key: FLINK-31224
> URL: https://issues.apache.org/jira/browse/FLINK-31224
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
>Affects Versions: table-store-0.3.1
>Reporter: Jun Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.4.0
>
>
> We can divide the metrics of the flink table store into three categories:
> SourceMetrics, WriterMetrics, and CompactionMetrics
> Some metrics are as follows :
> {code:java}
> public class WriterMetrics {
> private final Counter deltaManifestCount;
> private final Counter deltaTotalFileSize;
> private final Counter deltaTotalRowCount;
> private final Counter changelogManifestCount;
> private final Counter changelogTotalFileSize;
> private final Counter changelogTotalRowCount;
> }
> public class SourceMetrics {
> private final Counter assignedSplits;
> private final Counter finishedSplits;
> private final Counter splitReaderFetchCalls;
> }
> public class CompactionMetrics {
> private final Counter beforeCompactionFileCount;
> private final Counter beforeCompactionDeltaTotalFileSize;
> private final Counter beforeCompactionDeltaTotalRowCount;
> private final Counter afterCompactionFileCount;
> private final Counter afterCompactionDeltaTotalFileSize;
> private final Counter afterCompactionDeltaTotalRowCount;
> private final Counter compactChangelogFileCount;
> private final Counter compactChangelogTotalFileSize;
> private final Counter compactChangelogTotalRowCount;
> }
> {code}
>
> My idea is that we may not know which metrics users really need at first. We
> can add some core metrics first, and then we can collect users' needs and add
> some user metrics later.
>
> [~lzljs3620320] [~nicholasjiang] do you have any advice? thanks
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-26465) Optimize SortMergeReader: use loser tree to reduce comparisons
[ https://issues.apache.org/jira/browse/FLINK-26465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-26465. Resolution: Fixed > Optimize SortMergeReader: use loser tree to reduce comparisons > -- > > Key: FLINK-26465 > URL: https://issues.apache.org/jira/browse/FLINK-26465 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Aiden Gong >Priority: Minor > Labels: pull-request-available, stale-assigned > Fix For: table-store-0.4.0 > > > See https://en.wikipedia.org/wiki/K-way_merge_algorithm -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30857) Create table does not create topic with multiple partitions
[
https://issues.apache.org/jira/browse/FLINK-30857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30857.
Resolution: Fixed
> Create table does not create topic with multiple partitions
> ---
>
> Key: FLINK-30857
> URL: https://issues.apache.org/jira/browse/FLINK-30857
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Vicky Papavasileiou
>Priority: Major
>
>
> {code:java}
> CREATE CATALOG table_store_catalog WITH (
> 'type'='table-store',
> 'warehouse'='s3://my-bucket/table-store'
> );
> USE CATALOG table_store_catalog;
> SET 'execution.checkpointing.interval' = '10 s';
> CREATE TABLE word_count_kafka (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> ) WITH (
> 'log.system' = 'kafka',
> 'kafka.bootstrap.servers' = 'broker:9092',
> 'kafka.topic' = 'word_count_log',
> 'bucket'='4'
> );
> {code}
>
> The created topic has only one partition
> {code:java}
> Topic: word_count_log TopicId: udeJwBIkRsSybkf1EerphA PartitionCount: 1
> ReplicationFactor: 1 Configs:
> Topic: word_count_log Partition: 0 Leader: 1 Replicas: 1 Isr:
> 1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30857) Create table does not create topic with multiple partitions
[
https://issues.apache.org/jira/browse/FLINK-30857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706179#comment-17706179
]
Jingsong Lee commented on FLINK-30857:
--
Kafka topic should be created by users..
> Create table does not create topic with multiple partitions
> ---
>
> Key: FLINK-30857
> URL: https://issues.apache.org/jira/browse/FLINK-30857
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Vicky Papavasileiou
>Priority: Major
>
>
> {code:java}
> CREATE CATALOG table_store_catalog WITH (
> 'type'='table-store',
> 'warehouse'='s3://my-bucket/table-store'
> );
> USE CATALOG table_store_catalog;
> SET 'execution.checkpointing.interval' = '10 s';
> CREATE TABLE word_count_kafka (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> ) WITH (
> 'log.system' = 'kafka',
> 'kafka.bootstrap.servers' = 'broker:9092',
> 'kafka.topic' = 'word_count_log',
> 'bucket'='4'
> );
> {code}
>
> The created topic has only one partition
> {code:java}
> Topic: word_count_log TopicId: udeJwBIkRsSybkf1EerphA PartitionCount: 1
> ReplicationFactor: 1 Configs:
> Topic: word_count_log Partition: 0 Leader: 1 Replicas: 1 Isr:
> 1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31039) ChangelogWithKeyFileStoreTableITCase in table store is not stable
[ https://issues.apache.org/jira/browse/FLINK-31039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31039. Resolution: Fixed > ChangelogWithKeyFileStoreTableITCase in table store is not stable > - > > Key: FLINK-31039 > URL: https://issues.apache.org/jira/browse/FLINK-31039 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Priority: Major > > FAILURE! - in > org.apache.flink.table.store.connector.ChangelogWithKeyFileStoreTableITCase > Error: testFullCompactionChangelogProducerStreamingRandom Time elapsed: > 600.077 s <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 60 > milliseconds > at java.lang.Thread.sleep(Native Method) > at > org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:244) > at > org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:114) > at > org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106) > at > org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80) > at > org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:222) > at > org.apache.flink.table.store.connector.ChangelogWithKeyFileStoreTableITCase.checkFullCompactionTestResult(ChangelogWithKeyFileStoreTableITCase.java:395) > at > org.apache.flink.table.store.connector.ChangelogWithKeyFileStoreTableITCase.testFullCompactionChangelogProducerRandom(ChangelogWithKeyFileStoreTableITCase.java:343) > at > org.apache.flink.table.store.connector.ChangelogWithKeyFileStoreTableITCase.testFullCompactionChangelogProducerStreamingRandom(ChangelogWithKeyFileStoreTableITCase.java:300) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:750) > [INFO] > [INFO] Results: > [INFO] > Error: Errors: > Error: > ChangelogWithKeyFileStoreTableITCase.testFullCompactionChangelogProducerStreamingRandom:300->testFullCompactionChangelogProducerRandom:343->checkFullCompactionTestResult:395 > » TestTimedOut > https://github.com/apache/flink-table-store/actions/runs/4161755735/jobs/7200106408 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31011) upgrade hiverunner version
[ https://issues.apache.org/jira/browse/FLINK-31011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31011. Resolution: Fixed > upgrade hiverunner version > -- > > Key: FLINK-31011 > URL: https://issues.apache.org/jira/browse/FLINK-31011 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.1 >Reporter: Jun Zhang >Priority: Major > Fix For: table-store-0.4.0 > > > The current HiveRunner Test framework (version 4) has some bugs. for example, > when we rename a table, it does not rename the location, I test that, the > last version (6) is fine, so we should upgrade the HiveRunner to last version -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31074) Introduce value filter for table store
[ https://issues.apache.org/jira/browse/FLINK-31074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31074. Resolution: Fixed > Introduce value filter for table store > -- > > Key: FLINK-31074 > URL: https://issues.apache.org/jira/browse/FLINK-31074 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Priority: Major > > Currently table store supports filter key/partition to get files from > partition. Besides key stats, table store also has value stats which can be > used for filter too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31013) Session window aggregation cannot trigger window using event time
[
https://issues.apache.org/jira/browse/FLINK-31013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31013.
Resolution: Fixed
> Session window aggregation cannot trigger window using event time
> -
>
> Key: FLINK-31013
> URL: https://issues.apache.org/jira/browse/FLINK-31013
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.4.0
>Reporter: Jane Chan
>Priority: Major
> Fix For: table-store-0.4.0
>
>
> {code:sql}
> -- test against Flink 1.16.0
> create catalog fscat with (
> 'type' = 'table-store',
> 'warehouse' = 'file:///tmp/fscat'
> );
> use catalog fscat;
> create table events (
> `id` int,
> `type` string,
> `date` TIMESTAMP(3),
> watermark for `date` AS `date`);
>
> insert into events
> values (1, 'T1', to_timestamp('2018-01-24', '-MM-dd')),
> (2, 'T1', to_timestamp('2018-01-26', '-MM-dd')),
> (1, 'T2', to_timestamp('2018-01-28', '-MM-dd')),
> (1, 'T2', to_timestamp('2018-01-28', '-MM-dd'));
> -- no output
> select `id`,
> `type`,
> COUNT(1) as event_cnt,
> session_start(`date`, interval '1' DAY) as ss,
> session_end(`date`, interval '1' DAY) as se
> from events group by `id`, `type`, session(`date`, interval '1' DAY);
> -- explain plan
> == Abstract Syntax Tree ==
> LogicalProject(id=[$0], type=[$1], event_cnt=[$3], ss=[SESSION_START($2)],
> se=[SESSION_END($2)])
> +- LogicalAggregate(group=[{0, 1, 2}], event_cnt=[COUNT()])
> +- LogicalProject(id=[$0], type=[$1], $f2=[$SESSION($2, 8640:INTERVAL
> DAY)])
> +- LogicalWatermarkAssigner(rowtime=[date], watermark=[$2])
> +- LogicalTableScan(table=[[fscat, default, events]])
> == Optimized Physical Plan ==
> Calc(select=[id, type, event_cnt, w$start AS ss, w$end AS se])
> +- GroupWindowAggregate(groupBy=[id, type], window=[SessionGroupWindow('w$,
> date, 8640)], properties=[w$start, w$end, w$rowtime, w$proctime],
> select=[id, type, COUNT(*) AS event_cnt, start('w$) AS w$start, end('w$) AS
> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime])
> +- Exchange(distribution=[hash[id, type]])
> +- TableSourceScan(table=[[fscat, default, events, watermark=[date]]],
> fields=[id, type, date])
> == Optimized Execution Plan ==
> Calc(select=[id, type, event_cnt, w$start AS ss, w$end AS se])
> +- GroupWindowAggregate(groupBy=[id, type], window=[SessionGroupWindow('w$,
> date, 8640)], properties=[w$start, w$end, w$rowtime, w$proctime],
> select=[id, type, COUNT(*) AS event_cnt, start('w$) AS w$start, end('w$) AS
> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime])
> +- Exchange(distribution=[hash[id, type]])
> +- TableSourceScan(table=[[fscat, default, events, watermark=[date]]],
> fields=[id, type, date])
> -- however, if switch to filesystem source, the window can be triggered
> normally
> CREATE TEMPORARY TABLE `fscat`.`default`.`event_file_source` (
> `id` INT,
> `type` VARCHAR(2147483647),
> `date` TIMESTAMP(3),
> WATERMARK FOR `date` AS `date`
> ) WITH (
> 'format' = 'csv',
> 'path' = '/tmp/events.csv',
> 'source.monitor-interval' = '1 min',
> 'connector' = 'filesystem'
> );
> // cat events.csv
> 1,T1,2018-01-24 00:00:00.000
> 2,T1,2018-01-26 00:00:00.000
> 1,T2,2018-01-28 00:00:00.000
> 1,T2,2018-01-28 00:00:00.000
> -- same query using filesystem source
> select `id`, `type`, COUNT(1) as event_cnt, session_start(`date`, interval
> '1' DAY) as ss, session_end(`date`, interval '1' DAY) as se from
> event_file_source group by `id`, `type`, session(`date`, interval '1' DAY);
> -- output
> id type event_cnt
> ss se
> 1 T1 1 2018-01-24
> 00:00:00.000 2018-01-25 00:00:00.000
> 2 T1 1 2018-01-26
> 00:00:00.000 2018-01-27 00:00:00.000{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30580) [umbrella] Refactor tests for table store
[ https://issues.apache.org/jira/browse/FLINK-30580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30580. Resolution: Fixed > [umbrella] Refactor tests for table store > - > > Key: FLINK-30580 > URL: https://issues.apache.org/jira/browse/FLINK-30580 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > This is a umbrella issue to improve tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31129) Introduce FlinkEmbeddedHiveRunner for junit5 in table store
[ https://issues.apache.org/jira/browse/FLINK-31129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31129. Resolution: Fixed > Introduce FlinkEmbeddedHiveRunner for junit5 in table store > --- > > Key: FLINK-31129 > URL: https://issues.apache.org/jira/browse/FLINK-31129 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Priority: Major > > Introduce FlinkEmbeddedHiveRunner for junit5 in table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30603) CompactActionITCase in table store is unstable
[ https://issues.apache.org/jira/browse/FLINK-30603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30603. Resolution: Fixed > CompactActionITCase in table store is unstable > -- > > Key: FLINK-30603 > URL: https://issues.apache.org/jira/browse/FLINK-30603 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0, table-store-0.3.1 > > > https://github.com/apache/flink-table-store/actions/runs/3927960511/jobs/6715071149 > Error: Failures: > Error:CompactActionITCase.testStreamingCompact:193 expected:<[+I > 1|100|15|20221208, +I 1|100|15|20221209]> but was:<[+I 1|100|15|20221208]> > [INFO] > Error: Tests run: 221, Failures: 1, Errors: 0, Skipped: 4 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30740) INSERT to Kafka does not work when Kafka config auto.create.topics.enabled is set to false
[
https://issues.apache.org/jira/browse/FLINK-30740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30740.
Resolution: Fixed
https://github.com/apache/incubator-paimon/issues/740
> INSERT to Kafka does not work when Kafka config auto.create.topics.enabled is
> set to false
> --
>
> Key: FLINK-30740
> URL: https://issues.apache.org/jira/browse/FLINK-30740
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.3.0
>Reporter: Bruno Cadonna
>Priority: Major
>
> If I use Kafka as the log system and set {{auto.create.topics.enabled}} to
> false in Kafka INSERTs do not work.
> Steps to reproduce:
> # Start a Kafka broker and set {{auto.create.topics.enabled}} to false
> # Issue the following statements
> {code:sql}
> CREATE CATALOG table_store_catalog WITH (
>'type'='table-store',
>'warehouse'=
> );
> USE CATALOG table_store_catalog;
> CREATE TABLE word_count (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> ) WITH (
>'log.system' = 'kafka',
>'kafka.bootstrap.servers' = ,
>'kafka.topic' = 'test-topic,
>'log.consistency' = 'eventual'
> );
>
> INSERT INTO word_count VALUES ('foo', 1);
> {code}
> The task manager logs show:
> {code}
> flink-sandbox-taskmanager-1 | 2023-01-18 12:46:17,085 WARN
> org.apache.flink.table.store.shaded.org.apache.kafka.clients.NetworkClient []
> - [Producer clientId=producer-1] Error while fetching metadata with
> correlation id 544 : {test-topic=UNKNOWN_TOPIC_OR_PARTITION}
> {code}
> The INSERT job on the task manager fails with
> {code:java}
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:301)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:291)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:282)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:739)
> at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:443)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:304)
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:302)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at akka.actor.Actor.aroundReceive(Actor.scala:537)
> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> at
[jira] [Closed] (FLINK-28661) Introduce generic mode for table store catalog
[ https://issues.apache.org/jira/browse/FLINK-28661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28661. Resolution: Fixed > Introduce generic mode for table store catalog > -- > > Key: FLINK-28661 > URL: https://issues.apache.org/jira/browse/FLINK-28661 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > - Introduce a generic option for catalog. > - Table Store catalog can store Flink generic tables when generic is true. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30248) Spark writer supports insert overwrite
[ https://issues.apache.org/jira/browse/FLINK-30248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30248. Resolution: Fixed > Spark writer supports insert overwrite > -- > > Key: FLINK-30248 > URL: https://issues.apache.org/jira/browse/FLINK-30248 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27628) Table Store records and fetches incorrect results with NaN
[
https://issues.apache.org/jira/browse/FLINK-27628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-27628.
Resolution: Fixed
> Table Store records and fetches incorrect results with NaN
> --
>
> Key: FLINK-27628
> URL: https://issues.apache.org/jira/browse/FLINK-27628
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.2.0
>Reporter: Caizhi Weng
>Priority: Major
>
> Use the following test data and SQL to reproduce this issue.
> gao.csv:
> {code}
> 1.0,2.0,aaa
> 0.0,0.0,aaa
> 1.0,1.0,aaa
> 0.0,0.0,aaa
> 1.0,0.0,aaa
> 0.0,0.0,aaa
> -1.0,0.0,aaa
> 1.0,-1.0,aaa
> 1.0,-2.0,aaa
> {code}
> Flink SQL:
> {code}
> Flink SQL> create table T ( a double, b double, c string ) WITH ( 'connector'
> = 'filesystem', 'path' = '/tmp/gao.csv', 'format' = 'csv' );
> [INFO] Execute statement succeed.
> Flink SQL> create table S ( a string, b double ) WITH ( 'path' = '/tmp/store'
> );
> [INFO] Execute statement succeed.
> Flink SQL> insert into S select c, a / b from T;
> [INFO] Submitting SQL update statement to the cluster...
> [INFO] SQL update statement has been successfully submitted to the cluster:
> Job ID: 851d7b3c233061733bdabbf30f20d16f
> Flink SQL> select c, a / b from T;
> +-+---+
> | c |EXPR$1 |
> +-+---+
> | aaa | 0.5 |
> | aaa | NaN |
> | aaa | 1.0 |
> | aaa | NaN |
> | aaa | Infinity |
> | aaa | NaN |
> | aaa | -Infinity |
> | aaa | -1.0 |
> | aaa | -0.5 |
> +-+---+
> 9 rows in set
> Flink SQL> select * from S;
> +-+---+
> | a | b |
> +-+---+
> | aaa | -Infinity |
> | aaa | -Infinity |
> | aaa | -Infinity |
> | aaa | -Infinity |
> | aaa | -Infinity |
> | aaa | -Infinity |
> | aaa | -Infinity |
> | aaa | -1.0 |
> | aaa | -0.5 |
> +-+---+
> 9 rows in set
> {code}
> Note that this issue may also affect {{FieldStatsCollector}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29953) Get rid of flink-connector-hive dependency in flink-table-store-hive
[ https://issues.apache.org/jira/browse/FLINK-29953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29953. Resolution: Fixed > Get rid of flink-connector-hive dependency in flink-table-store-hive > > > Key: FLINK-29953 > URL: https://issues.apache.org/jira/browse/FLINK-29953 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > It is unnecessary for the tablestore to rely on it in the test. Its > incompatible modifications will make the tablestore troublesome. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29756) Support materialized column to improve query performance for complex types
[ https://issues.apache.org/jira/browse/FLINK-29756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29756. Resolution: Fixed https://github.com/apache/incubator-paimon/issues/735 > Support materialized column to improve query performance for complex types > -- > > Key: FLINK-29756 > URL: https://issues.apache.org/jira/browse/FLINK-29756 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Nicholas Jiang >Priority: Minor > Fix For: table-store-0.4.0 > > > In the world of data warehouse, it is very common to use one or more columns > from a complex type such as a map, or to put many subfields into it. These > operations can greatly affect query performance because: > # These operations are very wasteful IO. For example, if we have a field > type of Map, which contains dozens of subfields, we need to read the entire > column when reading this column. And Spark will traverse the entire map to > get the value of the target key. > # Cannot take advantage of vectorized reads when reading nested type columns. > # Filter pushdown cannot be used when reading nested columns. > It is necessary to introduce the materialized column feature in Flink Table > Store, which transparently solves the above problems of arbitrary columnar > storage (not just Parquet). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29490) Timestamp LTZ is unsupported in table store
[ https://issues.apache.org/jira/browse/FLINK-29490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29490. Resolution: Fixed > Timestamp LTZ is unsupported in table store > > > Key: FLINK-29490 > URL: https://issues.apache.org/jira/browse/FLINK-29490 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Due to orc format limitation, timestamp ltz is unsupported now. We should fix > this, and validate this type cross multiple engines (hive spark trino). > We need to careful about time zone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30326) SQL Client should not create table directory structure
[ https://issues.apache.org/jira/browse/FLINK-30326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30326. Resolution: Fixed I close this, I think it is difficult to implement in paimon, if necessary, please raise the issue again in paimon. > SQL Client should not create table directory structure > -- > > Key: FLINK-30326 > URL: https://issues.apache.org/jira/browse/FLINK-30326 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Alex Sorokoumov >Priority: Major > > The SQL client creates the schema part of the directory structure for new > tables. This behavior is incorrect because the SQL client could be attached > to a SQL gateway that does not necessarily have access to DFSes for all > catalogs/tables. > I propose to change the behavior such that FTS sink creates the schema, > either on the fly if `auto-create=true` or just as a separate job if > `auto-create=false`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30590) Remove set default value manually for table options
[ https://issues.apache.org/jira/browse/FLINK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30590. Resolution: Fixed > Remove set default value manually for table options > --- > > Key: FLINK-30590 > URL: https://issues.apache.org/jira/browse/FLINK-30590 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Priority: Major > > Remove set default value manually in `CoreOptions.setDefaultValues` which may > cause wrong error information and it's not needed anymore -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30317) Create table in FTS catalog with s3 warehouse throws DatabaseNotExistException
[
https://issues.apache.org/jira/browse/FLINK-30317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30317.
Resolution: Fixed
> Create table in FTS catalog with s3 warehouse throws DatabaseNotExistException
> --
>
> Key: FLINK-30317
> URL: https://issues.apache.org/jira/browse/FLINK-30317
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Alex Sorokoumov
>Priority: Major
>
> {noformat}
> Flink SQL> CREATE CATALOG my_catalog WITH (
> > 'type'='table-store',
> > 'warehouse'='s3://bucket/my-tablestore'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> USE CATALOG my_catalog;
> [INFO] Execute statement succeed.
> Flink SQL> CREATE TABLE word_count (
> > word STRING PRIMARY KEY NOT ENFORCED,
> > cnt BIGINT
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.catalog.exceptions.DatabaseNotExistException: Database
> default does not exist in Catalog my_catalog. {noformat}
> Creating the table in the default catalog works though:
> {noformat}
> Flink SQL> use catalog default_catalog;
> [INFO] Execute statement succeed.
> Flink SQL> CREATE TABLE word_count (
> > word STRING PRIMARY KEY NOT ENFORCED,
> > cnt BIGINT
> > ) WITH (
> >'connector'='table-store',
> >'path'='s3://bucket/my-tablestore',
> >'auto-create'='true'
> > );
> [INFO] Execute statement succeed.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30317) Create table in FTS catalog with s3 warehouse throws DatabaseNotExistException
[
https://issues.apache.org/jira/browse/FLINK-30317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706174#comment-17706174
]
Jingsong Lee commented on FLINK-30317:
--
https://github.com/apache/incubator-paimon/issues/733
> Create table in FTS catalog with s3 warehouse throws DatabaseNotExistException
> --
>
> Key: FLINK-30317
> URL: https://issues.apache.org/jira/browse/FLINK-30317
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Alex Sorokoumov
>Priority: Major
>
> {noformat}
> Flink SQL> CREATE CATALOG my_catalog WITH (
> > 'type'='table-store',
> > 'warehouse'='s3://bucket/my-tablestore'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> USE CATALOG my_catalog;
> [INFO] Execute statement succeed.
> Flink SQL> CREATE TABLE word_count (
> > word STRING PRIMARY KEY NOT ENFORCED,
> > cnt BIGINT
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.catalog.exceptions.DatabaseNotExistException: Database
> default does not exist in Catalog my_catalog. {noformat}
> Creating the table in the default catalog works though:
> {noformat}
> Flink SQL> use catalog default_catalog;
> [INFO] Execute statement succeed.
> Flink SQL> CREATE TABLE word_count (
> > word STRING PRIMARY KEY NOT ENFORCED,
> > cnt BIGINT
> > ) WITH (
> >'connector'='table-store',
> >'path'='s3://bucket/my-tablestore',
> >'auto-create'='true'
> > );
> [INFO] Execute statement succeed.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30390) Ensure that no compaction is in progress before closing the writer
[ https://issues.apache.org/jira/browse/FLINK-30390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30390. Resolution: Fixed > Ensure that no compaction is in progress before closing the writer > -- > > Key: FLINK-30390 > URL: https://issues.apache.org/jira/browse/FLINK-30390 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > When the writer does not generate a new submission file, it will be closed. > (In AbstractFileStoreWrite) However, at this time, there may be asynchronous > interactions that have not been completed and are forced to close, which will > cause some strange exceptions to be printed in the log. > We can avoid this situation, ensure that no compaction is in progress before > closing the writer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30564) Select from a new table with Kafka LogStore crashes with UnknownTopicOrPartitionException
[
https://issues.apache.org/jira/browse/FLINK-30564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30564.
Resolution: Fixed
> Select from a new table with Kafka LogStore crashes with
> UnknownTopicOrPartitionException
> -
>
> Key: FLINK-30564
> URL: https://issues.apache.org/jira/browse/FLINK-30564
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Alex Sorokoumov
>Priority: Major
>
> Selecting from newly created table that uses Kafka as a Log Store creates a
> job that crash-loops with {{UnknownTopicOrPartitionException: This server
> does not host this topic-partition}} exception. This happens because neither
> {{CREATE TABLE}} nor {{SELECT FROM}} create the underlying topic.
> Steps to reproduce:
> {noformat}
> CREATE TABLE word_count (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> ) WITH (
> 'connector' = 'table-store',
> 'path' = 's3://my-bucket/table-store',
> 'log.system' = 'kafka',
> 'kafka.bootstrap.servers' = 'broker:9092',
> 'kafka.topic' = 'word_count_log',
> 'auto-create' = 'true',
> 'log.changelog-mode' = 'all',
> 'log.consistency' = 'transactional'
> );
> SELECT * FROM word_count; {noformat}
>
> JM logs:
> {noformat}
> flink | 2023-01-04 23:27:24,292 ERROR
> org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext [] -
> Exception while handling result from async call in SourceCoordinator-Source:
> word_count[1]. Triggering job failover.
> flink | org.apache.flink.util.FlinkRuntimeException: Failed to list
> subscribed topic partitions due to
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.KafkaSourceEnumerator.checkPartitionChanges(KafkaSourceEnumerator.java:234)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$null$1(ExecutorNotifier.java:83)
> ~[flink-dist-1.16.0.jar:1.16.0]
> flink | at
> org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
> [flink-dist-1.16.0.jar:1.16.0]
> flink | at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_352]
> flink | at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_352]
> flink | at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]
> flink | Caused by: java.lang.RuntimeException: Failed to get
> metadata for topics [word_count_log].
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.subscriber.KafkaSubscriberUtils.getTopicMetadata(KafkaSubscriberUtils.java:47)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.subscriber.TopicListSubscriber.getSubscribedTopicPartitions(TopicListSubscriber.java:52)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.KafkaSourceEnumerator.getSubscribedTopicPartitions(KafkaSourceEnumerator.java:219)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$2(ExecutorNotifier.java:80)
> ~[flink-dist-1.16.0.jar:1.16.0]
> flink | ... 7 more
> flink | Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.table.store.shaded.org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
> This server does not host this topic-partition. {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30394) [umbrella] Refactor filesystem support in table store
[ https://issues.apache.org/jira/browse/FLINK-30394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30394. Resolution: Fixed > [umbrella] Refactor filesystem support in table store > - > > Key: FLINK-30394 > URL: https://issues.apache.org/jira/browse/FLINK-30394 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > - Let other computing engines, such as hive, spark, trino, support object > storage file systems, such as OSS and s3. > - Let table store access different file systems from Flink cluster according > to configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30564) Select from a new table with Kafka LogStore crashes with UnknownTopicOrPartitionException
[
https://issues.apache.org/jira/browse/FLINK-30564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706173#comment-17706173
]
Jingsong Lee commented on FLINK-30564:
--
https://github.com/apache/incubator-paimon/issues/732
> Select from a new table with Kafka LogStore crashes with
> UnknownTopicOrPartitionException
> -
>
> Key: FLINK-30564
> URL: https://issues.apache.org/jira/browse/FLINK-30564
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Alex Sorokoumov
>Priority: Major
>
> Selecting from newly created table that uses Kafka as a Log Store creates a
> job that crash-loops with {{UnknownTopicOrPartitionException: This server
> does not host this topic-partition}} exception. This happens because neither
> {{CREATE TABLE}} nor {{SELECT FROM}} create the underlying topic.
> Steps to reproduce:
> {noformat}
> CREATE TABLE word_count (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> ) WITH (
> 'connector' = 'table-store',
> 'path' = 's3://my-bucket/table-store',
> 'log.system' = 'kafka',
> 'kafka.bootstrap.servers' = 'broker:9092',
> 'kafka.topic' = 'word_count_log',
> 'auto-create' = 'true',
> 'log.changelog-mode' = 'all',
> 'log.consistency' = 'transactional'
> );
> SELECT * FROM word_count; {noformat}
>
> JM logs:
> {noformat}
> flink | 2023-01-04 23:27:24,292 ERROR
> org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext [] -
> Exception while handling result from async call in SourceCoordinator-Source:
> word_count[1]. Triggering job failover.
> flink | org.apache.flink.util.FlinkRuntimeException: Failed to list
> subscribed topic partitions due to
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.KafkaSourceEnumerator.checkPartitionChanges(KafkaSourceEnumerator.java:234)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$null$1(ExecutorNotifier.java:83)
> ~[flink-dist-1.16.0.jar:1.16.0]
> flink | at
> org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
> [flink-dist-1.16.0.jar:1.16.0]
> flink | at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_352]
> flink | at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_352]
> flink | at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_352]
> flink | at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]
> flink | Caused by: java.lang.RuntimeException: Failed to get
> metadata for topics [word_count_log].
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.subscriber.KafkaSubscriberUtils.getTopicMetadata(KafkaSubscriberUtils.java:47)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.subscriber.TopicListSubscriber.getSubscribedTopicPartitions(TopicListSubscriber.java:52)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.table.store.shaded.connector.kafka.source.enumerator.KafkaSourceEnumerator.getSubscribedTopicPartitions(KafkaSourceEnumerator.java:219)
> ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
> flink | at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$2(ExecutorNotifier.java:80)
> ~[flink-dist-1.16.0.jar:1.16.0]
> flink | ... 7 more
> flink | Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.table.store.shaded.org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
> This server does not host this topic-partition. {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30000) Introduce FileSystemFactory to create FileSystem from custom configuration
[ https://issues.apache.org/jira/browse/FLINK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-3. Resolution: Fixed > Introduce FileSystemFactory to create FileSystem from custom configuration > -- > > Key: FLINK-3 > URL: https://issues.apache.org/jira/browse/FLINK-3 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, table store uses static Flink FileSystem. This can not support: > 1. Use another FileSystem different from checkpoint FileSystem. > 2. Use FileSystem in Hive and Spark from custom configuration instead of > using FileSystem.initialize. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30039) Insert overwrite show excetion that query and insert schema do not match
[ https://issues.apache.org/jira/browse/FLINK-30039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30039. Resolution: Fixed > Insert overwrite show excetion that query and insert schema do not match > > > Key: FLINK-30039 > URL: https://issues.apache.org/jira/browse/FLINK-30039 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: 1.16.0 > Environment: Flink 1.16.0 >Reporter: StarBoy1005 >Priority: Critical > Attachments: screenshot-2.png, screenshot-3.png > > > After change a table's bucket num in table store,I used a insert overwrite > sql to reorganize records to new files (I guess form old orc file to the new). > Everytime if select column contains the partition column,then get the next > exception: > !screenshot-2.png! > When exclude the partition field,it do perform the overwrite. > I wonder the “EXPR$3: STRING NOT NULL” means what , -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30031) flink table store run abnormally when shade flink
[
https://issues.apache.org/jira/browse/FLINK-30031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30031.
Resolution: Fixed
> flink table store run abnormally when shade flink
> -
>
> Key: FLINK-30031
> URL: https://issues.apache.org/jira/browse/FLINK-30031
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: 1.15.0, table-store-0.2.1
>Reporter: zhenzhenhua
>Priority: Major
>
> {color:#172b4d}I try to sink flink-table-store in Apache SeaTunnel, SeaTunnel
> use the Flink version is 13.6.{color}
> {color:#172b4d}To avoid flink conflict,I use maven-shade-plugin plugin to
> shade flink dependency.{color}
> {color:#172b4d}However, runing build jar to write flink-table-store occur
> error,throw exception:{color}
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/flink/table/store/codegen/CodeGenerator
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:127)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClass(ComponentClassLoader.java:106)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> org.apache.flink.table.store.codegen.CodeGenLoader.discover(CodeGenLoader.java:123)
> at
> org.apache.flink.table.store.codegen.CodeGenUtils.generateRecordComparator(CodeGenUtils.java:65)
> at
> org.apache.flink.table.store.file.utils.KeyComparatorSupplier.(KeyComparatorSupplier.java:40)
> at
> org.apache.flink.table.store.file.KeyValueFileStore.(KeyValueFileStore.java:59)
> at
> org.apache.flink.table.store.table.ChangelogWithKeyFileStoreTable.(ChangelogWithKeyFileStoreTable.java:103)
> at
> org.apache.flink.table.store.table.FileStoreTableFactory.create(FileStoreTableFactory.java:72)
> at
> org.apache.flink.table.store.table.FileStoreTableFactory.create(FileStoreTableFactory.java:50)
> at org.example.TestWrite.main(TestWrite.java:24)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.table.store.codegen.CodeGenerator
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:127)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClass(ComponentClassLoader.java:106)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 27 more
> {code}
>
>
> {code:java}
> // pom.xml
>
>
> org.apache.flink
> flink-table-store-core
> 0.2.1
>
>
> org.apache.flink
> flink-table-store-format
> 0.2.1
>
>
> org.apache.flink
> flink-table-common
> 1.15.0
>
>
> org.apache.flink
> flink-table-runtime
> 1.15.0
>
>
> org.apache.flink
> flink-connector-base
> 1.15.0
>
>
> org.apache.flink
> flink-shaded-jackson
> 2.12.1-13.0
>
>
> org.apache.flink
> flink-shaded-hadoop-2-uber
> 2.7.5-10.0
>
>
> org.apache.commons
> commons-lang3
> 3.3.2
>
>
> log4j
> log4j
> 1.2.17
>
>
>
>
>
> org.apache.maven.plugins
> maven-shade-plugin
> 3.1.1
>
>
> false
>
>
>
[jira] [Closed] (FLINK-29630) Junit 5.8.1 run unit test with temporary directory will occur Failed to delete temp directory.
[ https://issues.apache.org/jira/browse/FLINK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29630. Resolution: Fixed > Junit 5.8.1 run unit test with temporary directory will occur Failed to > delete temp directory. > -- > > Key: FLINK-29630 > URL: https://issues.apache.org/jira/browse/FLINK-29630 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Aiden Gong >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > Attachments: image-2022-10-14-09-12-33-903.png > > > Junit 5.8.1 run unit test with temporary directory will occur Failed to > delete temp directory. > My local : > windows 10 > jdk1.8 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-26937) Introduce Leveled compression for LSM
[ https://issues.apache.org/jira/browse/FLINK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-26937. Resolution: Fixed > Introduce Leveled compression for LSM > - > > Key: FLINK-26937 > URL: https://issues.apache.org/jira/browse/FLINK-26937 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: xingyuan cheng >Priority: Minor > Fix For: table-store-0.4.0 > > > Currently ORC is all ZLIB compression by default, in fact the files at level > 0, will definitely be rewritten and we can have different compression for > different levels. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29988) Improve upper case fields for hive metastore
[ https://issues.apache.org/jira/browse/FLINK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29988. Resolution: Fixed > Improve upper case fields for hive metastore > > > Key: FLINK-29988 > URL: https://issues.apache.org/jira/browse/FLINK-29988 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Fang Yong >Priority: Major > Labels: pull-request-available > > If the fields in the fts table are uppercase, there will be a mismatched > exception when used in the Hive. > 1. If it is not supported at the beginning, throw an exception when flink > creates a table to the hive metastore. > 2. If it is supported, so that no error is reported in the whole process, but > save lower case in hive metastore. We can check columns with the same name > when creating a table in Flink with hive metastore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27814) Add an abstraction layer for connectors to read and write row data instead of key-values
[
https://issues.apache.org/jira/browse/FLINK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-27814.
Resolution: Fixed
> Add an abstraction layer for connectors to read and write row data instead of
> key-values
>
>
> Key: FLINK-27814
> URL: https://issues.apache.org/jira/browse/FLINK-27814
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
>Affects Versions: table-store-0.2.0
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> Currently {{FileStore}} exposes an interface for reading and writing
> {{KeyValue}}. However connectors may have different ways to change a
> {{RowData}} to {{KeyValue}} under different {{WriteMode}}. This results in
> lots of {{if...else...}} branches and duplicated code.
> We need to add an abstraction layer for connectors to read and write row data
> instead of key-values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-27548) Improve quick-start of table store
[ https://issues.apache.org/jira/browse/FLINK-27548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-27548. Resolution: Fixed > Improve quick-start of table store > -- > > Key: FLINK-27548 > URL: https://issues.apache.org/jira/browse/FLINK-27548 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: ZhuoYu Chen >Priority: Minor > Fix For: table-store-0.4.0 > > > When the quick-start is completed, the stream job needs to be killed on the > flink page and the table needs to be dropped. > But the exiting of the stream job is asynchronous and we need to wait a while > between these two actions. Otherwise there will be residue in the file > directory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29636) Add micro benchmark module in flink table store
[ https://issues.apache.org/jira/browse/FLINK-29636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29636. Resolution: Fixed > Add micro benchmark module in flink table store > --- > > Key: FLINK-29636 > URL: https://issues.apache.org/jira/browse/FLINK-29636 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > > Currently there's a `flink-table-store-benchmark` to run a query in flink > cluster and collect metrics to measure the performance of > `flink-table-store`. There're some key operation steps such as > read/write/compaction in `flink-table-store`, we should add a > `flink-table-store-micro-benchmark` for them -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29290) Produce changelog during full compaction in Table Store
[ https://issues.apache.org/jira/browse/FLINK-29290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29290. Resolution: Fixed > Produce changelog during full compaction in Table Store > --- > > Key: FLINK-29290 > URL: https://issues.apache.org/jira/browse/FLINK-29290 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > > Currently Table Store only produces changelog directly from input. Some > downstream systems, however, require complete changelogs including both > UPDATE_BEFORE and UPDATE_AFTER messages. > We can only get these information during full compaction, so we should add a > feature to produce changelog during full compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29746) Add workflow in github for micro benchmarks
[ https://issues.apache.org/jira/browse/FLINK-29746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29746. Resolution: Fixed > Add workflow in github for micro benchmarks > --- > > Key: FLINK-29746 > URL: https://issues.apache.org/jira/browse/FLINK-29746 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Add workflow in github for micro benchmarks project -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28009) Optimize data split
[ https://issues.apache.org/jira/browse/FLINK-28009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28009. Resolution: Fixed > Optimize data split > --- > > Key: FLINK-28009 > URL: https://issues.apache.org/jira/browse/FLINK-28009 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.1.0 >Reporter: xingyuan cheng >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > Optimizing split data logic for large data volumes using parallel streams > https://github.com/apache/flink-table-store/pull/151 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29331) pre-aggregated merge supports changelog inputs
[ https://issues.apache.org/jira/browse/FLINK-29331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29331. Resolution: Fixed > pre-aggregated merge supports changelog inputs > -- > > Key: FLINK-29331 > URL: https://issues.apache.org/jira/browse/FLINK-29331 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > In FLINK-27626 , we have supported pre-agg merge, but no changelog inputs > support. > We can support changelog inputs for some function, like sum/count. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28718) SinkSavepointITCase.testRecoverFromSavepoint is unstable
[
https://issues.apache.org/jira/browse/FLINK-28718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-28718.
Resolution: Fixed
> SinkSavepointITCase.testRecoverFromSavepoint is unstable
>
>
> Key: FLINK-28718
> URL: https://issues.apache.org/jira/browse/FLINK-28718
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: table-store-0.4.0
>
>
> https://github.com/apache/flink-table-store/runs/7537817210?check_suite_focus=true
> {code:java}
> Error: Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 185.274 s <<< FAILURE! - in
> org.apache.flink.table.store.connector.sink.SinkSavepointITCase
> Error: testRecoverFromSavepoint Time elapsed: 180.157 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 18
> milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.table.store.connector.sink.SinkSavepointITCase.testRecoverFromSavepoint(SinkSavepointITCase.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:750)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-28690) UpdateSchema#fromCatalogTable lost column comment
[ https://issues.apache.org/jira/browse/FLINK-28690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28690. Resolution: Fixed > UpdateSchema#fromCatalogTable lost column comment > - > > Key: FLINK-28690 > URL: https://issues.apache.org/jira/browse/FLINK-28690 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.2.0 >Reporter: Jane Chan >Assignee: Jane Chan >Priority: Not a Priority > Fix For: table-store-0.4.0 > > > The reason is that > org.apache.flink.table.api.TableSchema#toPhysicalRowDataType lost column > comments, which leads to comparison failure in > AbstractTableStoreFactory#buildFileStoreTable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29080) Migrate all tests from managed table to catalog-based tests
[ https://issues.apache.org/jira/browse/FLINK-29080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29080. Resolution: Fixed > Migrate all tests from managed table to catalog-based tests > --- > > Key: FLINK-29080 > URL: https://issues.apache.org/jira/browse/FLINK-29080 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jane Chan >Priority: Major > Fix For: table-store-0.4.0 > > > To get rid of ManagedTableFactory and enable test on -Pflink-1.14 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29243) Introduce a COW mode for table store
[ https://issues.apache.org/jira/browse/FLINK-29243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29243. Resolution: Fixed changelog producer = full-compaction is the cow mode. > Introduce a COW mode for table store > > > Key: FLINK-29243 > URL: https://issues.apache.org/jira/browse/FLINK-29243 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, we can combine the COW (Copy on write) mode by three options, but > the combination may not be optimal and tedious. > We can introduce a option that specifically turns on the mode of COW. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29272) Document DataStream API (DataStream to Table) for table store
[ https://issues.apache.org/jira/browse/FLINK-29272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29272. Resolution: Fixed > Document DataStream API (DataStream to Table) for table store > - > > Key: FLINK-29272 > URL: https://issues.apache.org/jira/browse/FLINK-29272 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > We can have documentation to describe how to convert from DataStream to Table > to write to TableStore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29820) Support to read data from table store with specify snapshot id
[ https://issues.apache.org/jira/browse/FLINK-29820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702211#comment-17702211 ] Jingsong Lee commented on FLINK-29820: -- We have already supported this. > Support to read data from table store with specify snapshot id > -- > > Key: FLINK-29820 > URL: https://issues.apache.org/jira/browse/FLINK-29820 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Currently our queries can only read the latest snapshot from table store. We > need to support reading the data of the specified snapshot, in this way, we > can read historical versions or compare data across versions as needed. In > [#FLINK-29735] we have supported to query snapshots with `SELECT * FROM > MyTable$snapshots`, we can query data from table `wordcount` with `snapshot > 10` as follows: > `SELECT * FROM wordcount$snapshot$10 WHERE count>10` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29820) Support to read data from table store with specify snapshot id
[ https://issues.apache.org/jira/browse/FLINK-29820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29820. Resolution: Fixed > Support to read data from table store with specify snapshot id > -- > > Key: FLINK-29820 > URL: https://issues.apache.org/jira/browse/FLINK-29820 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Currently our queries can only read the latest snapshot from table store. We > need to support reading the data of the specified snapshot, in this way, we > can read historical versions or compare data across versions as needed. In > [#FLINK-29735] we have supported to query snapshots with `SELECT * FROM > MyTable$snapshots`, we can query data from table `wordcount` with `snapshot > 10` as follows: > `SELECT * FROM wordcount$snapshot$10 WHERE count>10` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29823) Support get schema for table snapshot
[ https://issues.apache.org/jira/browse/FLINK-29823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29823. Resolution: Fixed > Support get schema for table snapshot > - > > Key: FLINK-29823 > URL: https://issues.apache.org/jira/browse/FLINK-29823 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > > Support to create schema from table snapshot with given snapshot id -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29828) Support to read data from table with given snapshot id
[ https://issues.apache.org/jira/browse/FLINK-29828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29828. Resolution: Fixed > Support to read data from table with given snapshot id > -- > > Key: FLINK-29828 > URL: https://issues.apache.org/jira/browse/FLINK-29828 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Support reading data from table with given snapshot id from table, users can > query table data as follows: > `SELECT t1.word, t1.cnt, t2.cnt FROM word_count$snapshot$10 t1 JOIN > word_count$snapshot$11 t2 ON t1.word = t2.word and t1.cnt != t2.cnt` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29882) LargeDataITCase is not stable
[ https://issues.apache.org/jira/browse/FLINK-29882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29882. Resolution: Fixed > LargeDataITCase is not stable > - > > Key: FLINK-29882 > URL: https://issues.apache.org/jira/browse/FLINK-29882 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jane Chan >Priority: Major > Fix For: table-store-0.4.0 > > > https://github.com/apache/flink-table-store/actions/runs/3391781964/jobs/5637271002 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29832) Improve switch to default database in docs
[ https://issues.apache.org/jira/browse/FLINK-29832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29832. Resolution: Fixed > Improve switch to default database in docs > -- > > Key: FLINK-29832 > URL: https://issues.apache.org/jira/browse/FLINK-29832 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > Attachments: image-2022-11-01-16-40-47-539.png > > > `FlinkCatalogFactory` creates a default database named `default` in table > store. The `default` is a keyword in SQL, and when we create a new database, > we cant execute `use default` to switch to `default` directly. We can switch > to default database "use `default`;" in flink table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30227) Add LeafEmptyFunction for predicate without fields
[ https://issues.apache.org/jira/browse/FLINK-30227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30227. Resolution: Fixed > Add LeafEmptyFunction for predicate without fields > -- > > Key: FLINK-30227 > URL: https://issues.apache.org/jira/browse/FLINK-30227 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Fang Yong >Priority: Major > > In addition to `LeafBinaryFunction` and `LeafUnaryFunction`, we should add > `LeafEmptyFunction` for predicate -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30080) Introduce public programming api and dependency jar for table store
[ https://issues.apache.org/jira/browse/FLINK-30080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30080. Resolution: Fixed > Introduce public programming api and dependency jar for table store > --- > > Key: FLINK-30080 > URL: https://issues.apache.org/jira/browse/FLINK-30080 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > > Users need to access tablestore through programming interfaces, but do not > want to use a computing engine such as Flink or Spark. > We can expose the programming api to read and write the tablestore, and also > need to expose the corresponding dependency jar. Note that this dependency > may not conflict with multiple versions of Flink, which is conducive to the > integration of third-party systems. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30276) [umbrella] Flink free for table store core
[ https://issues.apache.org/jira/browse/FLINK-30276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30276. Resolution: Fixed > [umbrella] Flink free for table store core > -- > > Key: FLINK-30276 > URL: https://issues.apache.org/jira/browse/FLINK-30276 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > In FLINK-30080, We need a core that does not rely on specific Flink versions > to support flexible deployment and ecology. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31002) Provide data sampling query
[ https://issues.apache.org/jira/browse/FLINK-31002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31002. Resolution: Fixed this should be implemented by flink sql > Provide data sampling query > --- > > Key: FLINK-31002 > URL: https://issues.apache.org/jira/browse/FLINK-31002 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > > Want to take several randomly from each partition, but the limit is always > fixed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30628) Kerberos in HiveCatalog is not work
[ https://issues.apache.org/jira/browse/FLINK-30628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30628. Resolution: Fixed > Kerberos in HiveCatalog is not work > --- > > Key: FLINK-30628 > URL: https://issues.apache.org/jira/browse/FLINK-30628 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0, table-store-0.3.1 > > > We should read kerberos keytab from catalog options and doAs for hive > metastore client. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31179) Make data structures serializable
[ https://issues.apache.org/jira/browse/FLINK-31179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31179. Resolution: Fixed > Make data structures serializable > - > > Key: FLINK-31179 > URL: https://issues.apache.org/jira/browse/FLINK-31179 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31322) Improve merge-into action
[ https://issues.apache.org/jira/browse/FLINK-31322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31322. Resolution: Fixed > Improve merge-into action > - > > Key: FLINK-31322 > URL: https://issues.apache.org/jira/browse/FLINK-31322 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: yuzelin >Priority: Major > > Umbrella issue for improving flink merge-into action. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31315) FlinkActionsE2eTest.testMergeInto is unstable
[
https://issues.apache.org/jira/browse/FLINK-31315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31315.
Resolution: Fixed
> FlinkActionsE2eTest.testMergeInto is unstable
> -
>
> Key: FLINK-31315
> URL: https://issues.apache.org/jira/browse/FLINK-31315
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: table-store-0.4.0
>
>
> {code:java}
> Error: Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 320.272 s <<< FAILURE! - in
> org.apache.flink.table.store.tests.FlinkActionsE2eTest
> 82Error: testMergeInto Time elapsed: 111.826 s <<< FAILURE!
> 83org.opentest4j.AssertionFailedError:
> 84Result is still unexpected after 60 retries.
> 85Expected: {3, v_3, creation, 02-27=1, 2, v_2, creation, 02-27=1, 6, v_6,
> creation, 02-28=1, 1, v_1, creation, 02-27=1, 8, v_8, insert, 02-29=1, 11,
> v_11, insert, 02-29=1, 7, Seven, matched_upsert, 02-28=1, 5, v_5, creation,
> 02-28=1, 10, v_10, creation, 02-28=1, 9, v_9, creation, 02-28=1}
> 86Actual: {4, v_4, creation, 02-27=1, 8, v_8, creation, 02-28=1, 3, v_3,
> creation, 02-27=1, 7, v_7, creation, 02-28=1, 2, v_2, creation, 02-27=1, 6,
> v_6, creation, 02-28=1, 1, v_1, creation, 02-27=1, 5, v_5, creation, 02-28=1,
> 10, v_10, creation, 02-28=1, 9, v_9, creation, 02-28=1}
> 87at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39)
> 88at org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
> 89at
> org.apache.flink.table.store.tests.E2eTestBase.checkResult(E2eTestBase.java:261)
> 90at
> org.apache.flink.table.store.tests.FlinkActionsE2eTest.testMergeInto(FlinkActionsE2eTest.java:355)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31338) support infer parallelism for flink table store
[ https://issues.apache.org/jira/browse/FLINK-31338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31338. Resolution: Fixed Just use github pr/issues. > support infer parallelism for flink table store > > > Key: FLINK-31338 > URL: https://issues.apache.org/jira/browse/FLINK-31338 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jun Zhang >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > When using flink to query the fts table, we can config the scan parallelism > by set the scan.parallelism, but the user may do not know how much > parallelism should be used, setting a too large parallelism will cause > resource waste, setting the parallelism too small will cause the query to be > slow, so we can add parallelism infer. > The function is enabled by default. the parallelism is equal to the number of > read splits. Of course, the user can manually turn off the infer function. In > order to prevent too many datafiles from causing excessive parallelism, we > also set a max infer parallelism. When the infer parallelism exceeds the > setting, use the max parallelism. > In addition, we also need to compare with the limit in the select query > statement to get a more appropriate parallelism in the case of limit > pushdown, for example we have a sql select * from table limit 1, and finally > we infer the parallelism is 10, but we only one parallel is needed , besause > we only need one data . -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31417) Hadoop version unknown when TrinoPageSourceBase.getNextPage
[
https://issues.apache.org/jira/browse/FLINK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31417.
Resolution: Fixed
> Hadoop version unknown when TrinoPageSourceBase.getNextPage
> ---
>
> Key: FLINK-31417
> URL: https://issues.apache.org/jira/browse/FLINK-31417
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.4.0
>Reporter: nonggia.liang
>Priority: Major
>
> Exception thrown when quering flink-table-store by trino
> {code:java}
> 2023-03-13T11:46:36.694+0800ERROR SplitRunner-11-113
> io.trino.execution.executor.TaskExecutor Error processing Split
> 20230313_034504_0_jdcet.1.0.0-11 {} (start = 3.599627617710298E10, wall =
> 89264 ms, cpu = 0 ms, wait = 1 ms, calls = 1)java.lang.NoClassDefFoundError:
> Could not initialize class
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderUtils
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:257)
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:649)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createRecordReader(OrcReaderFactory.java:284)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:98)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:56)
> at
> org.apache.flink.table.store.file.utils.FileUtils.createFormatReader(FileUtils.java:108)
> at
> org.apache.flink.table.store.file.io.KeyValueDataFileRecordReader.(KeyValueDataFileRecordReader.java:55)
> at
> org.apache.flink.table.store.file.io.KeyValueFileReaderFactory.createRecordReader(KeyValueFileReaderFactory.java:95)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.lambda$readerForRun$1(MergeTreeReaders.java:89)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForRun(MergeTreeReaders.java:92)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForSection(MergeTreeReaders.java:74)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.lambda$createReader$2(KeyValueFileStoreRead.java:195)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.createReader(KeyValueFileStoreRead.java:204)
> at
> org.apache.flink.table.store.table.source.KeyValueTableRead.createReader(KeyValueTableRead.java:44)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:76)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.lambda$createPageSource$0(TrinoPageSourceProvider.java:52)
> at
> org.apache.flink.table.store.trino.ClassLoaderUtils.runWithContextClassLoader(ClassLoaderUtils.java:30)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:51)
> at
> io.trino.split.PageSourceManager.createPageSource(PageSourceManager.java:68)
> at
> io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:308)
> at io.trino.operator.Driver.processInternal(Driver.java:388)at
> io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)at
> io.trino.operator.Driver.tryWithLock(Driver.java:685)at
> io.trino.operator.Driver.processFor(Driver.java:285)at
> io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
> at
> io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
> at
> io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
> at io.trino.$gen.Trino_366_020230313_034413_2.run(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-03-13T11:46:36.775+0800ERROR remote-task-callback-2
> io.trino.execution.scheduler.PipelinedStageExecution Pipelined stage
> execution for stage 20230313_034504_0_jdcet.1
> failedjava.lang.ExceptionInInitializerErrorat
>
[jira] [Closed] (FLINK-31391) Introduce lookup changelog producer
[ https://issues.apache.org/jira/browse/FLINK-31391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31391. Resolution: Fixed > Introduce lookup changelog producer > --- > > Key: FLINK-31391 > URL: https://issues.apache.org/jira/browse/FLINK-31391 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, only full-compaction can produce changelog, some merge-engine must > have changelog producing, for example, partial-update and aggregation. But > full-compaction is very heavy, write amplification is big huge... > We should introduce a new changelog producer, supports lower latency to > produce changelog. > > POC: https://github.com/apache/flink-table-store/pull/590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31417) Hadoop version unknown when TrinoPageSourceBase.getNextPage
[
https://issues.apache.org/jira/browse/FLINK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702207#comment-17702207
]
Jingsong Lee commented on FLINK-31417:
--
Thanks [~nonggia] , this has been fixed in 0.3, will cp to 0.4.
> Hadoop version unknown when TrinoPageSourceBase.getNextPage
> ---
>
> Key: FLINK-31417
> URL: https://issues.apache.org/jira/browse/FLINK-31417
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.4.0
>Reporter: nonggia.liang
>Priority: Major
>
> Exception thrown when quering flink-table-store by trino
> {code:java}
> 2023-03-13T11:46:36.694+0800ERROR SplitRunner-11-113
> io.trino.execution.executor.TaskExecutor Error processing Split
> 20230313_034504_0_jdcet.1.0.0-11 {} (start = 3.599627617710298E10, wall =
> 89264 ms, cpu = 0 ms, wait = 1 ms, calls = 1)java.lang.NoClassDefFoundError:
> Could not initialize class
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderUtils
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:257)
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:649)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createRecordReader(OrcReaderFactory.java:284)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:98)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:56)
> at
> org.apache.flink.table.store.file.utils.FileUtils.createFormatReader(FileUtils.java:108)
> at
> org.apache.flink.table.store.file.io.KeyValueDataFileRecordReader.(KeyValueDataFileRecordReader.java:55)
> at
> org.apache.flink.table.store.file.io.KeyValueFileReaderFactory.createRecordReader(KeyValueFileReaderFactory.java:95)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.lambda$readerForRun$1(MergeTreeReaders.java:89)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForRun(MergeTreeReaders.java:92)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForSection(MergeTreeReaders.java:74)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.lambda$createReader$2(KeyValueFileStoreRead.java:195)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.createReader(KeyValueFileStoreRead.java:204)
> at
> org.apache.flink.table.store.table.source.KeyValueTableRead.createReader(KeyValueTableRead.java:44)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:76)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.lambda$createPageSource$0(TrinoPageSourceProvider.java:52)
> at
> org.apache.flink.table.store.trino.ClassLoaderUtils.runWithContextClassLoader(ClassLoaderUtils.java:30)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:51)
> at
> io.trino.split.PageSourceManager.createPageSource(PageSourceManager.java:68)
> at
> io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:308)
> at io.trino.operator.Driver.processInternal(Driver.java:388)at
> io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)at
> io.trino.operator.Driver.tryWithLock(Driver.java:685)at
> io.trino.operator.Driver.processFor(Driver.java:285)at
> io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
> at
> io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
> at
> io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
> at io.trino.$gen.Trino_366_020230313_034413_2.run(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-03-13T11:46:36.775+0800ERROR remote-task-callback-2
> io.trino.execution.scheduler.PipelinedStageExecution Pipelined stage
> execution for stage 20230313_034504_0_jdcet.1
>
[jira] [Closed] (FLINK-31461) Supports schema historical version expiring
[ https://issues.apache.org/jira/browse/FLINK-31461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31461. Resolution: Fixed Use github pr/issues. > Supports schema historical version expiring > --- > > Key: FLINK-31461 > URL: https://issues.apache.org/jira/browse/FLINK-31461 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > > Schema evolution will generate multiple versions of schema. When the > specified version of the schema is no longer referenced by snapshot, it > should be deleted -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31462) [Flink] Supports full calculation from specified snapshots in streaming mode
[ https://issues.apache.org/jira/browse/FLINK-31462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31462. Resolution: Fixed Use github pr/issues. > [Flink] Supports full calculation from specified snapshots in streaming mode > > > Key: FLINK-31462 > URL: https://issues.apache.org/jira/browse/FLINK-31462 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Ming Li >Priority: Major > Labels: pull-request-available > > Currently, the table store provides a startup mode for incremental > consumption from a specified snapshot in streaming mode. We can provide a > startup mode for incremental consumption after full calculation from a > specified snapshot. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31429) Support CTAS(create table as) streaming job with schema changes in table store
[ https://issues.apache.org/jira/browse/FLINK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702206#comment-17702206 ] Jingsong Lee commented on FLINK-31429: -- Please use github pr/issues. > Support CTAS(create table as) streaming job with schema changes in table store > -- > > Key: FLINK-31429 > URL: https://issues.apache.org/jira/browse/FLINK-31429 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Fix For: table-store-0.4.0 > > > Currently CDC connectors for Flink has the ability to stream out records > changes and schema changes of a database table. Flink Table Store should have > the ability to directly consume these changes, including schema changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31451) Flink Table Store Ecosystem: Introduce Presto Reader for table store
[ https://issues.apache.org/jira/browse/FLINK-31451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31451. Resolution: Fixed Use github pr/issues. > Flink Table Store Ecosystem: Introduce Presto Reader for table store > > > Key: FLINK-31451 > URL: https://issues.apache.org/jira/browse/FLINK-31451 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Zaihu Shi >Assignee: Zaihu Shi >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > Introduce Presto Reader for table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31479) Close blocking iterators in tests
[ https://issues.apache.org/jira/browse/FLINK-31479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31479. Resolution: Fixed Use github issues. > Close blocking iterators in tests > - > > Key: FLINK-31479 > URL: https://issues.apache.org/jira/browse/FLINK-31479 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Guojun Li >Priority: Major > > Several blocking iterators are not closed in `ContinuousFileStoreITCase` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31467) Support time travel for Spark 3.3
[ https://issues.apache.org/jira/browse/FLINK-31467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31467. Resolution: Fixed Use github issues. > Support time travel for Spark 3.3 > - > > Key: FLINK-31467 > URL: https://issues.apache.org/jira/browse/FLINK-31467 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: yuzelin >Priority: Major > Labels: pull-request-available > > Support Spark 3.3 `VERSION AS OF` and `TIMESTAMP AS OF` Syntax -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31462) [Flink] Supports full calculation from specified snapshots in streaming mode
[ https://issues.apache.org/jira/browse/FLINK-31462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17701571#comment-17701571 ] Jingsong Lee commented on FLINK-31462: -- Watermark will always move forward. What I don't understand is whether these changes are necessary. > [Flink] Supports full calculation from specified snapshots in streaming mode > > > Key: FLINK-31462 > URL: https://issues.apache.org/jira/browse/FLINK-31462 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Ming Li >Priority: Major > > Currently, the table store provides a startup mode for incremental > consumption from a specified snapshot in streaming mode. We can provide a > startup mode for incremental consumption after full calculation from a > specified snapshot. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26465) Optimize SortMergeReader: use loser tree to reduce comparisons
[ https://issues.apache.org/jira/browse/FLINK-26465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17701565#comment-17701565 ] Jingsong Lee commented on FLINK-26465: -- [~Ming Li] Yes, you can, but I think this is a very complicated one, you can have some try. > Optimize SortMergeReader: use loser tree to reduce comparisons > -- > > Key: FLINK-26465 > URL: https://issues.apache.org/jira/browse/FLINK-26465 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Aiden Gong >Priority: Minor > Labels: pull-request-available, stale-assigned > Fix For: table-store-0.4.0 > > > See https://en.wikipedia.org/wiki/K-way_merge_algorithm -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31430) Support migrating states between different instances of TableWriteImpl and AbstractFileStoreWrite
[
https://issues.apache.org/jira/browse/FLINK-31430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31430.
Resolution: Fixed
master: 1af16a5a1cb27b6b72bbcef9e5862fda84d8c996
> Support migrating states between different instances of TableWriteImpl and
> AbstractFileStoreWrite
> -
>
> Key: FLINK-31430
> URL: https://issues.apache.org/jira/browse/FLINK-31430
> Project: Flink
> Issue Type: Sub-task
> Components: Table Store
>Affects Versions: table-store-0.4.0
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.4.0
>
>
> Currently {{Table}} and {{TableWrite}} in Flink Table Store have a fixed
> schema. However to consume schema changes, Flink Table Store CDC sinks should
> have the ability to change its schema during a streaming job.
> This require us to pause and store the states of a {{TableWrite}}, then
> create a {{TableWrite}} with newer schema and recover the states in the new
> {{TableWrite}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31459) add UPDATE COLUMN POSITION for flink table store
[ https://issues.apache.org/jira/browse/FLINK-31459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31459. Assignee: Jun Zhang Resolution: Fixed master: 14d87e4a85e716771c59a5e2fbb840fd1852a0f6 > add UPDATE COLUMN POSITION for flink table store > > > Key: FLINK-31459 > URL: https://issues.apache.org/jira/browse/FLINK-31459 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.1 >Reporter: Jun Zhang >Assignee: Jun Zhang >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31465) [Flink] Fix shortcode errors in docs
[
https://issues.apache.org/jira/browse/FLINK-31465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31465.
Fix Version/s: table-store-0.4.0
Assignee: Ming Li
Resolution: Fixed
master: dd05a70d0b66fd3bbf1afe0cd1e8362405f024c7
> [Flink] Fix shortcode errors in docs
>
>
> Key: FLINK-31465
> URL: https://issues.apache.org/jira/browse/FLINK-31465
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
>Reporter: Ming Li
>Assignee: Ming Li
>Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.4.0
>
>
> When running docs with hugo, I get the following exception:
> {code:java}
> hugo v0.111.3+extended darwin/amd64 BuildDate=unknown
> Error: Error building site:
> "/xxx/flink-table-store/docs/content/docs/how-to/writing-tables.md:303:1":
> failed to extract shortcode: shortcode "tabs" must be closed or
> self-closed{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31253) Port itcases to Flink 1.15 and 1.14
[ https://issues.apache.org/jira/browse/FLINK-31253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31253. Resolution: Fixed master: 7663615366a16d3481c67aa093a5f95973ae552e > Port itcases to Flink 1.15 and 1.14 > --- > > Key: FLINK-31253 > URL: https://issues.apache.org/jira/browse/FLINK-31253 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Chao Tian >Priority: Major > Labels: pull-request-available, starter > Fix For: table-store-0.4.0 > > > At present, only common has tests. We need to copy a part of itcase to 1.14 > and 1.15 to ensure normal work. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31460) Fix hive catalog and connector jar name in the create release script for table store
[ https://issues.apache.org/jira/browse/FLINK-31460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31460. Fix Version/s: table-store-0.4.0 Assignee: zhuangchong Resolution: Fixed master: d51ca84c8d711f74bbdd38e4acab627de78c2681 > Fix hive catalog and connector jar name in the create release script for > table store > > > Key: FLINK-31460 > URL: https://issues.apache.org/jira/browse/FLINK-31460 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: zhuangchong >Assignee: zhuangchong >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31462) [Flink] Supports full calculation from specified snapshots in streaming mode
[ https://issues.apache.org/jira/browse/FLINK-31462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700620#comment-17700620 ] Jingsong Lee commented on FLINK-31462: -- Thanks [~Ming Li] for your reporting, can you explain what case for? > [Flink] Supports full calculation from specified snapshots in streaming mode > > > Key: FLINK-31462 > URL: https://issues.apache.org/jira/browse/FLINK-31462 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Ming Li >Priority: Major > > Currently, the table store provides a startup mode for incremental > consumption from a specified snapshot in streaming mode. We can provide a > startup mode for incremental consumption after full calculation from a > specified snapshot. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31461) Supports schema historical version expiring
[ https://issues.apache.org/jira/browse/FLINK-31461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700494#comment-17700494 ] Jingsong Lee edited comment on FLINK-31461 at 3/15/23 3:48 AM: --- I think this is good! was (Author: lzljs3620320): I think this is good > Supports schema historical version expiring > --- > > Key: FLINK-31461 > URL: https://issues.apache.org/jira/browse/FLINK-31461 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Shammon FY >Priority: Major > > Schema evolution will generate multiple versions of schema. When the > specified version of the schema is no longer referenced by snapshot, it > should be deleted -- This message was sent by Atlassian Jira (v8.20.10#820010)
