[jira] [Updated] (FLINK-33206) Verify the existence of hbase table before read/write
[ https://issues.apache.org/jira/browse/FLINK-33206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-33206: --- Labels: pull-request-available stale-blocker (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as a Blocker but is unassigned and neither itself nor its Sub-Tasks have been updated for 1 days. I have gone ahead and marked it "stale-blocker". If this ticket is a Blocker, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Verify the existence of hbase table before read/write > - > > Key: FLINK-33206 > URL: https://issues.apache.org/jira/browse/FLINK-33206 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase >Affects Versions: hbase-3.0.1 >Reporter: tanjialiang >Priority: Blocker > Labels: pull-request-available, stale-blocker > Attachments: image-2023-10-08-16-54-05-917.png > > > Currently, we do not verify the existence of hbase table before read/write, > and the error would make the user confused. > The `HBaseSinkFunction` throws `TableNotFoundException` when do flush. > The `inputFormat` throws not obvious enough. > !image-2023-10-08-16-54-05-917.png! > So i think we should verify the existence of hbase table when call `open` > function. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32714) JDBC: Add dialect for OceanBase database
[
https://issues.apache.org/jira/browse/FLINK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32714:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> JDBC: Add dialect for OceanBase database
>
>
> Key: FLINK-32714
> URL: https://issues.apache.org/jira/browse/FLINK-32714
> Project: Flink
> Issue Type: New Feature
>Reporter: He Wang
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> OceanBase is a distributed relational database, the community edition of
> OceanBase is open sourced at [https://github.com/oceanbase/oceanbase.]
> The enterprise edition of OceanBase is compatible with MySql and Oracle,
> which means we can reuse almost all the dialect rules.
> The difference from other databases is that we must provide the compatibility
> mode firstly, then the connector can determine which dialect to use, so a
> startup option like 'compatible-mode' is needed.
> A dialect implementation for OceanBase is like below:
> {code:java}
> package org.apache.flink.connector.jdbc.databases.oceanbase;
> import org.apache.flink.connector.jdbc.converter.JdbcRowConverter;
> import org.apache.flink.connector.jdbc.databases.mysql.dialect.MySqlDialect;
> import org.apache.flink.connector.jdbc.databases.oracle.dialect.OracleDialect;
> import org.apache.flink.connector.jdbc.dialect.AbstractDialect;
> import org.apache.flink.table.types.logical.LogicalTypeRoot;
> import org.apache.flink.table.types.logical.RowType;
> import javax.annotation.Nonnull;
> import java.util.Optional;
> import java.util.Set;
> /** JDBC dialect for OceanBase. */
> public class OceanBaseDialect extends AbstractDialect {
> private static final long serialVersionUID = 1L;
> private final AbstractDialect dialect;
> public OceanBaseDialect(@Nonnull String compatibleMode) {
> switch (compatibleMode.toLowerCase()) {
> case "mysql":
> this.dialect = new MySqlDialect();
> break;
> case "oracle":
> this.dialect = new OracleDialect();
> break;
> default:
> throw new IllegalArgumentException(
> "Unsupported compatible mode: " + compatibleMode);
> }
> }
> @Override
> public String dialectName() {
> return "OceanBase";
> }
> @Override
> public Optional defaultDriverName() {
> return Optional.of("com.oceanbase.jdbc.Driver");
> }
> @Override
> public Set supportedTypes() {
> return dialect.supportedTypes();
> }
> @Override
> public JdbcRowConverter getRowConverter(RowType rowType) {
> return dialect.getRowConverter(rowType);
> }
> @Override
> public String getLimitClause(long limit) {
> return dialect.getLimitClause(limit);
> }
> @Override
> public String quoteIdentifier(String identifier) {
> return dialect.quoteIdentifier(identifier);
> }
> @Override
> public Optional getUpsertStatement(
> String tableName, String[] fieldNames, String[] conditionFields) {
> return dialect.getUpsertStatement(tableName, fieldNames,
> conditionFields);
> }
> @Override
> public Optional timestampPrecisionRange() {
> return dialect.timestampPrecisionRange();
> }
> @Override
> public Optional decimalPrecisionRange() {
> return dialect.decimalPrecisionRange();
> }
> @Override
> public String appendDefaultUrlProperties(String url) {
> return dialect.appendDefaultUrlProperties(url);
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32720) Add GENERATE_SERIES support in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32720:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Add GENERATE_SERIES support in SQL & Table API
> --
>
> Key: FLINK-32720
> URL: https://issues.apache.org/jira/browse/FLINK-32720
> Project: Flink
> Issue Type: Improvement
>Reporter: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available, stale-major
>
> GENERATE_SERIES Function
> Description
> Constructs an array of values between {{start}} and {{{}end{}}}, inclusive.
> Parameters {{start}} and {{end}} can be an {{INT}} or {{{}BIGINT{}}}.
> {{{}step{}}}, if supplied, specifies the step size. The step can be positive
> or negative. If not supplied, {{step}} defaults to {{{}1{}}}. Parameter
> {{step}} must be an {{{}INT{}}}.
> Syntax
> The syntax for the GENERATE_SERIES function is:
> {code:java}
> GENERATE_SERIES(start, end)
> GENERATE_SERIES(start, end, step){code}
>
> Example
> Let's look at some function examples and explore how to use the SPLIT
> function.
> For example:
>
> {code:java}
> SELECT GENERATE_SERISE(1, 5);
> Result: [1,2,3,4,5]
> SELECT GENERATE_SERISE(0, 10, 2);
> Result: [0, 2, 4, 6, 8, 10] {code}
> see also:
> 1.PostgreSQL: PostgreSQL offers a function called {{generate_series}} which
> generates a set of contiguous integers from a start to an end value. An
> optional 'step' parameter is available to specify the increment between each
> integer.
> https://www.postgresql.org/docs/current/functions-srf.html
> 2.ksqlDB: As you mentioned, ksqlDB provides a function called
> {{GENERATE_SERIES}} that generates a series of numbers, starting from a given
> start value, incrementing each time by a step value, until it reaches or
> exceeds a given end value.
> https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/scalar-functions/#generate_series
> 3.BigQuery: BigQuery has a function called {{GENERATE_ARRAY}} that generates
> an array consisting of integers from the start value to the end value, with
> each integer incremented by the step value. You can find more details in the
> https://cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#generate_array
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33207) Scan hbase table will throw error when table is empty
[ https://issues.apache.org/jira/browse/FLINK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-33207: --- Labels: pull-request-available stale-blocker (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as a Blocker but is unassigned and neither itself nor its Sub-Tasks have been updated for 1 days. I have gone ahead and marked it "stale-blocker". If this ticket is a Blocker, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Scan hbase table will throw error when table is empty > - > > Key: FLINK-33207 > URL: https://issues.apache.org/jira/browse/FLINK-33207 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Affects Versions: hbase-3.0.1 >Reporter: tanjialiang >Priority: Blocker > Labels: pull-request-available, stale-blocker > > When i scan the empty hbase table, it will throw an error when > createInputSplits, we should return empty split instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32815) Add HASHCODE support in Table API
[
https://issues.apache.org/jira/browse/FLINK-32815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32815:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Add HASHCODE support in Table API
> -
>
> Key: FLINK-32815
> URL: https://issues.apache.org/jira/browse/FLINK-32815
> Project: Flink
> Issue Type: Improvement
>Reporter: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available, stale-major
>
> *This is an implementation of HASHCODE internal function*
> The {{hashcode}} function generates a hash code for a given input value,
> including support for computing hash values of binary data types. It creates
> a unique integer that represents the value passed to the function.
> *Brief change log*
> * {{HASHCODE}} for Table API
> *Syntax:*
> {code:java}
> HASHCODE(value){code}
> *Arguments:*
> * value: the value to be hashed.
> *Returns:* The function returns a unique integer representing the hash code
> of the value. If the input argument is NULL, the function returns NULL.
> Because it is an internal function, so it will not support sql anymore.
> *see also:*
> Java:
> [https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html#hashCode--]
> Python: [https://docs.python.org/3/library/functions.html#hash]
> C#: [https://docs.microsoft.com/en-us/dotnet/api/system.object.gethashcode]
> SQL Server:
> [https://docs.microsoft.com/en-us/sql/t-sql/functions/checksum-transact-sql]
> MySQL:
> [https://dev.mysql.com/doc/refman/8.0/en/encryption-functions.html#function_md5]
> PostgreSQL: [https://www.postgresql.org/docs/current/pgcrypto-hash.html]
> Oracle:
> [https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/ORA-HASH.html]
> Google Cloud BigQuery:
> [https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#farm_fingerprint]
> AWS Redshift: [https://docs.aws.amazon.com/redshift/latest/dg/MD5.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32722) 239 exit code in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32722:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> 239 exit code in flink-runtime
> --
>
> Key: FLINK-32722
> URL: https://issues.apache.org/jira/browse/FLINK-32722
> Project: Flink
> Issue Type: Bug
> Components: Test Infrastructure
>Affects Versions: 1.16.2
>Reporter: Matthias Pohl
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51852=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=be5a4b15-4b23-56b1-7582-795f58a645a2=8418
> {code:java}
> Aug 01 01:03:49 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test)
> on project flink-runtime: There are test failures.
> Aug 01 01:03:49 [ERROR]
> Aug 01 01:03:49 [ERROR] Please refer to
> /__w/2/s/flink-runtime/target/surefire-reports for the individual test
> results.
> Aug 01 01:03:49 [ERROR] Please refer to dump files (if any exist)
> [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Aug 01 01:03:49 [ERROR] ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> Aug 01 01:03:49 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseG1GC -Xms256m -Xmx768m
> -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter1803120121827605294.jar
> /__w/2/s/flink-runtime/target/surefire 2023-08-01T00-58-17_520-jvmRun1
> surefire9107652818401825168tmp surefire_261701267003130520249tmp
> Aug 01 01:03:49 [ERROR] Error occurred in starting fork, check output in log
> Aug 01 01:03:49 [ERROR] Process Exit Code: 239
> Aug 01 01:03:49 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Aug 01 01:03:49 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseG1GC -Xms256m -Xmx768m
> -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter1803120121827605294.jar
> /__w/2/s/flink-runtime/target/surefire 2023-08-01T00-58-17_520-jvmRun1
> surefire9107652818401825168tmp surefire_261701267003130520249tmp
> Aug 01 01:03:49 [ERROR] Error occurred in starting fork, check output in log
> Aug 01 01:03:49 [ERROR] Process Exit Code: 239
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:405)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:321)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
> Aug 01 01:03:49 [ERROR] at
>
[jira] [Updated] (FLINK-32711) Type mismatch when proctime function used as parameter
[
https://issues.apache.org/jira/browse/FLINK-32711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32711:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Type mismatch when proctime function used as parameter
> --
>
> Key: FLINK-32711
> URL: https://issues.apache.org/jira/browse/FLINK-32711
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: Aitozi
>Priority: Major
> Labels: pull-request-available, stale-major
>
> reproduce case:
> {code:sql}
> SELECT TYPEOF(PROCTIME())
> {code}
> this query will fail with
> org.apache.flink.table.planner.codegen.CodeGenException: Mismatch of
> function's argument data type 'TIMESTAMP_LTZ(3) NOT NULL' and actual argument
> type 'TIMESTAMP_LTZ(3)'.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32665) Support read null value for csv format
[ https://issues.apache.org/jira/browse/FLINK-32665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32665: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support read null value for csv format > -- > > Key: FLINK-32665 > URL: https://issues.apache.org/jira/browse/FLINK-32665 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Fang Yong >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > when there is null column in a file with csv format, it will throw exception > when flink job try to parse these data -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-9631) use Files.createDirectories instead of directory.mkdirs
[
https://issues.apache.org/jira/browse/FLINK-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9631:
--
Labels: auto-deprioritized-minor auto-unassigned stale-assigned (was:
auto-deprioritized-minor auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> use Files.createDirectories instead of directory.mkdirs
> ---
>
> Key: FLINK-9631
> URL: https://issues.apache.org/jira/browse/FLINK-9631
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.4.2, 1.5.0
> Environment: flink1.4
> jdk1.8 latest
> linux 2.6
>Reporter: makeyang
>Assignee: Yangyang ZHANG
>Priority: Not a Priority
> Labels: auto-deprioritized-minor, auto-unassigned, stale-assigned
>
> job can't be run due to below exception:
> {color:#6a8759}Could not create RocksDB data directory{color}
> but with this exception, I can't tell exactly why.
> so I suggest Files.createDirectories which throw exception be used rather
> than File.mkdirs
>
> I have some more suggestions:
> # should we use Files.createDirectories to relpace File.mkdirs?
> # each time task manager throw exception to jobmanager, should IP+nodeId be
> contained in exception, which means we should define more flink exception
> which is used to wrap other exceptions such as jdk exceptions?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32293) Support vector with long index
[ https://issues.apache.org/jira/browse/FLINK-32293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32293: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support vector with long index > -- > > Key: FLINK-32293 > URL: https://issues.apache.org/jira/browse/FLINK-32293 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Zhipeng Zhang >Priority: Major > Labels: pull-request-available, stale-major > > Currently in Flink ML, we only support sparse and dense vector with `int` as > index and `double` as value. > > However, there are real-world cases that the index of a vector could exceed > the range of `INT.MAX`. Thus we need to support vector with `long` index. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32732) auto offset reset should be exposed to user
[
https://issues.apache.org/jira/browse/FLINK-32732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32732:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> auto offset reset should be exposed to user
> ---
>
> Key: FLINK-32732
> URL: https://issues.apache.org/jira/browse/FLINK-32732
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: 1.16.1
>Reporter: xiaogang zhou
>Priority: Major
> Labels: pull-request-available, stale-major
>
> {code:java}
> // code placeholder
> maybeOverride(
> ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
>
> startingOffsetsInitializer.getAutoOffsetResetStrategy().name().toLowerCase(),
> true); {code}
> now flink override the auto.offset.reset with the scan.startup.mode config,
> and user's explicit config does not take effect. I think maybe we should
> expose this to customer?
>
> I think after consuming kafka records from earliest to latest, the
> scan.startup.mode should no longer influence the kafka scan behave. So I
> suggest change the override to false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31615) Fix some parts forgot to translate in "Table API" page of "Table API & SQL"
[ https://issues.apache.org/jira/browse/FLINK-31615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31615: --- Labels: auto-deprioritized-minor chinese-translation (was: chinese-translation stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Fix some parts forgot to translate in "Table API" page of "Table API & SQL" > > > Key: FLINK-31615 > URL: https://issues.apache.org/jira/browse/FLINK-31615 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Hang Ruan >Priority: Not a Priority > Labels: auto-deprioritized-minor, chinese-translation > > The query_state_warning in "Table API" page of "Table API & SQL" is still in > English. And some comments in codes are in English. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33020) OpensearchSinkTest.testAtLeastOnceSink timed out
[
https://issues.apache.org/jira/browse/FLINK-33020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33020:
---
Labels: pull-request-available stale-blocker (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as a
Blocker but is unassigned and neither itself nor its Sub-Tasks have been
updated for 1 days. I have gone ahead and marked it "stale-blocker". If this
ticket is a Blocker, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> OpensearchSinkTest.testAtLeastOnceSink timed out
>
>
> Key: FLINK-33020
> URL: https://issues.apache.org/jira/browse/FLINK-33020
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Opensearch
>Affects Versions: opensearch-1.0.2
>Reporter: Martijn Visser
>Priority: Blocker
> Labels: pull-request-available, stale-blocker
>
> https://github.com/apache/flink-connector-opensearch/actions/runs/6061205003/job/16446139552#step:13:1029
> {code:java}
> Error: Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.837
> s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.opensearch.OpensearchSinkTest
> Error:
> org.apache.flink.streaming.connectors.opensearch.OpensearchSinkTest.testAtLeastOnceSink
> Time elapsed: 5.022 s <<< ERROR!
> java.util.concurrent.TimeoutException: testAtLeastOnceSink() timed out after
> 5 seconds
> at
> org.junit.jupiter.engine.extension.TimeoutInvocation.createTimeoutException(TimeoutInvocation.java:70)
> at
> org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:59)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> at
>
[jira] [Updated] (FLINK-32667) Use standalone store and embedding writer for jobs with no-restart-strategy in session cluster
[ https://issues.apache.org/jira/browse/FLINK-32667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32667: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Use standalone store and embedding writer for jobs with no-restart-strategy > in session cluster > -- > > Key: FLINK-32667 > URL: https://issues.apache.org/jira/browse/FLINK-32667 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: Fang Yong >Priority: Major > Labels: pull-request-available, stale-assigned > > When a flink session cluster use zk or k8s high availability service, it will > store jobs in zk or ConfigMap. When we submit flink olap jobs to the session > cluster, they always turn off restart strategy. These jobs with > no-restart-strategy should not be stored in zk or ConfigMap in k8s -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33116) CliClientTest.testCancelExecutionInteractiveMode fails with NPE on AZP
[
https://issues.apache.org/jira/browse/FLINK-33116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33116:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> CliClientTest.testCancelExecutionInteractiveMode fails with NPE on AZP
> --
>
> Key: FLINK-33116
> URL: https://issues.apache.org/jira/browse/FLINK-33116
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53309=logs=ce3801ad-3bd5-5f06-d165-34d37e757d90=5e4d9387-1dcc-5885-a901-90469b7e6d2f=12264
> fails as
> {noformat}
> Sep 18 02:26:15 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> Sep 18 02:26:15 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> Sep 18 02:26:15 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> Sep 18 02:26:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> Sep 18 02:26:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> Sep 18 02:26:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32471) IS_NOT_NULL can add to SUITABLE_FILTER_TO_PUSH
[ https://issues.apache.org/jira/browse/FLINK-32471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32471: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > IS_NOT_NULL can add to SUITABLE_FILTER_TO_PUSH > -- > > Key: FLINK-32471 > URL: https://issues.apache.org/jira/browse/FLINK-32471 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Reporter: grandfisher >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > According to FLINK-31273: > The reason for the error is that other filters conflict with IS_NULL, but in > fact it won't conflict with IS_NOT_NULL, because operators in > SUITABLE_FILTER_TO_PUSH such as 'SqlKind.GREATER_THAN' has an implicit > filter 'IS_NOT_NULL' according to SQL Semantics. > > So we think it is feasible to add IS_NOT_NULL to the SUITABLE_FILTER_TO_PUSH > list. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33115) AbstractHadoopRecoverableWriterITCase is hanging with timeout on AZP
[
https://issues.apache.org/jira/browse/FLINK-33115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33115:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> AbstractHadoopRecoverableWriterITCase is hanging with timeout on AZP
>
>
> Key: FLINK-33115
> URL: https://issues.apache.org/jira/browse/FLINK-33115
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53281=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9=14239
> is failing as
> {noformat}
> Sep 15 11:33:02
> ==
> Sep 15 11:33:02 Process produced no output for 900 seconds.
> Sep 15 11:33:02
> ==
> ...
> Sep 15 11:33:03 at
> java.io.DataInputStream.read(DataInputStream.java:149)
> Sep 15 11:33:03 at
> org.apache.flink.runtime.fs.hdfs.HadoopDataInputStream.read(HadoopDataInputStream.java:96)
> Sep 15 11:33:03 at
> sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> Sep 15 11:33:03 at
> sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> Sep 15 11:33:03 at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> Sep 15 11:33:03 - locked <0xbfa7a760> (a
> java.io.InputStreamReader)
> Sep 15 11:33:03 at
> java.io.InputStreamReader.read(InputStreamReader.java:184)
> Sep 15 11:33:03 at java.io.BufferedReader.fill(BufferedReader.java:161)
> Sep 15 11:33:03 at
> java.io.BufferedReader.readLine(BufferedReader.java:324)
> Sep 15 11:33:03 - locked <0xbfa7a760> (a
> java.io.InputStreamReader)
> Sep 15 11:33:03 at
> java.io.BufferedReader.readLine(BufferedReader.java:389)
> Sep 15 11:33:03 at
> org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWriterITCase.getContentsOfFile(AbstractHadoopRecoverableWriterITCase.java:387)
> Sep 15 11:33:03 at
> org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWriterITCase.testResumeAfterMultiplePersist(AbstractHadoopRecoverableWriterITCase.java:377)
> Sep 15 11:33:03 at
> org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWriterITCase.testResumeAfterMultiplePersistWithMultiPartUploads(AbstractHadoopRecoverableWriterITCase.java:330)
> Sep 15 11:33:03 at
> org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWri
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33114) RightOuterJoinTaskTest.testCancelRightOuterJoinTaskWhileBuilding:232 Task thread did not finish within 60 seconds
[
https://issues.apache.org/jira/browse/FLINK-33114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33114:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> RightOuterJoinTaskTest.testCancelRightOuterJoinTaskWhileBuilding:232 Task
> thread did not finish within 60 seconds
> -
>
> Key: FLINK-33114
> URL: https://issues.apache.org/jira/browse/FLINK-33114
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53250=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8
> failed as
> {noformat}
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:520)
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$24(ClassBasedTestDescriptor.java:510)
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$10(TestMethodTestDescriptor.java:243)
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:276)
> Sep 14 22:34:21 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Sep 14 22:34:21 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$14(TestMethodTestDescriptor.java:276)
> Sep 14 22:34:21 at
> java.util.ArrayList.forEach(ArrayList.java:1259)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32743) Flink kafka connector source can directly parse data collected from kafka-connect
[ https://issues.apache.org/jira/browse/FLINK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32743: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Flink kafka connector source can directly parse data collected from > kafka-connect > - > > Key: FLINK-32743 > URL: https://issues.apache.org/jira/browse/FLINK-32743 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Xiaojian Sun >Priority: Major > Labels: pull-request-available, stale-major > > If the data in Kafka is collected through the Kafka connect system, it is > hoped that the data collected through Flink has already excluded the schema -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20886) Add the option to get a threaddump on checkpoint timeouts
[ https://issues.apache.org/jira/browse/FLINK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20886: --- Labels: auto-deprioritized-major auto-deprioritized-minor stale-assigned usability (was: auto-deprioritized-major auto-deprioritized-minor usability) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Add the option to get a threaddump on checkpoint timeouts > - > > Key: FLINK-20886 > URL: https://issues.apache.org/jira/browse/FLINK-20886 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Reporter: Nico Kruber >Assignee: Zakelly Lan >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > stale-assigned, usability > > For debugging checkpoint timeouts, I was thinking about the following > addition to Flink: > When a checkpoint times out and the async thread is still running, create a > thread dump [1] and either add this to the checkpoint stats, log it, or write > it out. > This may help identifying where the checkpoint is stuck (maybe a lock, could > also be in a third party lib like the FS connectors,...). It would give us > some insights into what the thread is currently doing. > Limiting the scope of the threads would be nice but may not be possible in > the general case since additional threads (spawned by the FS connector lib, > or otherwise connected) may interact with the async thread(s) by e.g. going > through the same locks. Maybe we can reduce the thread dumps to all async > threads of the failed checkpoint + all thready that interact with it, e.g. > via locks? > I'm also not sure whether the ability to have thread dumps or not should be > user-configurable (Could it contain sensitive information from other jobs if > you run a session cluster? Is that even relevant since we don't give > isolation guarantees anyway?). If it is configurable, it should be on by > default. > [1] https://crunchify.com/how-to-generate-java-thread-dump-programmatically/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32733) Add AlgoOperator for FpGrowth
[ https://issues.apache.org/jira/browse/FLINK-32733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32733: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add AlgoOperator for FpGrowth > - > > Key: FLINK-32733 > URL: https://issues.apache.org/jira/browse/FLINK-32733 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Affects Versions: ml-2.4.0 >Reporter: Yindi Wang >Priority: Major > Labels: pull-request-available, stale-major > > _Frequent Pattern Growth_ Algorithm is the method of finding frequent > patterns. > The distributed version of FpGrowth algorithm is described in Li et al., PFP: > Parallel FP-Growth for Query Recommendation. This algorithm will be an > addition to the > _org.apache.flink.ml.recommendation_ package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32523) NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout on AZP
[
https://issues.apache.org/jira/browse/FLINK-32523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32523:
---
Labels: pull-request-available stale-assigned test-stability (was:
pull-request-available test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout
> on AZP
> ---
>
> Key: FLINK-32523
> URL: https://issues.apache.org/jira/browse/FLINK-32523
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.2, 1.18.0, 1.17.1
>Reporter: Sergey Nuyanzin
>Assignee: Hangxiang Yu
>Priority: Critical
> Labels: pull-request-available, stale-assigned, test-stability
> Fix For: 1.18.0, 1.17.2
>
> Attachments: failure.log
>
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50795=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=8638
> fails with timeout
> {noformat}
> Jul 03 01:26:35 org.junit.runners.model.TestTimedOutException: test timed out
> after 10 milliseconds
> Jul 03 01:26:35 at java.lang.Object.wait(Native Method)
> Jul 03 01:26:35 at java.lang.Object.wait(Object.java:502)
> Jul 03 01:26:35 at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:61)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.verifyAllOperatorsNotifyAborted(NotifyCheckpointAbortedITCase.java:198)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted(NotifyCheckpointAbortedITCase.java:189)
> Jul 03 01:26:35 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 03 01:26:35 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 03 01:26:35 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 03 01:26:35 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 03 01:26:35 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Jul 03 01:26:35 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Jul 03 01:26:35 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32714) JDBC: Add dialect for OceanBase database
[
https://issues.apache.org/jira/browse/FLINK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32714:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> JDBC: Add dialect for OceanBase database
>
>
> Key: FLINK-32714
> URL: https://issues.apache.org/jira/browse/FLINK-32714
> Project: Flink
> Issue Type: New Feature
>Reporter: He Wang
>Priority: Major
> Labels: pull-request-available, stale-major
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> OceanBase is a distributed relational database, the community edition of
> OceanBase is open sourced at [https://github.com/oceanbase/oceanbase.]
> The enterprise edition of OceanBase is compatible with MySql and Oracle,
> which means we can reuse almost all the dialect rules.
> The difference from other databases is that we must provide the compatibility
> mode firstly, then the connector can determine which dialect to use, so a
> startup option like 'compatible-mode' is needed.
> A dialect implementation for OceanBase is like below:
> {code:java}
> package org.apache.flink.connector.jdbc.databases.oceanbase;
> import org.apache.flink.connector.jdbc.converter.JdbcRowConverter;
> import org.apache.flink.connector.jdbc.databases.mysql.dialect.MySqlDialect;
> import org.apache.flink.connector.jdbc.databases.oracle.dialect.OracleDialect;
> import org.apache.flink.connector.jdbc.dialect.AbstractDialect;
> import org.apache.flink.table.types.logical.LogicalTypeRoot;
> import org.apache.flink.table.types.logical.RowType;
> import javax.annotation.Nonnull;
> import java.util.Optional;
> import java.util.Set;
> /** JDBC dialect for OceanBase. */
> public class OceanBaseDialect extends AbstractDialect {
> private static final long serialVersionUID = 1L;
> private final AbstractDialect dialect;
> public OceanBaseDialect(@Nonnull String compatibleMode) {
> switch (compatibleMode.toLowerCase()) {
> case "mysql":
> this.dialect = new MySqlDialect();
> break;
> case "oracle":
> this.dialect = new OracleDialect();
> break;
> default:
> throw new IllegalArgumentException(
> "Unsupported compatible mode: " + compatibleMode);
> }
> }
> @Override
> public String dialectName() {
> return "OceanBase";
> }
> @Override
> public Optional defaultDriverName() {
> return Optional.of("com.oceanbase.jdbc.Driver");
> }
> @Override
> public Set supportedTypes() {
> return dialect.supportedTypes();
> }
> @Override
> public JdbcRowConverter getRowConverter(RowType rowType) {
> return dialect.getRowConverter(rowType);
> }
> @Override
> public String getLimitClause(long limit) {
> return dialect.getLimitClause(limit);
> }
> @Override
> public String quoteIdentifier(String identifier) {
> return dialect.quoteIdentifier(identifier);
> }
> @Override
> public Optional getUpsertStatement(
> String tableName, String[] fieldNames, String[] conditionFields) {
> return dialect.getUpsertStatement(tableName, fieldNames,
> conditionFields);
> }
> @Override
> public Optional timestampPrecisionRange() {
> return dialect.timestampPrecisionRange();
> }
> @Override
> public Optional decimalPrecisionRange() {
> return dialect.decimalPrecisionRange();
> }
> @Override
> public String appendDefaultUrlProperties(String url) {
> return dialect.appendDefaultUrlProperties(url);
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28879) New File Sink s3 end-to-end test failed with Output hash mismatch
[
https://issues.apache.org/jira/browse/FLINK-28879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-28879:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> New File Sink s3 end-to-end test failed with Output hash mismatch
> -
>
> Key: FLINK-28879
> URL: https://issues.apache.org/jira/browse/FLINK-28879
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / FileSystem, Tests
>Affects Versions: 1.16.0, 1.17.2
>Reporter: Huang Xingbo
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> 2022-08-09T00:50:02.8229585Z Aug 09 00:50:02 FAIL File Streaming Sink: Output
> hash mismatch. Got 6037b01ca0ffc61a95c12cb475c661a8, expected
> 6727342fdd3aae2129e61fc8f433fb6f.
> 2022-08-09T00:50:02.8230700Z Aug 09 00:50:02 head hexdump of actual:
> 2022-08-09T00:50:02.8477319Z Aug 09 00:50:02 000 E r r o r
> e x e c u t i n g
> 2022-08-09T00:50:02.8478206Z Aug 09 00:50:02 010 a w s c o
> m m a n d : s 3
> 2022-08-09T00:50:02.8479475Z Aug 09 00:50:02 020 c p - - q
> u i e t s 3 : / /
> 2022-08-09T00:50:02.8480205Z Aug 09 00:50:02 030 f l i n k -
> i n t e g r a t i o
> 2022-08-09T00:50:02.8480924Z Aug 09 00:50:02 040 n - t e s t
> s / t e m p / t e s
> 2022-08-09T00:50:02.8481612Z Aug 09 00:50:02 050 t _ f i l e
> _ s i n k - 1 d 3 d
> 2022-08-09T00:50:02.8483048Z Aug 09 00:50:02 060 4 0 0 8 - b
> 0 b f - 4 2 6 5 - b
> 2022-08-09T00:50:02.8483618Z Aug 09 00:50:02 070 e 0 e - 3 b
> 9 f 7 8 2 c 5 5 2 d
> 2022-08-09T00:50:02.8484222Z Aug 09 00:50:02 080 / h o s t
> d i r / / t e m p -
> 2022-08-09T00:50:02.8484831Z Aug 09 00:50:02 090 t e s t - d
> i r e c t o r y - 2
> 2022-08-09T00:50:02.8485719Z Aug 09 00:50:02 0a0 3 9 3 7 7 8
> 2 6 8 0 / t e m p /
> 2022-08-09T00:50:02.8486427Z Aug 09 00:50:02 0b0 t e s t _ f
> i l e _ s i n k - 1
> 2022-08-09T00:50:02.8487134Z Aug 09 00:50:02 0c0 d 3 d 4 0 0
> 8 - b 0 b f - 4 2 6
> 2022-08-09T00:50:02.8487826Z Aug 09 00:50:02 0d0 5 - b e 0 e
> - 3 b 9 f 7 8 2 c 5
> 2022-08-09T00:50:02.8488511Z Aug 09 00:50:02 0e0 5 2 d - -
> e x c l u d e ' *
> 2022-08-09T00:50:02.8489202Z Aug 09 00:50:02 0f0 ' - - i n
> c l u d e ' * / p
> 2022-08-09T00:50:02.8489891Z Aug 09 00:50:02 100 a r t - [ !
> / ] * ' - - r e c
> 2022-08-09T00:50:02.8490385Z Aug 09 00:50:02 110 u r s i v e
> \n
> 2022-08-09T00:50:02.8490822Z Aug 09 00:50:02 117
> 2022-08-09T00:50:02.8502212Z Aug 09 00:50:02 Stopping job timeout watchdog
> (with pid=141134)
> 2022-08-09T00:50:06.8430959Z rm: cannot remove
> '/home/vsts/work/1/s/flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/lib/flink-shaded-netty-tcnative-static-*.jar':
> No such file or directory
> 2022-08-09T00:50:06.9278248Z Aug 09 00:50:06
> 5ccfeb22307c2a88625a38b9537acc79001d1b29094ef40fd70692ce11407502
> 2022-08-09T00:50:06.9618147Z Aug 09 00:50:06
> 5ccfeb22307c2a88625a38b9537acc79001d1b29094ef40fd70692ce11407502
> 2022-08-09T00:50:06.9645077Z Aug 09 00:50:06 [FAIL] Test script contains
> errors.
> 2022-08-09T00:50:06.9666227Z Aug 09 00:50:06 Checking of logs skipped.
> 2022-08-09T00:50:06.9671891Z Aug 09 00:50:06
> 2022-08-09T00:50:06.9673050Z Aug 09 00:50:06 [FAIL] 'New File Sink s3
> end-to-end test' failed after 3 minutes and 42 seconds! Test exited with exit
> code 1
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39667=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=4136
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32570) Deprecate API that uses Flink's Time implementation (related to FLINK-14638)
[ https://issues.apache.org/jira/browse/FLINK-32570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32570: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Deprecate API that uses Flink's Time implementation (related to FLINK-14638) > > > Key: FLINK-32570 > URL: https://issues.apache.org/jira/browse/FLINK-32570 > Project: Flink > Issue Type: Technical Debt >Affects Versions: 1.19.0 >Reporter: Matthias Pohl >Assignee: Matthias Pohl >Priority: Major > Labels: pull-request-available, stale-assigned > > The plan is to resolve FLINK-14038 with Flink 2.0. As a preparation, we have > to deprecate related @Public API . -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31825) Stopping minikube fails with timeout
[
https://issues.apache.org/jira/browse/FLINK-31825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31825:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Stopping minikube fails with timeout
>
>
> Key: FLINK-31825
> URL: https://issues.apache.org/jira/browse/FLINK-31825
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hadoop Compatibility, Tests
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> Currently there is not so much information in logs...
> {noformat}
> Apr 13 01:27:02 * Stopping node "minikube" ...
> ==
> === WARNING: This task took already 95% of the available time budget of 286
> minutes ===
> ==
> ==
> The following Java processes are running (JPS)
> ==
> 243413 Jps
> ==
> Printing stack trace of Java process 243413
> ==
> 243413: No such process
> ==
> The following Java processes are running (JPS)
> ==
> 243516 Jps
> ==
> Printing stack trace of Java process 243516
> ==
> 243516: No such process
> =
> === WARNING: Killing task ===
> =
> Terminated
> Apr 13 05:48:53 [FAIL] Test script contains errors.
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48096=logs=bbb1e2a2-a43c-55c8-fb48-5cfe7a8a0ca6=ba24ad14-6ea3-5ee3-c4ec-9e7cd2c9e754=5290
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32722) 239 exit code in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32722:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> 239 exit code in flink-runtime
> --
>
> Key: FLINK-32722
> URL: https://issues.apache.org/jira/browse/FLINK-32722
> Project: Flink
> Issue Type: Bug
> Components: Test Infrastructure
>Affects Versions: 1.16.2
>Reporter: Matthias Pohl
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51852=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=be5a4b15-4b23-56b1-7582-795f58a645a2=8418
> {code:java}
> Aug 01 01:03:49 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test)
> on project flink-runtime: There are test failures.
> Aug 01 01:03:49 [ERROR]
> Aug 01 01:03:49 [ERROR] Please refer to
> /__w/2/s/flink-runtime/target/surefire-reports for the individual test
> results.
> Aug 01 01:03:49 [ERROR] Please refer to dump files (if any exist)
> [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Aug 01 01:03:49 [ERROR] ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> Aug 01 01:03:49 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseG1GC -Xms256m -Xmx768m
> -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter1803120121827605294.jar
> /__w/2/s/flink-runtime/target/surefire 2023-08-01T00-58-17_520-jvmRun1
> surefire9107652818401825168tmp surefire_261701267003130520249tmp
> Aug 01 01:03:49 [ERROR] Error occurred in starting fork, check output in log
> Aug 01 01:03:49 [ERROR] Process Exit Code: 239
> Aug 01 01:03:49 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Aug 01 01:03:49 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseG1GC -Xms256m -Xmx768m
> -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter1803120121827605294.jar
> /__w/2/s/flink-runtime/target/surefire 2023-08-01T00-58-17_520-jvmRun1
> surefire9107652818401825168tmp surefire_261701267003130520249tmp
> Aug 01 01:03:49 [ERROR] Error occurred in starting fork, check output in log
> Aug 01 01:03:49 [ERROR] Process Exit Code: 239
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:405)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:321)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> Aug 01 01:03:49 [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> Aug 01 01:03:49
[jira] [Updated] (FLINK-29390) Pulsar SQL Connector: SQLClient E2E testing
[ https://issues.apache.org/jira/browse/FLINK-29390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-29390: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Pulsar SQL Connector: SQLClient E2E testing > --- > > Key: FLINK-29390 > URL: https://issues.apache.org/jira/browse/FLINK-29390 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > Labels: pull-request-available, stale-assigned > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32665) Support read null value for csv format
[ https://issues.apache.org/jira/browse/FLINK-32665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32665: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support read null value for csv format > -- > > Key: FLINK-32665 > URL: https://issues.apache.org/jira/browse/FLINK-32665 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available, stale-major > > when there is null column in a file with csv format, it will throw exception > when flink job try to parse these data -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28440) EventTimeWindowCheckpointingITCase failed with restore
[
https://issues.apache.org/jira/browse/FLINK-28440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-28440:
---
Labels: auto-deprioritized-critical pull-request-available stale-assigned
test-stability (was: auto-deprioritized-critical pull-request-available
test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> EventTimeWindowCheckpointingITCase failed with restore
> --
>
> Key: FLINK-28440
> URL: https://issues.apache.org/jira/browse/FLINK-28440
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.16.0, 1.17.0, 1.18.0
>Reporter: Huang Xingbo
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: auto-deprioritized-critical, pull-request-available,
> stale-assigned, test-stability
> Fix For: 1.18.0
>
> Attachments: image-2023-02-01-00-51-54-506.png,
> image-2023-02-01-01-10-01-521.png, image-2023-02-01-01-19-12-182.png,
> image-2023-02-01-16-47-23-756.png, image-2023-02-01-16-57-43-889.png,
> image-2023-02-02-10-52-56-599.png, image-2023-02-03-10-09-07-586.png,
> image-2023-02-03-12-03-16-155.png, image-2023-02-03-12-03-56-614.png
>
>
> {code:java}
> Caused by: java.lang.Exception: Exception while creating
> StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:256)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:722)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:698)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:665)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:904)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for WindowOperator_0a448493b4782967b150582570326227_(2/4) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:353)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:165)
> ... 11 more
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
> /tmp/junit1835099326935900400/junit1113650082510421526/52ee65b7-033f-4429-8ddd-adbe85e27ced
> (No such file or directory)
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:87)
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.hasNext(StateChangelogHandleStreamHandleReader.java:69)
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.readBackendHandle(ChangelogBackendRestoreOperation.java:96)
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.restore(ChangelogBackendRestoreOperation.java:75)
> at
> org.apache.flink.state.changelog.ChangelogStateBackend.restore(ChangelogStateBackend.java:92)
> at
>
[jira] [Updated] (FLINK-32910) Silence curls in test code
[
https://issues.apache.org/jira/browse/FLINK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32910:
---
Labels: pull-request-available stale-assigned starter (was:
pull-request-available starter)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Silence curls in test code
> --
>
> Key: FLINK-32910
> URL: https://issues.apache.org/jira/browse/FLINK-32910
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
>Affects Versions: 1.16.2, 1.18.0, 1.17.1
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available, stale-assigned, starter
>
> We use {{curl}} in several locations to download artifacts. Usually, the a
> progress bar is printed which spams the console output of the test execution.
> This issue is about cleaning this up.
> Parameters to consider (depending on the usecase):
> * {{\-L}}/{{\-\-location}} redirects the curl command and retries if the
> server reported that the artifact was moved
> * {{\-O}}/{{\-\-remote-name}} writes output to file matching the remote name
> (which was extracted from the URL) instead of stdout; alternative:
> {{\-o}}/{{\-\-output}} writes output to a file with the given name instead of
> stdout
> * {{\-f}}/{{\-\-fail}} makes curl command fail with non-0 exit code for HTTP
> error codes
> * {{\-s \-S}}/{{\-\-silent \-\-show-error}} doesn't print progress bar but
> shows error
> * {{\-r}}/{{\-\-retry}} Retries certain errors
> {{curl}} uses a default config file {{${user.home}/.curlrc}}. But one could
> make it more explicit using {{\-K}}/{{\-\-config}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33056) NettyClientServerSslTest#testValidSslConnection fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33056:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Critical,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> NettyClientServerSslTest#testValidSslConnection fails on AZP
>
>
> Key: FLINK-33056
> URL: https://issues.apache.org/jira/browse/FLINK-33056
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration, Runtime / Coordination
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
> Attachments: logs-cron_azure-test_cron_azure_core-1694048924.zip
>
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53020=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=8592
> fails with
> {noformat}
> Test testValidSslConnection[SSL provider =
> JDK](org.apache.flink.runtime.io.network.netty.NettyClientServerSslTest) is
> running.
>
> 01:20:31,479 [main] INFO
> org.apache.flink.runtime.io.network.netty.NettyConfig[] - NettyConfig
> [server address: localhost/127.0.0.1, server port range: 36717, ssl enabled:
> true, memory segment size (bytes): 1024, transport type: AUTO, number of
> server threads: 1 (manual), number of client thr
> eads: 1 (manual), server connect backlog: 0 (use Netty's default), client
> connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
> default)]
> 01:20:31,479 [main] INFO
> org.apache.flink.runtime.io.network.netty.NettyServer[] - Transport
> type 'auto': using EPOLL.
> 01:20:31,475 [Flink Netty Client (42359) Thread 0] WARN
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline [] -
> An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> javax.net.ssl.SSLHandshakeException: server certificate with unknown
> fingerprint: CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
> ~[flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
> ~[flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
>
[jira] [Updated] (FLINK-31406) Do not delete jobgraph on scale only last-state upgrades
[ https://issues.apache.org/jira/browse/FLINK-31406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31406: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Do not delete jobgraph on scale only last-state upgrades > > > Key: FLINK-31406 > URL: https://issues.apache.org/jira/browse/FLINK-31406 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available, stale-assigned > > Currently the operator always deletes the jobgraph from HA metadata so that > it's regenerated for last-state upgrades. > This is unnecessary for scale only operations. Keeping the jobgraph can > greately speed up startup time for some jobs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-16987) FLIP-95: Add new table source and sink interfaces
[ https://issues.apache.org/jira/browse/FLINK-16987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16987: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > FLIP-95: Add new table source and sink interfaces > - > > Key: FLINK-16987 > URL: https://issues.apache.org/jira/browse/FLINK-16987 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > Proper support for handling changelogs, more efficient processing of data > through the new Blink planner, and unified interfaces that are DataStream API > agnostic make it necessary to rework the table source and sink interfaces. > The goals of this FLIP are: > * *Simplify the current interface architecture*: > ** Merge upsert, retract, and append sinks. > ** Unify batch and streaming sources. > ** Unify batch and streaming sinks. > * *Allow sources to produce a changelog*: > ** UpsertTableSources have been requested a lot by users. Now is the time to > open the internal planner capabilities via the new interfaces. > ** According to FLIP-105, we would like to support changelogs for processing > formats such as [Debezium|https://debezium.io/]. > * *Don't rely on DataStream API for source and sinks*: > ** According to FLIP-32, the Table API and SQL should be independent of the > DataStream API which is why the `table-common` module has no dependencies on > `flink-streaming-java`. > ** Source and sink implementations should only depend on the `table-common` > module after FLIP-27. > ** Until FLIP-27 is ready, we still put most of the interfaces in > `table-common` and strictly separate interfaces that communicate with a > planner and actual runtime reader/writers. > * *Implement efficient sources and sinks without planner dependencies*: > ** Make Blink's internal data structures available to connectors. > ** Introduce stable interfaces for data structures that can be marked as > `@PublicEvolving`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31615) Fix some parts forgot to translate in "Table API" page of "Table API & SQL"
[ https://issues.apache.org/jira/browse/FLINK-31615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31615: --- Labels: chinese-translation stale-minor (was: chinese-translation) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Fix some parts forgot to translate in "Table API" page of "Table API & SQL" > > > Key: FLINK-31615 > URL: https://issues.apache.org/jira/browse/FLINK-31615 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Hang Ruan >Priority: Minor > Labels: chinese-translation, stale-minor > > The query_state_warning in "Table API" page of "Table API & SQL" is still in > English. And some comments in codes are in English. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-13876) Remove ExecutionConfig field from PojoSerializer
[
https://issues.apache.org/jira/browse/FLINK-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-13876:
---
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Remove ExecutionConfig field from PojoSerializer
>
>
> Key: FLINK-13876
> URL: https://issues.apache.org/jira/browse/FLINK-13876
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
>Reporter: Dawid Wysakowicz
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> The PojoSerializers stores an instance of ExecutionConfig as internal field,
> even though that the only information it may ever need are the registered
> kryo serializers.
> This has a few drawbacks:
> * It blocks the evolution of {{ExecutionConfig}} as serializers where stored
> in a state. Therefore any change to ExecutionConfig must be backwards
> compatible in respect to java serialization
> * It probably already introduced a bug, as upon restore the Snapshot actually
> recreates the serializer with an empty ExecutionConfig (see
> org.apache.flink.api.java.typeutils.runtime.PojoSerializerSnapshot#restoreSerializer)
> I suggest to remove the field completely and adjust corresponding usages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32866) Unify the TestLoggerExtension config of junit5
[
https://issues.apache.org/jira/browse/FLINK-32866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32866:
---
Labels: pull-request-available stale-assigned starter (was:
pull-request-available starter)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Unify the TestLoggerExtension config of junit5
> --
>
> Key: FLINK-32866
> URL: https://issues.apache.org/jira/browse/FLINK-32866
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Minor
> Labels: pull-request-available, stale-assigned, starter
>
> Some modules added the {{TestLoggerExtension}} to the
> {{org.junit.jupiter.api.extension.Extension}} resource file. All test classes
> of these modules don't need add the
> {{@ExtendWith(TestLoggerExtension.class)}} at class level.
> This JIRA propose clean up the {{@ExtendWith(TestLoggerExtension.class)}}
> for modules that added the {{TestLoggerExtension}} to the
> {{org.junit.jupiter.api.extension.Extension}} resource file.
> Update: We could also investigate in what extend we could remove the
> {{org.junit.jupiter.api.extension.Extension}} resource files from certain
> modules. It only has to be present once on the classpath. A single location
> for this configuration would be nice (e.g. putting it in
> {{flink-test-utils-parent/flink-test-utils-junit}}). But I think not all
> modules have this one as there dependency. So, having a single resource file
> for configuring this might be not possible. But at least, we can reduce the
> number of files to a minimum.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32582) Move TypeSerializerUpgradeTestBase from Kafka connector into flink-connector-common
[
https://issues.apache.org/jira/browse/FLINK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32582:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Move TypeSerializerUpgradeTestBase from Kafka connector into
> flink-connector-common
> ---
>
> Key: FLINK-32582
> URL: https://issues.apache.org/jira/browse/FLINK-32582
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Reporter: Matthias Pohl
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> The externalization of connectors made caused problems with the Flink's test
> data generation. The Kafka connector relied on TypeSerializerUpgradeTestBase
> for some test cases which was fine prior to FLINK-27518 where the test data
> generation was handled individually.
> With FLINK-27518 the process was automated in Flink 1.18. For now, the
> TypeSerializerUpgradeTestBase class was just copied over into the Kafka
> connector since it was the only connector that would utilize this test base.
> But we might want to provide a more generalized solution where the test base
> is provided by {{flink-connector-common}} to offer a generalized approach for
> any connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33025) BatchArrowPythonOverWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33025:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Critical,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> BatchArrowPythonOverWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount
> fails on AZP
> -
>
> Key: FLINK-33025
> URL: https://issues.apache.org/jira/browse/FLINK-33025
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.3, 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=52958=logs=5cae8624-c7eb-5c51-92d3-4d2dacedd221=5acec1b4-945b-59ca-34f8-168928ce5199=22618
> fails on AZP as
> {noformat}
> Sep 03 05:05:38 05:05:38.220 [ERROR] Failures:
> Sep 03 05:05:38 05:05:38.220 [ERROR]
> BatchArrowPythonOverWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount:122->ArrowPythonAggregateFunctionOperatorTestBase.assertOutputEquals:62
>
> Sep 03 05:05:38 Expected size: 4 but was: 3 in:
> Sep 03 05:05:38 [Record @ (undef) : +I(c1,c2,0,0,0),
> Sep 03 05:05:38 Record @ (undef) : +I(c1,c4,1,0,0),
> Sep 03 05:05:38 Record @ (undef) : +I(c1,c6,2,10,2)]
> {noformat}
> probably related to FLINK-26990
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32471) IS_NOT_NULL can add to SUITABLE_FILTER_TO_PUSH
[ https://issues.apache.org/jira/browse/FLINK-32471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32471: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > IS_NOT_NULL can add to SUITABLE_FILTER_TO_PUSH > -- > > Key: FLINK-32471 > URL: https://issues.apache.org/jira/browse/FLINK-32471 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Reporter: grandfisher >Priority: Major > Labels: pull-request-available, stale-major > > According to FLINK-31273: > The reason for the error is that other filters conflict with IS_NULL, but in > fact it won't conflict with IS_NOT_NULL, because operators in > SUITABLE_FILTER_TO_PUSH such as 'SqlKind.GREATER_THAN' has an implicit > filter 'IS_NOT_NULL' according to SQL Semantics. > > So we think it is feasible to add IS_NOT_NULL to the SUITABLE_FILTER_TO_PUSH > list. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30519) Add e2e tests for operator dynamic config
[ https://issues.apache.org/jira/browse/FLINK-30519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-30519: --- Labels: auto-deprioritized-critical stale-assigned starter (was: stale-assigned stale-critical starter) Priority: Major (was: Critical) This issue was labeled "stale-critical" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Critical, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Add e2e tests for operator dynamic config > - > > Key: FLINK-30519 > URL: https://issues.apache.org/jira/browse/FLINK-30519 > Project: Flink > Issue Type: Technical Debt > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Labels: auto-deprioritized-critical, stale-assigned, starter > > The dynamic config feature is currently not covered by e2e tests and is > subject to accidental regressions, as shown in: > https://issues.apache.org/jira/browse/FLINK-30329 > We should add an e2e test that covers this -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26974) Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-26974:
---
Labels: auto-deprioritized-major stale-assigned test-stability (was:
auto-deprioritized-major test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
> -
>
> Key: FLINK-26974
> URL: https://issues.apache.org/jira/browse/FLINK-26974
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0, 1.16.0, 1.17.0
>Reporter: Yun Gao
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: auto-deprioritized-major, stale-assigned, test-stability
>
> {code:java}
> Mar 31 10:49:17 === FAILURES
> ===
> Mar 31 10:49:17 __
> EmbeddedThreadDependencyTests.test_add_python_file __
> Mar 31 10:49:17
> Mar 31 10:49:17 self =
> testMethod=test_add_python_file>
> Mar 31 10:49:17
> Mar 31 10:49:17 def test_add_python_file(self):
> Mar 31 10:49:17 python_file_dir = os.path.join(self.tempdir,
> "python_file_dir_" + str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir)
> Mar 31 10:49:17 python_file_path = os.path.join(python_file_dir,
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nraise
> Exception('This function should not be called!')")
> Mar 31 10:49:17 self.t_env.add_python_file(python_file_path)
> Mar 31 10:49:17
> Mar 31 10:49:17 python_file_dir_with_higher_priority = os.path.join(
> Mar 31 10:49:17 self.tempdir, "python_file_dir_" +
> str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir_with_higher_priority)
> Mar 31 10:49:17 python_file_path_higher_priority =
> os.path.join(python_file_dir_with_higher_priority,
> Mar 31 10:49:17
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path_higher_priority, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nreturn a + 2")
> Mar 31 10:49:17
> self.t_env.add_python_file(python_file_path_higher_priority)
> Mar 31 10:49:17
> Mar 31 10:49:17 def plus_two(i):
> Mar 31 10:49:17 from test_dependency_manage_lib import add_two
> Mar 31 10:49:17 return add_two(i)
> Mar 31 10:49:17
> Mar 31 10:49:17 self.t_env.create_temporary_system_function(
> Mar 31 10:49:17 "add_two", udf(plus_two, DataTypes.BIGINT(),
> DataTypes.BIGINT()))
> Mar 31 10:49:17 table_sink = source_sink_utils.TestAppendSink(
> Mar 31 10:49:17 ['a', 'b'], [DataTypes.BIGINT(),
> DataTypes.BIGINT()])
> Mar 31 10:49:17 self.t_env.register_table_sink("Results", table_sink)
> Mar 31 10:49:17 t = self.t_env.from_elements([(1, 2), (2, 5), (3,
> 1)], ['a', 'b'])
> Mar 31 10:49:17 > t.select(expr.call("add_two", t.a),
> t.a).execute_insert("Results").wait()
> Mar 31 10:49:17
> Mar 31 10:49:17 pyflink/table/tests/test_dependency.py:63:
> Mar 31 10:49:17 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _
> Mar 31 10:49:17 pyflink/table/table_result.py:76: in wait
> Mar 31 10:49:17 get_method(self._j_table_result, "await")()
> Mar 31 10:49:17
> .tox/py38-cython/lib/python3.8/site-packages/py4j/java_gateway.py:1321: in
> __call__
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34001=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=27239
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32605) JoinITCase.testFullOuterJoinWithMultipleKeys fails with TimeoutException: Futures timed out after [20 seconds]
[
https://issues.apache.org/jira/browse/FLINK-32605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32605:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> JoinITCase.testFullOuterJoinWithMultipleKeys fails with TimeoutException:
> Futures timed out after [20 seconds]
> --
>
> Key: FLINK-32605
> URL: https://issues.apache.org/jira/browse/FLINK-32605
> Project: Flink
> Issue Type: Bug
> Components: Runtime / RPC, Table SQL / API
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> While execution of
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51254=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11681
> there was an exception
> {noformat}
> Jul 14 04:35:32 Caused by: java.lang.Exception: Could not create actor system
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaBootstrapTools.startLocalActorSystem(AkkaBootstrapTools.java:238)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:349)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:327)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:247)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.minicluster.MiniCluster.createLocalRpcService(MiniCluster.java:1188)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:355)
> Jul 14 04:35:32 at
> org.apache.flink.client.program.PerJobMiniClusterFactory.submitJob(PerJobMiniClusterFactory.java:77)
> Jul 14 04:35:32 at
> org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:85)
> Jul 14 04:35:32 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2218)
> Jul 14 04:35:32 at
> org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110)
> Jul 14 04:35:32 at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeQueryOperation(TableEnvironmentImpl.java:992)
> Jul 14 04:35:32 ... 102 more
> Jul 14 04:35:32 Caused by: java.util.concurrent.TimeoutException: Futures
> timed out after [20 seconds]
> Jul 14 04:35:32 at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
> Jul 14 04:35:32 at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
> Jul 14 04:35:32 at
> scala.concurrent.Await$.$anonfun$result$1(package.scala:223)
> Jul 14 04:35:32 at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
> Jul 14 04:35:32 at scala.concurrent.Await$.result(package.scala:146)
> Jul 14 04:35:32 at
> akka.stream.SystemMaterializer.(SystemMaterializer.scala:90)
> Jul 14 04:35:32 at
> akka.stream.SystemMaterializer$.createExtension(SystemMaterializer.scala:39)
> Jul 14 04:35:32 at
> akka.stream.SystemMaterializer$.createExtension(SystemMaterializer.scala:32)
> Jul 14 04:35:32 at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1165)
> Jul 14 04:35:32 at
> akka.actor.ActorSystemImpl.$anonfun$loadExtensions$1(ActorSystem.scala:1208)
> Jul 14 04:35:32 at scala.collection.Iterator.foreach(Iterator.scala:943)
> Jul 14 04:35:32 at
> scala.collection.Iterator.foreach$(Iterator.scala:943)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.RobustActorSystem.create(RobustActorSystem.java:54)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaUtils.createActorSystem(AkkaUtils.java:421)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaBootstrapTools.startActorSystem(AkkaBootstrapTools.java:253)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaBootstrapTools.startLocalActorSystem(AkkaBootstrapTools.java:236)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28879) New File Sink s3 end-to-end test failed with Output hash mismatch
[
https://issues.apache.org/jira/browse/FLINK-28879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-28879:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> New File Sink s3 end-to-end test failed with Output hash mismatch
> -
>
> Key: FLINK-28879
> URL: https://issues.apache.org/jira/browse/FLINK-28879
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / FileSystem, Tests
>Affects Versions: 1.16.0, 1.17.2
>Reporter: Huang Xingbo
>Priority: Major
> Labels: stale-major, test-stability
>
> {code:java}
> 2022-08-09T00:50:02.8229585Z Aug 09 00:50:02 FAIL File Streaming Sink: Output
> hash mismatch. Got 6037b01ca0ffc61a95c12cb475c661a8, expected
> 6727342fdd3aae2129e61fc8f433fb6f.
> 2022-08-09T00:50:02.8230700Z Aug 09 00:50:02 head hexdump of actual:
> 2022-08-09T00:50:02.8477319Z Aug 09 00:50:02 000 E r r o r
> e x e c u t i n g
> 2022-08-09T00:50:02.8478206Z Aug 09 00:50:02 010 a w s c o
> m m a n d : s 3
> 2022-08-09T00:50:02.8479475Z Aug 09 00:50:02 020 c p - - q
> u i e t s 3 : / /
> 2022-08-09T00:50:02.8480205Z Aug 09 00:50:02 030 f l i n k -
> i n t e g r a t i o
> 2022-08-09T00:50:02.8480924Z Aug 09 00:50:02 040 n - t e s t
> s / t e m p / t e s
> 2022-08-09T00:50:02.8481612Z Aug 09 00:50:02 050 t _ f i l e
> _ s i n k - 1 d 3 d
> 2022-08-09T00:50:02.8483048Z Aug 09 00:50:02 060 4 0 0 8 - b
> 0 b f - 4 2 6 5 - b
> 2022-08-09T00:50:02.8483618Z Aug 09 00:50:02 070 e 0 e - 3 b
> 9 f 7 8 2 c 5 5 2 d
> 2022-08-09T00:50:02.8484222Z Aug 09 00:50:02 080 / h o s t
> d i r / / t e m p -
> 2022-08-09T00:50:02.8484831Z Aug 09 00:50:02 090 t e s t - d
> i r e c t o r y - 2
> 2022-08-09T00:50:02.8485719Z Aug 09 00:50:02 0a0 3 9 3 7 7 8
> 2 6 8 0 / t e m p /
> 2022-08-09T00:50:02.8486427Z Aug 09 00:50:02 0b0 t e s t _ f
> i l e _ s i n k - 1
> 2022-08-09T00:50:02.8487134Z Aug 09 00:50:02 0c0 d 3 d 4 0 0
> 8 - b 0 b f - 4 2 6
> 2022-08-09T00:50:02.8487826Z Aug 09 00:50:02 0d0 5 - b e 0 e
> - 3 b 9 f 7 8 2 c 5
> 2022-08-09T00:50:02.8488511Z Aug 09 00:50:02 0e0 5 2 d - -
> e x c l u d e ' *
> 2022-08-09T00:50:02.8489202Z Aug 09 00:50:02 0f0 ' - - i n
> c l u d e ' * / p
> 2022-08-09T00:50:02.8489891Z Aug 09 00:50:02 100 a r t - [ !
> / ] * ' - - r e c
> 2022-08-09T00:50:02.8490385Z Aug 09 00:50:02 110 u r s i v e
> \n
> 2022-08-09T00:50:02.8490822Z Aug 09 00:50:02 117
> 2022-08-09T00:50:02.8502212Z Aug 09 00:50:02 Stopping job timeout watchdog
> (with pid=141134)
> 2022-08-09T00:50:06.8430959Z rm: cannot remove
> '/home/vsts/work/1/s/flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/lib/flink-shaded-netty-tcnative-static-*.jar':
> No such file or directory
> 2022-08-09T00:50:06.9278248Z Aug 09 00:50:06
> 5ccfeb22307c2a88625a38b9537acc79001d1b29094ef40fd70692ce11407502
> 2022-08-09T00:50:06.9618147Z Aug 09 00:50:06
> 5ccfeb22307c2a88625a38b9537acc79001d1b29094ef40fd70692ce11407502
> 2022-08-09T00:50:06.9645077Z Aug 09 00:50:06 [FAIL] Test script contains
> errors.
> 2022-08-09T00:50:06.9666227Z Aug 09 00:50:06 Checking of logs skipped.
> 2022-08-09T00:50:06.9671891Z Aug 09 00:50:06
> 2022-08-09T00:50:06.9673050Z Aug 09 00:50:06 [FAIL] 'New File Sink s3
> end-to-end test' failed after 3 minutes and 42 seconds! Test exited with exit
> code 1
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39667=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=4136
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31860) FlinkDeployments never finalize when namespace is deleted
[
https://issues.apache.org/jira/browse/FLINK-31860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31860:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> FlinkDeployments never finalize when namespace is deleted
> -
>
> Key: FLINK-31860
> URL: https://issues.apache.org/jira/browse/FLINK-31860
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.3.1
> Environment: Apache Flink Kubernetes Operator 1.3.1
> Kubernetes 1.24.9
>Reporter: Jayme Howard
>Assignee: Jayme Howard
>Priority: Blocker
> Labels: pull-request-available, stale-assigned
>
> This appears to be a pretty straightforward issue, but I don't know the
> codebase well enough to propose a fix. When a FlinkDeployment is present in
> a namespace, and the namespace is deleted, the FlinkDeployment never
> reconciles and fails to complete its finalizer. This leads to the namespace
> being blocked from deletion indefinitely, requiring manual manipulation to
> remove the finalizer on the FlinkDeployment.
>
> Namespace conditions:
> {code:java}
> conditions:
> - lastTransitionTime: '2023-04-18T22:17:48Z'
> message: All resources successfully discovered
> reason: ResourcesDiscovered
> status: 'False'
> type: NamespaceDeletionDiscoveryFailure
> - lastTransitionTime: '2023-03-23T18:27:37Z'
> message: All legacy kube types successfully parsed
> reason: ParsedGroupVersions
> status: 'False'
> type: NamespaceDeletionGroupVersionParsingFailure
> - lastTransitionTime: '2023-03-23T18:27:37Z'
> message: All content successfully deleted, may be waiting on finalization
> reason: ContentDeleted
> status: 'False'
> type: NamespaceDeletionContentFailure
> - lastTransitionTime: '2023-03-23T18:27:37Z'
> message: 'Some resources are remaining: flinkdeployments.flink.apache.org
> has 2
> resource instances'
> reason: SomeResourcesRemain
> status: 'True'
> type: NamespaceContentRemaining
> - lastTransitionTime: '2023-03-23T18:27:37Z'
> message: 'Some content in the namespace has finalizers remaining:
> flinkdeployments.flink.apache.org/finalizer
> in 2 resource instances'
> reason: SomeFinalizersRemain
> status: 'True'
> type: NamespaceFinalizersRemaining
> phase: Terminating {code}
> FlinkDeployment example (some fields redacted):
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkDeployment
> metadata:
> creationTimestamp: '2023-03-23T18:27:02Z'
> deletionGracePeriodSeconds: 0
> deletionTimestamp: '2023-03-23T18:27:35Z'
> finalizers:
> - flinkdeployments.flink.apache.org/finalizer
> generation: 3
> name:
> namespace:
> resourceVersion: '10565277081'
> uid: e50d2683-6c0c-467e-b10c-fe0f4e404692
> spec:
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: '2'
> flinkVersion: v1_16
> image:
> job:
> args: []
> entryClass:
> jarURI:
> parallelism: 2
> state: running
> upgradeMode: stateless
> jobManager:
> replicas: 1
> resource:
> cpu: 1
> memory: 2048m
> logConfiguration:
> log4j-console.properties: '# This affects logging for both user code and
> Flink rootLogger.level = INFO rootLogger.appenderRef.console.ref =
> ConsoleAppender rootLogger.appenderRef.rolling.ref = RollingFileAppender
> # Uncomment this if you want to _only_ change Flink''s logging
> #logger.flink.name = org.apache.flink #logger.flink.level = INFO #
> The following lines keep the log level of common libraries/connectors on
> # log level INFO. The root logger does not override this. You have to
> manually # change the log levels here. logger.akka.name = akka
> logger.akka.level = INFO logger.kafka.name= org.apache.kafka
> logger.kafka.level = INFO logger.hadoop.name = org.apache.hadoop
> logger.hadoop.level = INFO logger.zookeeper.name = org.apache.zookeeper
> logger.zookeeper.level = INFO # Log all infos to the console
> appender.console.name = ConsoleAppender
[jira] [Updated] (FLINK-31951) Mix schema record source creates corrupt record
[
https://issues.apache.org/jira/browse/FLINK-31951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31951:
---
Labels: fix-provided pull-request-available stale-assigned (was:
fix-provided pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Mix schema record source creates corrupt record
> ---
>
> Key: FLINK-31951
> URL: https://issues.apache.org/jira/browse/FLINK-31951
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Kevin Tseng
>Assignee: Kevin Tseng
>Priority: Minor
> Labels: fix-provided, pull-request-available, stale-assigned
> Fix For: 1.18.0
>
>
> This seems to be an unexpected side effect with how AvroDeserializationSchema
> class was written.
> Sometimes we do not have control over what record comes through a Kafka Topic.
> In current implementation, if AvroDeserializationSchema encountered a record
> byte array that does not conform to the specified Schema / SpecificRecord
> type, it will cause future record to be deserialized incorrectly.
> Origin of the issue is with how
> {code:java}
> AvroDeserializationSchema.deserialize{code}
> handles exception, and how
> {code:java}
> AvroDeserializationSchema.checkAvroInitialized{code}
> handles initialization of Decoder object
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31825) Stopping minikube fails with timeout
[
https://issues.apache.org/jira/browse/FLINK-31825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31825:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Stopping minikube fails with timeout
>
>
> Key: FLINK-31825
> URL: https://issues.apache.org/jira/browse/FLINK-31825
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hadoop Compatibility, Tests
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: stale-major, test-stability
>
> Currently there is not so much information in logs...
> {noformat}
> Apr 13 01:27:02 * Stopping node "minikube" ...
> ==
> === WARNING: This task took already 95% of the available time budget of 286
> minutes ===
> ==
> ==
> The following Java processes are running (JPS)
> ==
> 243413 Jps
> ==
> Printing stack trace of Java process 243413
> ==
> 243413: No such process
> ==
> The following Java processes are running (JPS)
> ==
> 243516 Jps
> ==
> Printing stack trace of Java process 243516
> ==
> 243516: No such process
> =
> === WARNING: Killing task ===
> =
> Terminated
> Apr 13 05:48:53 [FAIL] Test script contains errors.
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48096=logs=bbb1e2a2-a43c-55c8-fb48-5cfe7a8a0ca6=ba24ad14-6ea3-5ee3-c4ec-9e7cd2c9e754=5290
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-16987) FLIP-95: Add new table source and sink interfaces
[ https://issues.apache.org/jira/browse/FLINK-16987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16987: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > FLIP-95: Add new table source and sink interfaces > - > > Key: FLINK-16987 > URL: https://issues.apache.org/jira/browse/FLINK-16987 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > Labels: pull-request-available, stale-major > > Proper support for handling changelogs, more efficient processing of data > through the new Blink planner, and unified interfaces that are DataStream API > agnostic make it necessary to rework the table source and sink interfaces. > The goals of this FLIP are: > * *Simplify the current interface architecture*: > ** Merge upsert, retract, and append sinks. > ** Unify batch and streaming sources. > ** Unify batch and streaming sinks. > * *Allow sources to produce a changelog*: > ** UpsertTableSources have been requested a lot by users. Now is the time to > open the internal planner capabilities via the new interfaces. > ** According to FLIP-105, we would like to support changelogs for processing > formats such as [Debezium|https://debezium.io/]. > * *Don't rely on DataStream API for source and sinks*: > ** According to FLIP-32, the Table API and SQL should be independent of the > DataStream API which is why the `table-common` module has no dependencies on > `flink-streaming-java`. > ** Source and sink implementations should only depend on the `table-common` > module after FLIP-27. > ** Until FLIP-27 is ready, we still put most of the interfaces in > `table-common` and strictly separate interfaces that communicate with a > planner and actual runtime reader/writers. > * *Implement efficient sources and sinks without planner dependencies*: > ** Make Blink's internal data structures available to connectors. > ** Introduce stable interfaces for data structures that can be marked as > `@PublicEvolving`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32851) [JUnit5 Migration] The rest package of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32851: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > [JUnit5 Migration] The rest package of flink-runtime module > --- > > Key: FLINK-32851 > URL: https://issues.apache.org/jira/browse/FLINK-32851 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Matt Wang >Priority: Minor > Labels: pull-request-available, stale-assigned > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33056) NettyClientServerSslTest#testValidSslConnection fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33056:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> NettyClientServerSslTest#testValidSslConnection fails on AZP
>
>
> Key: FLINK-33056
> URL: https://issues.apache.org/jira/browse/FLINK-33056
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration, Runtime / Coordination
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
> Attachments: logs-cron_azure-test_cron_azure_core-1694048924.zip
>
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53020=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=8592
> fails with
> {noformat}
> Test testValidSslConnection[SSL provider =
> JDK](org.apache.flink.runtime.io.network.netty.NettyClientServerSslTest) is
> running.
>
> 01:20:31,479 [main] INFO
> org.apache.flink.runtime.io.network.netty.NettyConfig[] - NettyConfig
> [server address: localhost/127.0.0.1, server port range: 36717, ssl enabled:
> true, memory segment size (bytes): 1024, transport type: AUTO, number of
> server threads: 1 (manual), number of client thr
> eads: 1 (manual), server connect backlog: 0 (use Netty's default), client
> connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
> default)]
> 01:20:31,479 [main] INFO
> org.apache.flink.runtime.io.network.netty.NettyServer[] - Transport
> type 'auto': using EPOLL.
> 01:20:31,475 [Flink Netty Client (42359) Thread 0] WARN
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline [] -
> An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> javax.net.ssl.SSLHandshakeException: server certificate with unknown
> fingerprint: CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
> ~[flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
> ~[flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
>
[jira] [Updated] (FLINK-32995) TPC-DS end-to-end test fails with chmod: cannot access '../target/generator/dsdgen_linux':
[
https://issues.apache.org/jira/browse/FLINK-32995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32995:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Critical,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> TPC-DS end-to-end test fails with chmod: cannot access
> '../target/generator/dsdgen_linux':
> ---
>
> Key: FLINK-32995
> URL: https://issues.apache.org/jira/browse/FLINK-32995
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=52773=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=0f3adb59-eefa-51c6-2858-3654d9e0749d=5504
> fails as
> {noformat}
> Aug 29 10:03:20 [INFO] 10:03:20 Generating TPC-DS qualification data, this
> need several minutes, please wait...
> chmod: cannot access '../target/generator/dsdgen_linux': No such file or
> directory
> Aug 29 10:03:20 [FAIL] Test script contains errors.
> Aug 29 10:03:20 Checking for errors...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33018:
---
Labels: auto-deprioritized-blocker pull-request-available (was:
pull-request-available stale-blocker)
Priority: Critical (was: Blocker)
This issue was labeled "stale-blocker" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually a Blocker,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Critical
> Labels: auto-deprioritized-blocker, pull-request-available
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32223: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32418) ClassNotFoundException when using flink-protobuf with sql-client
[
https://issues.apache.org/jira/browse/FLINK-32418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32418:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> ClassNotFoundException when using flink-protobuf with sql-client
>
>
> Key: FLINK-32418
> URL: https://issues.apache.org/jira/browse/FLINK-32418
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.16.2
>Reporter: Michael Kreis
>Assignee: Michael Kreis
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: full-stacktrace.log
>
>
> When the protobuf format in the kafka connector is used via the sql-client it
> is not able to load the generated protobuf classes which are either passed
> via `-j /protobuf-classes.jar` or added in the script via ADD JAR
> '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded
> but when the protobuf classes are loaded a ClassNotFoundException occurs.
> executed command:
> {code:java}
> sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
> {code}
> protobuf-table.sql
> {code:sql}
> ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
> ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
> SHOW JARS;
> CREATE TABLE POSITIONS(id BIGINT) WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'topic' = 'protbuf-topic',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'flink-protobuf',
> 'properties.security.protocol' = 'SASL_PLAINTEXT',
> 'properties.sasl.mechanism' = 'SCRAM-SHA-512',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.scram.ScramLoginModule required
> username="user" password="";',
> 'scan.startup.mode' = 'earliest-offset',
> 'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
> 'protobuf.ignore-parse-errors' = 'true'
> );
> SELECT * FROM POSITIONS;
> {code}
> exception in the log:
> {code:java}
> Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown
> Source)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
> Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Unknown Source)
> at
> org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
> ... 36 more
> {code}
> This also seems somehow related to FLINK-30318
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24241) test_table_environment_api.py fail with NPE
[
https://issues.apache.org/jira/browse/FLINK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-24241:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> test_table_environment_api.py fail with NPE
> ---
>
> Key: FLINK-24241
> URL: https://issues.apache.org/jira/browse/FLINK-24241
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Table SQL / Planner
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23876=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23263
> {code}
> Sep 10 03:03:39 E py4j.protocol.Py4JJavaError: An error
> occurred while calling o16211.execute.
> Sep 10 03:03:39 E : java.lang.NullPointerException
> Sep 10 03:03:39 E at
> java.util.Objects.requireNonNull(Objects.java:203)
> Sep 10 03:03:39 E at
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:144)
> Sep 10 03:03:39 E at
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:108)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.(FlinkRelMetadataQuery.java:78)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.instance(FlinkRelMetadataQuery.java:59)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:39)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:38)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.RelOptCluster.getMetadataQuery(RelOptCluster.java:178)
> Sep 10 03:03:39 E at
> org.apache.calcite.rel.metadata.RelMdUtil.clearCache(RelMdUtil.java:965)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.hep.HepPlanner.buildFinalPlan(HepPlanner.java:942)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.hep.HepPlanner.buildFinalPlan(HepPlanner.java:939)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:194)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:63)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:60)
> Sep 10 03:03:39 E at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> Sep 10 03:03:39 E at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> Sep 10 03:03:39 E at
> scala.collection.Iterator$class.foreach(Iterator.scala:891)
> Sep 10 03:03:39 E at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> Sep 10 03:03:39 E at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> Sep 10 03:03:39 E at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> Sep 10 03:03:39 E at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
> Sep 10 03:03:39 E at
> scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:60)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:55)
> Sep 10 03:03:39 E at
[jira] [Updated] (FLINK-13876) Remove ExecutionConfig field from PojoSerializer
[
https://issues.apache.org/jira/browse/FLINK-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-13876:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Remove ExecutionConfig field from PojoSerializer
>
>
> Key: FLINK-13876
> URL: https://issues.apache.org/jira/browse/FLINK-13876
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> The PojoSerializers stores an instance of ExecutionConfig as internal field,
> even though that the only information it may ever need are the registered
> kryo serializers.
> This has a few drawbacks:
> * It blocks the evolution of {{ExecutionConfig}} as serializers where stored
> in a state. Therefore any change to ExecutionConfig must be backwards
> compatible in respect to java serialization
> * It probably already introduced a bug, as upon restore the Snapshot actually
> recreates the serializer with an empty ExecutionConfig (see
> org.apache.flink.api.java.typeutils.runtime.PojoSerializerSnapshot#restoreSerializer)
> I suggest to remove the field completely and adjust corresponding usages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32582) Move TypeSerializerUpgradeTestBase from Kafka connector into flink-connector-common
[
https://issues.apache.org/jira/browse/FLINK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32582:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Move TypeSerializerUpgradeTestBase from Kafka connector into
> flink-connector-common
> ---
>
> Key: FLINK-32582
> URL: https://issues.apache.org/jira/browse/FLINK-32582
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Reporter: Matthias Pohl
>Priority: Major
> Labels: pull-request-available, stale-major
>
> The externalization of connectors made caused problems with the Flink's test
> data generation. The Kafka connector relied on TypeSerializerUpgradeTestBase
> for some test cases which was fine prior to FLINK-27518 where the test data
> generation was handled individually.
> With FLINK-27518 the process was automated in Flink 1.18. For now, the
> TypeSerializerUpgradeTestBase class was just copied over into the Kafka
> connector since it was the only connector that would utilize this test base.
> But we might want to provide a more generalized solution where the test base
> is provided by {{flink-connector-common}} to offer a generalized approach for
> any connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33025) BatchArrowPythonOverWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33025:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> BatchArrowPythonOverWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount
> fails on AZP
> -
>
> Key: FLINK-33025
> URL: https://issues.apache.org/jira/browse/FLINK-33025
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.3, 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=52958=logs=5cae8624-c7eb-5c51-92d3-4d2dacedd221=5acec1b4-945b-59ca-34f8-168928ce5199=22618
> fails on AZP as
> {noformat}
> Sep 03 05:05:38 05:05:38.220 [ERROR] Failures:
> Sep 03 05:05:38 05:05:38.220 [ERROR]
> BatchArrowPythonOverWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount:122->ArrowPythonAggregateFunctionOperatorTestBase.assertOutputEquals:62
>
> Sep 03 05:05:38 Expected size: 4 but was: 3 in:
> Sep 03 05:05:38 [Record @ (undef) : +I(c1,c2,0,0,0),
> Sep 03 05:05:38 Record @ (undef) : +I(c1,c4,1,0,0),
> Sep 03 05:05:38 Record @ (undef) : +I(c1,c6,2,10,2)]
> {noformat}
> probably related to FLINK-26990
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32701) Potential Memory Leak in Flink CEP due to Persistent Starting States in NFAState
[
https://issues.apache.org/jira/browse/FLINK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32701:
---
Labels: CEP auto-deprioritized-critical cep (was: CEP cep stale-critical)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Critical,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> Potential Memory Leak in Flink CEP due to Persistent Starting States in
> NFAState
>
>
> Key: FLINK-32701
> URL: https://issues.apache.org/jira/browse/FLINK-32701
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.17.0, 1.16.1, 1.16.2, 1.17.1
>Reporter: Puneet Duggal
>Priority: Major
> Labels: CEP, auto-deprioritized-critical, cep
> Attachments: Screenshot 2023-07-26 at 11.45.06 AM.png, Screenshot
> 2023-07-26 at 11.50.28 AM.png
>
>
> Our team has encountered a potential memory leak issue while working with the
> Complex Event Processing (CEP) library in Flink v1.17.
> h2. Context
> The CEP Operator maintains a keyed state called NFAState, which holds two
> queues: one for partial matches and one for completed matches. When a key is
> first encountered, the CEP creates a starting computation state and stores it
> in the partial matches queue. As more events occur that match the defined
> conditions (e.g., a TAKE condition), additional computation states get added
> to the queue, with their specific type (normal, pending, end) depending on
> the pattern sequence.
> However, I have noticed that the starting computation state remains in the
> partial matches queue even after the pattern sequence has been completely
> matched. This is also the case for keys that have already timed out. As a
> result, the state gets stored for all keys that the CEP ever encounters,
> leading to a continual increase in the checkpoint size.
> h2. How to reproduce this
> # Pattern Sequence - A not_followed_by B within 5 mins
> # Time Characteristic - EventTime
> # StateBackend - HashMapStateBackend
> On my local machine, I started this pipeline and started sending events at
> the rate of 10 events per second (only A) and as expected after 5 mins, CEP
> started sending pattern matched output with the same rate. But the issue was
> that after every 2 mins (checkpoint interval), checkpoint size kept on
> increasing. Expectation was that after 5 mins (2-3 checkpoints), checkpoint
> size will remain constant since any window of 5 mins will consist of the same
> number of unique keys (older ones will get matched or timed out hence removed
> from state). But as you can see below attached images, checkpoint size kept
> on increasing till 40 checkpoints (around 1.5hrs).
> P.S. - After 3 checkpoints (6 mins), the checkpoint size was around 1.78MB.
> Hence assumption is that ideal checkpoint size for a 5 min window should be
> less than 1.78MB.
> As you can see after 39 checkpoints, I triggered a savepoint for this
> pipeline. After that I used a savepoint reader to investigate what all is
> getting stored in CEP states. Below code investigates NFAState of CEPOperator
> for potential memory leak.
> {code:java}
> import lombok.AllArgsConstructor;
> import lombok.Data;
> import lombok.NoArgsConstructor;
> import org.apache.flink.api.common.state.ValueState;
> import org.apache.flink.api.common.state.ValueStateDescriptor;
> import org.apache.flink.cep.nfa.NFAState;
> import org.apache.flink.cep.nfa.NFAStateSerializer;
> import org.apache.flink.configuration.Configuration;
> import org.apache.flink.runtime.state.filesystem.FsStateBackend;
> import org.apache.flink.state.api.OperatorIdentifier;
> import org.apache.flink.state.api.SavepointReader;
> import org.apache.flink.state.api.functions.KeyedStateReaderFunction;
> import org.apache.flink.streaming.api.datastream.DataStream;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.util.Collector;
> import org.junit.jupiter.api.Test;
> import java.io.Serializable;
> import java.util.Objects;
> public class NFAStateReaderTest {
> private static final String NFA_STATE_NAME = "nfaStateName";
> @Test
> public void testNfaStateReader() throws Exception {
> StreamExecutionEnvironment environment =
> StreamExecutionEnvironment.getExecutionEnvironment();
> SavepointReader savepointReader =
> SavepointReader.read(environment,
> "file:///opt/flink/savepoints/savepoint-093404-9bc0a38654df", new
> FsStateBackend("file:///abc"));
> DataStream stream =
>
[jira] [Updated] (FLINK-32512) SHOW JARS should not show the jars for temporary function
[ https://issues.apache.org/jira/browse/FLINK-32512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32512: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > SHOW JARS should not show the jars for temporary function > - > > Key: FLINK-32512 > URL: https://issues.apache.org/jira/browse/FLINK-32512 > Project: Flink > Issue Type: Bug > Components: Table SQL / Gateway >Affects Versions: 1.19.0 >Reporter: Fang Yong >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > According to > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/show/#show-jars, > `SHOW JARS` should only list the jars added by `ADD JAR` statement, but > currently it also show the jars for `CREATE TEMPORARY FUNCTION` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-21648) FLIP-151: Incremental snapshots for heap-based state backend
[ https://issues.apache.org/jira/browse/FLINK-21648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-21648: --- Labels: auto-deprioritized-major auto-unassigned stale-assigned (was: auto-deprioritized-major auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > FLIP-151: Incremental snapshots for heap-based state backend > > > Key: FLINK-21648 > URL: https://issues.apache.org/jira/browse/FLINK-21648 > Project: Flink > Issue Type: New Feature > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: auto-deprioritized-major, auto-unassigned, stale-assigned > Fix For: 1.18.0 > > > Umbrella ticket for > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32658) State should not be silently removed when ignore-unclaimed-state is false
[ https://issues.apache.org/jira/browse/FLINK-32658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32658: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > State should not be silently removed when ignore-unclaimed-state is false > - > > Key: FLINK-32658 > URL: https://issues.apache.org/jira/browse/FLINK-32658 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.18.0, 1.17.1 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: screenshot-1.png > > > When ignore-unclaimed-state is false and the old state is removed, flink > should throw exception. It's similar to removing a stateful operator. > This case occurs not only when the user removes state, but also when the > operator is replaced. > For example: upgrade FlinkKafkaConsumer to KafkaSource. All logical are not > changed, so the operator id isn't changed. The KafkaSource cannot resume from > the state of FlinkKafkaConsumer. However, the new flink job can start, and > the state is silently removed in the new job.(The old state is not physically > discarded, it is still stored in the state backend, but the new code will > never use it.) > It also brings an additional problem: the KafkaSource will snapshot 2 states, > it includes the new state of KafkaSource, and the union list state of > FlinkKafkaConsumer. Whenever a job resumes from checkpoint, the union List > state is inflated. Eventually the state size of kafka offset exceeded 200MB. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30699) Improved getRandomString method code format in the StringUtils class
[ https://issues.apache.org/jira/browse/FLINK-30699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-30699: --- Labels: auto-deprioritized-minor pull-request-available (was: pull-request-available stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Improved getRandomString method code format in the StringUtils class > > > Key: FLINK-30699 > URL: https://issues.apache.org/jira/browse/FLINK-30699 > Project: Flink > Issue Type: Improvement > Components: API / Core >Reporter: Bingye Chen >Priority: Not a Priority > Labels: auto-deprioritized-minor, pull-request-available > Attachments: image-2023-01-16-18-13-56-912.png, > image-2023-01-16-18-14-12-939.png > > > > !image-2023-01-16-18-13-56-912.png|width=398,height=148! > !image-2023-01-16-18-14-12-939.png|width=398,height=114! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32575) Unified the Cpu of JobManager Name
[ https://issues.apache.org/jira/browse/FLINK-32575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32575: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Unified the Cpu of JobManager Name > -- > > Key: FLINK-32575 > URL: https://issues.apache.org/jira/browse/FLINK-32575 > Project: Flink > Issue Type: New Feature >Reporter: Bo Cui >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > To set the jm CPU of the Yarn, use the `yarn.appmaster.vcores`. To set the jm > cpu of the k8s, use `kubernetes.jobmanager.cpu`. when there are yarn and k8s > clusters, managing these configurations is difficult. Add a unified name for > them for ease of use. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32806) EmbeddedJobResultStore keeps the non-dirty job entries forever
[
https://issues.apache.org/jira/browse/FLINK-32806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32806:
---
Labels: stale-assigned starter (was: starter)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> EmbeddedJobResultStore keeps the non-dirty job entries forever
> --
>
> Key: FLINK-32806
> URL: https://issues.apache.org/jira/browse/FLINK-32806
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.18.0, 1.17.1, 1.19.0
>Reporter: Matthias Pohl
>Assignee: hk__lrzy
>Priority: Major
> Labels: stale-assigned, starter
>
> The {{EmbeddedJobResultStore}} keeps the entries of cleaned-up jobs in-memory
> forever. We might want to add a TTL to have those entries be removed after a
> certain amount of time to allow maintaining the memory footprint of the
> {{EmbeddedJobResultStore}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30519) Add e2e tests for operator dynamic config
[ https://issues.apache.org/jira/browse/FLINK-30519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-30519: --- Labels: stale-assigned stale-critical starter (was: stale-assigned starter) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Critical but is unassigned and neither itself nor its Sub-Tasks have been updated for 14 days. I have gone ahead and marked it "stale-critical". If this ticket is critical, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add e2e tests for operator dynamic config > - > > Key: FLINK-30519 > URL: https://issues.apache.org/jira/browse/FLINK-30519 > Project: Flink > Issue Type: Technical Debt > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Critical > Labels: stale-assigned, stale-critical, starter > > The dynamic config feature is currently not covered by e2e tests and is > subject to accidental regressions, as shown in: > https://issues.apache.org/jira/browse/FLINK-30329 > We should add an e2e test that covers this -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32522) Kafka connector should depend on commons-collections instead of inherit from flink
[ https://issues.apache.org/jira/browse/FLINK-32522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32522: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Kafka connector should depend on commons-collections instead of inherit from > flink > -- > > Key: FLINK-32522 > URL: https://issues.apache.org/jira/browse/FLINK-32522 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.17.1 >Reporter: Ran Tao >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Attachments: image-2023-07-03-20-15-47-608.png, > image-2023-07-03-20-16-03-031.png > > > Currently, externalized sql connector rely on flink main repo. but flink main > repo has many test cases(especially in flink-python) reference > flink-sql-kafka-connector. > If we change the dependencies(e.g. commons-collections) in flink main repo, > it cause exception: > !image-2023-07-03-20-15-47-608.png! > > !https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png! > > So must add this dependency explicitly. Otherwise, it will cause external > connectors block the upgrade of flink main. Connectors shouldn't rely on > dependencies that may or may not be > available in Flink itself. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32605) JoinITCase.testFullOuterJoinWithMultipleKeys fails with TimeoutException: Futures timed out after [20 seconds]
[
https://issues.apache.org/jira/browse/FLINK-32605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32605:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> JoinITCase.testFullOuterJoinWithMultipleKeys fails with TimeoutException:
> Futures timed out after [20 seconds]
> --
>
> Key: FLINK-32605
> URL: https://issues.apache.org/jira/browse/FLINK-32605
> Project: Flink
> Issue Type: Bug
> Components: Runtime / RPC, Table SQL / API
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: stale-major, test-stability
>
> While execution of
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51254=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11681
> there was an exception
> {noformat}
> Jul 14 04:35:32 Caused by: java.lang.Exception: Could not create actor system
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaBootstrapTools.startLocalActorSystem(AkkaBootstrapTools.java:238)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:349)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:327)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:247)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.minicluster.MiniCluster.createLocalRpcService(MiniCluster.java:1188)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:355)
> Jul 14 04:35:32 at
> org.apache.flink.client.program.PerJobMiniClusterFactory.submitJob(PerJobMiniClusterFactory.java:77)
> Jul 14 04:35:32 at
> org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:85)
> Jul 14 04:35:32 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2218)
> Jul 14 04:35:32 at
> org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110)
> Jul 14 04:35:32 at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeQueryOperation(TableEnvironmentImpl.java:992)
> Jul 14 04:35:32 ... 102 more
> Jul 14 04:35:32 Caused by: java.util.concurrent.TimeoutException: Futures
> timed out after [20 seconds]
> Jul 14 04:35:32 at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
> Jul 14 04:35:32 at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
> Jul 14 04:35:32 at
> scala.concurrent.Await$.$anonfun$result$1(package.scala:223)
> Jul 14 04:35:32 at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
> Jul 14 04:35:32 at scala.concurrent.Await$.result(package.scala:146)
> Jul 14 04:35:32 at
> akka.stream.SystemMaterializer.(SystemMaterializer.scala:90)
> Jul 14 04:35:32 at
> akka.stream.SystemMaterializer$.createExtension(SystemMaterializer.scala:39)
> Jul 14 04:35:32 at
> akka.stream.SystemMaterializer$.createExtension(SystemMaterializer.scala:32)
> Jul 14 04:35:32 at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1165)
> Jul 14 04:35:32 at
> akka.actor.ActorSystemImpl.$anonfun$loadExtensions$1(ActorSystem.scala:1208)
> Jul 14 04:35:32 at scala.collection.Iterator.foreach(Iterator.scala:943)
> Jul 14 04:35:32 at
> scala.collection.Iterator.foreach$(Iterator.scala:943)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.RobustActorSystem.create(RobustActorSystem.java:54)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaUtils.createActorSystem(AkkaUtils.java:421)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaBootstrapTools.startActorSystem(AkkaBootstrapTools.java:253)
> Jul 14 04:35:32 at
> org.apache.flink.runtime.rpc.akka.AkkaBootstrapTools.startLocalActorSystem(AkkaBootstrapTools.java:236)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28475) kafka connector won't stop when the stopping offset is zero
[ https://issues.apache.org/jira/browse/FLINK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-28475: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > kafka connector won't stop when the stopping offset is zero > --- > > Key: FLINK-28475 > URL: https://issues.apache.org/jira/browse/FLINK-28475 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.15.0 >Reporter: Leo zhang >Assignee: Leo zhang >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.18.0 > > > when use kafka connector in bounded mode,and the stopping offset hapends to > be 0,the kafka connector won't stop,which is not expected. > I had traced the code, and found the stopping offset will be set to empty > when it is zero, and an empty stopping offset means no stopping offset when > serialized. This leads to a wrong execution. > I had fixed this in my personal branch,now I am logging this issue in Jira so > that I can make merge request. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29050) [JUnit5 Migration] Module: flink-hadoop-compatibility
[ https://issues.apache.org/jira/browse/FLINK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-29050: --- Labels: pull-request-available stale-assigned starter (was: pull-request-available starter) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > [JUnit5 Migration] Module: flink-hadoop-compatibility > - > > Key: FLINK-29050 > URL: https://issues.apache.org/jira/browse/FLINK-29050 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hadoop Compatibility, Tests >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available, stale-assigned, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31390) Optimize the FlinkChangelogModeInferenceProgram by avoiding unnecessary traversals.
[
https://issues.apache.org/jira/browse/FLINK-31390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31390:
---
Labels: auto-deprioritized-minor pull-request-available (was:
pull-request-available stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Optimize the FlinkChangelogModeInferenceProgram by avoiding unnecessary
> traversals.
> ---
>
> Key: FLINK-31390
> URL: https://issues.apache.org/jira/browse/FLINK-31390
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: Aitozi
>Priority: Not a Priority
> Labels: auto-deprioritized-minor, pull-request-available
>
> We can avoid the unnecessary traversals of the RelNode tree, since we are
> only interested in the first satisfied plan.
>
> FlinkChangelogModeInferenceProgram
> {code:java}
> val updateKindTraitVisitor = new SatisfyUpdateKindTraitVisitor(context)
> val finalRoot = requiredUpdateKindTraits.flatMap {
> requiredUpdateKindTrait =>
> updateKindTraitVisitor.visit(rootWithModifyKindSet,
> requiredUpdateKindTrait)
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32941) Table API Bridge `toDataStream(targetDataType)` function not working correctly for Java List
[
https://issues.apache.org/jira/browse/FLINK-32941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32941:
---
Labels: auto-deprioritized-critical bridge (was: bridge stale-critical)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Critical,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> Table API Bridge `toDataStream(targetDataType)` function not working
> correctly for Java List
>
>
> Key: FLINK-32941
> URL: https://issues.apache.org/jira/browse/FLINK-32941
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Tan Kim
>Priority: Major
> Labels: auto-deprioritized-critical, bridge
>
> When the code below is executed, only the first element of the list is
> assigned to the List variable in MyPoJo repeatedly.
> {code:java}
> case class Item(
> name: String
> )
> case class MyPojo(
> @DataTypeHist("RAW") items: java.util.List[Item]
> )
> ...
> tableEnv
> .sqlQuery("select items from table")
> .toDataStream(DataTypes.of(classOf[MyPoJo])) {code}
>
> For example, if you have the following list coming in as input,
> ["a","b","c"]
> The value actually stored in MyPojo's list variable is
> ["a","a","a"]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32862) Support INIT operation type to be compatible with DTS on Alibaba Cloud
[ https://issues.apache.org/jira/browse/FLINK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32862: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Support INIT operation type to be compatible with DTS on Alibaba Cloud > -- > > Key: FLINK-32862 > URL: https://issues.apache.org/jira/browse/FLINK-32862 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Hang Ruan >Assignee: Hang Ruan >Priority: Minor > Labels: pull-request-available, stale-assigned > > The operation type of canal json messages from DTS on Alibaba Cloud may > contain a new type `INIT`. We cannot handle these messages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32995) TPC-DS end-to-end test fails with chmod: cannot access '../target/generator/dsdgen_linux':
[
https://issues.apache.org/jira/browse/FLINK-32995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32995:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> TPC-DS end-to-end test fails with chmod: cannot access
> '../target/generator/dsdgen_linux':
> ---
>
> Key: FLINK-32995
> URL: https://issues.apache.org/jira/browse/FLINK-32995
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=52773=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=0f3adb59-eefa-51c6-2858-3654d9e0749d=5504
> fails as
> {noformat}
> Aug 29 10:03:20 [INFO] 10:03:20 Generating TPC-DS qualification data, this
> need several minutes, please wait...
> chmod: cannot access '../target/generator/dsdgen_linux': No such file or
> directory
> Aug 29 10:03:20 [FAIL] Test script contains errors.
> Aug 29 10:03:20 Checking for errors...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32753) Print JVM flags on AZP
[ https://issues.apache.org/jira/browse/FLINK-32753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32753: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Print JVM flags on AZP > -- > > Key: FLINK-32753 > URL: https://issues.apache.org/jira/browse/FLINK-32753 > Project: Flink > Issue Type: Improvement > Components: Build System / Azure Pipelines >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Minor > Labels: pull-request-available, stale-assigned > Fix For: 1.18.0 > > > I suggest printing JVM flags before the tests run, which could help > investigate the test failures (especially memory or GC related issue). An > example of pipeline output > [here|https://dev.azure.com/lzq82555906/flink-for-Zakelly/_build/results?buildId=122=logs=9dc1b5dc-bcfa-5f83-eaa7-0cb181ddc267=511d2595-ec54-5ab7-86ce-92f328796f20=165]. > You may search 'JVM information' in this log. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33018:
---
Labels: pull-request-available stale-blocker (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as a
Blocker but is unassigned and neither itself nor its Sub-Tasks have been
updated for 1 days. I have gone ahead and marked it "stale-blocker". If this
ticket is a Blocker, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
> Labels: pull-request-available, stale-blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30774) Introduce flink-utils module
[
https://issues.apache.org/jira/browse/FLINK-30774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-30774:
---
Labels: auto-deprioritized-major starter (was: stale-major starter)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Introduce flink-utils module
>
>
> Key: FLINK-30774
> URL: https://issues.apache.org/jira/browse/FLINK-30774
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Minor
> Labels: auto-deprioritized-major, starter
>
> Currently, utility methods generic utility classes like {{Preconditions}} or
> {{AbstractAutoCloseableRegistry}} are collected in {{flink-core}}. The flaw
> of this approach is that we cannot use those classes in modules like
> {{fink-migration-test-utils}}, {{flink-test-utils-junit}},
> {{flink-metrics-core}} or {{flink-annotations}}.
> We might want to have a generic {{flink-utils}} analogously to
> {{flink-test-utils}} that collects Flink-independent utility functionality
> that can be access by any module {{flink-core}} is depending on to make this
> utility functionality available in any Flink-related module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32564) Support cast from BYTES to NUMBER
[
https://issues.apache.org/jira/browse/FLINK-32564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32564:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Support cast from BYTES to NUMBER
> -
>
> Key: FLINK-32564
> URL: https://issues.apache.org/jira/browse/FLINK-32564
> Project: Flink
> Issue Type: Sub-task
>Reporter: Hanyu Zheng
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> We are dealing with a task that requires casting from the BYTES type to
> BIGINT. Specifically, we have a string '00T1p'. Our approach is to convert
> this string to BYTES and then cast the result to BIGINT with the following
> SQL query:
> {code:java}
> SELECT CAST((CAST('00T1p' as BYTES)) as BIGINT);{code}
> However, an issue arises when executing this query, likely due to an error in
> the conversion between BYTES and BIGINT. We aim to identify and rectify this
> issue so our query can run correctly. The tasks involved are:
> # Investigate and identify the specific reason for the failure of conversion
> from BYTES to BIGINT.
> # Design and implement a solution to ensure our query can function correctly.
> # Test this solution across all required scenarios to guarantee its
> functionality.
>
> see also
> 1. PostgreSQL: PostgreSQL supports casting from BYTES type (BYTEA) to NUMBER
> types (INTEGER, BIGINT, DECIMAL, etc.). In PostgreSQL, you can use CAST or
> type conversion operator （：：） for performing the conversion. URL:
> [https://www.postgresql.org/docs/current/sql-expressions.html#SQL-SYNTAX-TYPE-CASTS]
> 2. MySQL: MySQL supports casting from BYTES type (BLOB or BINARY) to NUMBER
> types (INTEGER, BIGINT, DECIMAL, etc.). In MySQL, you can use CAST or CONVERT
> functions for performing the conversion. URL:
> [https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html]
> 3. Microsoft SQL Server: SQL Server supports casting from BYTES type
> (VARBINARY, IMAGE) to NUMBER types (INT, BIGINT, NUMERIC, etc.). You can use
> CAST or CONVERT functions for performing the conversion. URL:
> [https://docs.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql]
> 4. Oracle Database: Oracle supports casting from RAW type (equivalent to
> BYTES) to NUMBER types (NUMBER, INTEGER, FLOAT, etc.). You can use the
> TO_NUMBER function for performing the conversion. URL:
> [https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/TO_NUMBER.html]
> 5. Apache Spark: Spark DataFrame supports casting binary types (BinaryType or
> ByteType) to numeric types (IntegerType, LongType, DecimalType, etc.) by
> using the {{cast}} function. URL:
> [https://spark.apache.org/docs/latest/api/sql/#cast]
>
> for the problem of bytes order may arise (little vs big endian).
>
> 1. Apache Hadoop: Hadoop, being an open-source framework, has to deal with
> byte order issues across different platforms and architectures. The Hadoop
> File System (HDFS) uses a technique called "sequence files," which include
> metadata to describe the byte order of the data. This metadata ensures that
> data is read and written correctly, regardless of the endianness of the
> platform.
> 2. Apache Avro: Avro is a data serialization system used by various big data
> frameworks like Hadoop and Apache Kafka. Avro uses a compact binary encoding
> format that includes a marker for the byte order. This allows Avro to handle
> endianness issues seamlessly when data is exchanged between systems with
> different byte orders.
> 3. Apache Parquet: Parquet is a columnar storage format used in big data
> processing frameworks like Apache Spark. Parquet uses a little-endian format
> for encoding numeric values, which is the most common format on modern
> systems. When reading or writing Parquet data, data processing engines
> typically handle any necessary byte order conversions transparently.
> 4. Apache Spark: Spark is a popular big data processing engine that can
> handle data on distributed systems. It relies on the underlying data formats
> it reads (e.g., Avro, Parquet, ORC) to manage byte order issues.
[jira] [Updated] (FLINK-31689) Filesystem sink fails when parallelism of compactor operator changed
[
https://issues.apache.org/jira/browse/FLINK-31689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31689:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Filesystem sink fails when parallelism of compactor operator changed
>
>
> Key: FLINK-31689
> URL: https://issues.apache.org/jira/browse/FLINK-31689
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.16.1
>Reporter: jirawech.s
>Assignee: jirawech.s
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: HelloFlinkHadoopSink.java
>
>
> I encounter this error when i tried to use Filesystem sink with Table SQL. I
> have not tested with Datastream API tho. You may refers to the error as below
> {code:java}
> // code placeholder
> java.util.NoSuchElementException
> at java.util.ArrayList$Itr.next(ArrayList.java:864)
> at
> org.apache.flink.connector.file.table.stream.compact.CompactOperator.initializeState(CompactOperator.java:119)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:286)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:700)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:676)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:643)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:917)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
> at java.lang.Thread.run(Thread.java:750) {code}
> I cannot attach the full reproducible code here, but you may follow my pseudo
> code in attachment and reproducible steps below
> 1. Create Kafka source
> 2. Set state.savepoints.dir
> 3. Set Job parallelism to 1
> 4. Create FileSystem Sink
> 5. Run the job and trigger savepoint with API
> {noformat}
> curl -X POST localhost:8081/jobs/:jobId/savepoints -d '{"cancel-job":
> false}'{noformat}
> {color:#172b4d}6. Cancel job, change parallelism to 2, and resume job from
> savepoint{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24241) test_table_environment_api.py fail with NPE
[
https://issues.apache.org/jira/browse/FLINK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-24241:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> test_table_environment_api.py fail with NPE
> ---
>
> Key: FLINK-24241
> URL: https://issues.apache.org/jira/browse/FLINK-24241
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Table SQL / Planner
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23876=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23263
> {code}
> Sep 10 03:03:39 E py4j.protocol.Py4JJavaError: An error
> occurred while calling o16211.execute.
> Sep 10 03:03:39 E : java.lang.NullPointerException
> Sep 10 03:03:39 E at
> java.util.Objects.requireNonNull(Objects.java:203)
> Sep 10 03:03:39 E at
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:144)
> Sep 10 03:03:39 E at
> org.apache.calcite.rel.metadata.RelMetadataQuery.(RelMetadataQuery.java:108)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.(FlinkRelMetadataQuery.java:78)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.instance(FlinkRelMetadataQuery.java:59)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:39)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.calcite.FlinkRelOptClusterFactory$$anon$1.get(FlinkRelOptClusterFactory.scala:38)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.RelOptCluster.getMetadataQuery(RelOptCluster.java:178)
> Sep 10 03:03:39 E at
> org.apache.calcite.rel.metadata.RelMdUtil.clearCache(RelMdUtil.java:965)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.hep.HepPlanner.buildFinalPlan(HepPlanner.java:942)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.hep.HepPlanner.buildFinalPlan(HepPlanner.java:939)
> Sep 10 03:03:39 E at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:194)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:63)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:60)
> Sep 10 03:03:39 E at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> Sep 10 03:03:39 E at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> Sep 10 03:03:39 E at
> scala.collection.Iterator$class.foreach(Iterator.scala:891)
> Sep 10 03:03:39 E at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> Sep 10 03:03:39 E at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> Sep 10 03:03:39 E at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> Sep 10 03:03:39 E at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
> Sep 10 03:03:39 E at
> scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> Sep 10 03:03:39 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:60)
> Sep 10 03:03:39 E at
>
[jira] [Updated] (FLINK-31857) Support pluginable observers mechanism
[ https://issues.apache.org/jira/browse/FLINK-31857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31857: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support pluginable observers mechanism > -- > > Key: FLINK-31857 > URL: https://issues.apache.org/jira/browse/FLINK-31857 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.5.0 >Reporter: Daren Wong >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > Currently, Kubernetes Operator uses AbstractFlinkResourceObserver to observe > Flink Job state, cluster state, JM/TM state, etc and update Custom Resource: > FlinkDeployment, FlinkSessionJob with the gathered info. > This Jira is to introduce a Flink plugin to allow user to implement custom > observer logic after the default observer logic is run. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32525) Update commons-beanutils to 1.9.4
[ https://issues.apache.org/jira/browse/FLINK-32525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32525: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Update commons-beanutils to 1.9.4 > - > > Key: FLINK-32525 > URL: https://issues.apache.org/jira/browse/FLINK-32525 > Project: Flink > Issue Type: Technical Debt > Components: Deployment / YARN >Reporter: Martijn Visser >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > YARN still tests with commons-beanutils 1.8.3 with a remark that beanutil > 1.9+ doesn't work with Hadoop, but Hadoop 2.10.2 (which is our minimum > supported version) uses beanutils 1.9.4 itself, per > https://github.com/apache/hadoop/blob/rel/release-2.10.2/hadoop-project/pom.xml#L861-L863 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32044) Improve catalog name check to keep consistent about human-readable exception log in FunctionCatalog
[
https://issues.apache.org/jira/browse/FLINK-32044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32044:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Improve catalog name check to keep consistent about human-readable exception
> log in FunctionCatalog
>
>
> Key: FLINK-32044
> URL: https://issues.apache.org/jira/browse/FLINK-32044
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.17.0
>Reporter: Ran Tao
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> {code:java}
> Catalog catalog = catalogManager.getCatalog(catalogName).get(); {code}
>
> We can do an improvement to check optional#get and throw more friendly log to
> users like other list operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-26115) Multiple Kafka connector tests failed due to The topic metadata failed to propagate to Kafka broker
[ https://issues.apache.org/jira/browse/FLINK-26115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-26115: --- Labels: auto-deprioritized-critical auto-deprioritized-major auto-deprioritized-minor test-stability (was: auto-deprioritized-critical auto-deprioritized-major stale-minor test-stability) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Multiple Kafka connector tests failed due to The topic metadata failed to > propagate to Kafka broker > --- > > Key: FLINK-26115 > URL: https://issues.apache.org/jira/browse/FLINK-26115 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.13.5, 1.14.3, 1.15.0 >Reporter: Yun Gao >Priority: Not a Priority > Labels: auto-deprioritized-critical, auto-deprioritized-major, > auto-deprioritized-minor, test-stability > > This issues tracks all the related issues with "The topic metadata failed to > propagate to Kafka broker" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28881) PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream test failure
[
https://issues.apache.org/jira/browse/FLINK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-28881:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> test failure
> ---
>
> Key: FLINK-28881
> URL: https://issues.apache.org/jira/browse/FLINK-28881
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: 1.16.0, gcp-pubsub-3.0.0
>Reporter: Leonard Xu
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> 2022-08-09T03:14:22.0113691Z Aug 09 03:14:22 [ERROR]
> org.apache.flink.streaming.connectors.gcp.pubsub.PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> Time elapsed: 1.867 s <<< FAILURE!
> 2022-08-09T03:14:22.0114504Z Aug 09 03:14:22 java.lang.AssertionError:
> 2022-08-09T03:14:22.0114903Z Aug 09 03:14:22
> 2022-08-09T03:14:22.0115263Z Aug 09 03:14:22 Expected: <[1, 2, 3]>
> 2022-08-09T03:14:22.0115679Z Aug 09 03:14:22 but: was <[1, 2]>
> 2022-08-09T03:14:22.0116232Z Aug 09 03:14:22 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 2022-08-09T03:14:22.0116871Z Aug 09 03:14:22 at
> org.junit.Assert.assertThat(Assert.java:964)
> 2022-08-09T03:14:22.0117580Z Aug 09 03:14:22 at
> org.junit.Assert.assertThat(Assert.java:930)
> 2022-08-09T03:14:22.0118460Z Aug 09 03:14:22 at
> org.apache.flink.streaming.connectors.gcp.pubsub.PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream(PubSubConsumingTest.java:119)
> {code}
> CI link:
> https://dev.azure.com/leonardBang/Azure_CI/_build/results?buildId=713=logs=3796201e-ea88-5776-0ea8-9ccca648a70c=8ca54b76-085e-5cf1-8060-2c500a258258
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30935) Add KafkaSerializer deserialize check when using SimpleVersionedSerializer
[
https://issues.apache.org/jira/browse/FLINK-30935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-30935:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Add KafkaSerializer deserialize check when using SimpleVersionedSerializer
> --
>
> Key: FLINK-30935
> URL: https://issues.apache.org/jira/browse/FLINK-30935
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Reporter: Ran Tao
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> Current kafka many implemented serializers do not deal with version check
> while other implementations of SimpleVersionedSerializer supports it.
> we can add it like many other connectors's implementation in case of
> incompatible or corrupt state when restoring from checkpoint.
>
> {code:java}
> @Override
> public int getVersion() {
> return CURRENT_VERSION;
> }
> @Override
> public KafkaPartitionSplit deserialize(int version, byte[] serialized) throws
> IOException {
> try (ByteArrayInputStream bais = new ByteArrayInputStream(serialized);
> DataInputStream in = new DataInputStream(bais)) {
> String topic = in.readUTF();
> int partition = in.readInt();
> long offset = in.readLong();
> long stoppingOffset = in.readLong();
> return new KafkaPartitionSplit(
> new TopicPartition(topic, partition), offset, stoppingOffset);
> }
> } {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32344) MongoDB connector support unbounded streaming read via ChangeStream feature
[ https://issues.apache.org/jira/browse/FLINK-32344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32344: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > MongoDB connector support unbounded streaming read via ChangeStream feature > --- > > Key: FLINK-32344 > URL: https://issues.apache.org/jira/browse/FLINK-32344 > Project: Flink > Issue Type: New Feature > Components: Connectors / MongoDB >Affects Versions: mongodb-1.0.1 >Reporter: Jiabao Sun >Assignee: Jiabao Sun >Priority: Major > Labels: pull-request-available, stale-assigned > > Change streams allow applications to access real-time data changes without > the complexity and risk of tailing the oplog. Applications can use change > streams to subscribe to all data changes on a single collection, a database, > or an entire deployment, and immediately react to them. Because change > streams use the aggregation framework, applications can also filter for > specific changes or transform the notifications at will. > We can use MongoDB change streams feature to support unbounded streaming read > for mongodb connector. > [Change > Streams|https://www.mongodb.com/docs/manual/changeStreams/#change-streams] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32512) SHOW JARS should not show the jars for temporary function
[ https://issues.apache.org/jira/browse/FLINK-32512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32512: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > SHOW JARS should not show the jars for temporary function > - > > Key: FLINK-32512 > URL: https://issues.apache.org/jira/browse/FLINK-32512 > Project: Flink > Issue Type: Bug > Components: Table SQL / Gateway >Affects Versions: 1.19.0 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available, stale-major > > According to > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/show/#show-jars, > `SHOW JARS` should only list the jars added by `ADD JAR` statement, but > currently it also show the jars for `CREATE TEMPORARY FUNCTION` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32575) Unified the Cpu of JobManager Name
[ https://issues.apache.org/jira/browse/FLINK-32575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32575: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Unified the Cpu of JobManager Name > -- > > Key: FLINK-32575 > URL: https://issues.apache.org/jira/browse/FLINK-32575 > Project: Flink > Issue Type: New Feature >Reporter: Bo Cui >Priority: Major > Labels: pull-request-available, stale-major > > To set the jm CPU of the Yarn, use the `yarn.appmaster.vcores`. To set the jm > cpu of the k8s, use `kubernetes.jobmanager.cpu`. when there are yarn and k8s > clusters, managing these configurations is difficult. Add a unified name for > them for ease of use. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30699) Improved getRandomString method code format in the StringUtils class
[ https://issues.apache.org/jira/browse/FLINK-30699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-30699: --- Labels: pull-request-available stale-minor (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Improved getRandomString method code format in the StringUtils class > > > Key: FLINK-30699 > URL: https://issues.apache.org/jira/browse/FLINK-30699 > Project: Flink > Issue Type: Improvement > Components: API / Core >Reporter: Bingye Chen >Priority: Minor > Labels: pull-request-available, stale-minor > Attachments: image-2023-01-16-18-13-56-912.png, > image-2023-01-16-18-14-12-939.png > > > > !image-2023-01-16-18-13-56-912.png|width=398,height=148! > !image-2023-01-16-18-14-12-939.png|width=398,height=114! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25447) batch query cannot generate plan when a sorted view into multi sinks
[
https://issues.apache.org/jira/browse/FLINK-25447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25447:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> batch query cannot generate plan when a sorted view into multi sinks
>
>
> Key: FLINK-25447
> URL: https://issues.apache.org/jira/browse/FLINK-25447
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.14.2
>Reporter: lincoln lee
>Assignee: Zheng yunhong
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.18.0, 1.17.2, 1.19.0
>
>
> A batch query write a sorted view into multi sinks will get a cannot plan
> exception
> {code}
> @Test
> def testSortedResultIntoMultiSinks(): Unit = {
> util.tableEnv.executeSql(
> s"""
> |CREATE TABLE Src (
> | `a` INT,
> | `b` BIGINT,
> | `c` STRING,
> | `d` STRING,
> | `e` STRING
> |) WITH (
> | 'connector' = 'values',
> | 'bounded' = 'true'
> |)
>""".stripMargin)
> val query = "SELECT * FROM Src order by c"
> val table = util.tableEnv.sqlQuery(query)
> util.tableEnv.registerTable("sortedTable", table)
> util.tableEnv.executeSql(
> s"""
> |CREATE TABLE sink1 (
> | `a` INT,
> | `b` BIGINT,
> | `c` STRING
> |) WITH (
> | 'connector' = 'filesystem',
> | 'format' = 'testcsv',
> | 'path' = '/tmp/test'
> |)
>""".stripMargin)
> util.tableEnv.executeSql(
> s"""
> |CREATE TABLE sink2 (
> | `a` INT,
> | `b` BIGINT,
> | `c` STRING,
> | `d` STRING
> |) WITH (
> | 'connector' = 'filesystem',
> | 'format' = 'testcsv',
> | 'path' = '/tmp/test'
> |)
> """.stripMargin)
> val stmtSet= util.tableEnv.createStatementSet()
> stmtSet.addInsertSql(
> "insert into sink1 select a, b, listagg(d) from sortedTable group by a,
> b")
> stmtSet.addInsertSql(
> "insert into sink2 select a, b, c, d from sortedTable")
> util.verifyExecPlan(stmtSet)
> }
> {code}
> {code}
> org.apache.flink.table.api.TableException: Cannot generate a valid
> execution plan for the given query:
> LogicalSink(table=[default_catalog.default_database.sink2], fields=[a, b,
> c, d])
> +- LogicalProject(inputs=[0..3])
> +- LogicalTableScan(table=[[IntermediateRelTable_0]])
> This exception indicates that the query uses an unsupported SQL feature.
> Please check the documentation for the set of currently supported SQL
> features.
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:76)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:62)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
> at scala.collection.Iterator.foreach(Iterator.scala:937)
> at scala.collection.Iterator.foreach$(Iterator.scala:937)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
> at scala.collection.IterableLike.foreach(IterableLike.scala:70)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
> at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:58)
> at
>
[jira] [Updated] (FLINK-32522) Kafka connector should depend on commons-collections instead of inherit from flink
[ https://issues.apache.org/jira/browse/FLINK-32522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32522: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Kafka connector should depend on commons-collections instead of inherit from > flink > -- > > Key: FLINK-32522 > URL: https://issues.apache.org/jira/browse/FLINK-32522 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.17.1 >Reporter: Ran Tao >Priority: Major > Labels: pull-request-available, stale-major > Attachments: image-2023-07-03-20-15-47-608.png, > image-2023-07-03-20-16-03-031.png > > > Currently, externalized sql connector rely on flink main repo. but flink main > repo has many test cases(especially in flink-python) reference > flink-sql-kafka-connector. > If we change the dependencies(e.g. commons-collections) in flink main repo, > it cause exception: > !image-2023-07-03-20-15-47-608.png! > > !https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png! > > So must add this dependency explicitly. Otherwise, it will cause external > connectors block the upgrade of flink main. Connectors shouldn't rely on > dependencies that may or may not be > available in Flink itself. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32396) Support timestamp for jdbc driver and gateway
[ https://issues.apache.org/jira/browse/FLINK-32396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32396: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support timestamp for jdbc driver and gateway > - > > Key: FLINK-32396 > URL: https://issues.apache.org/jira/browse/FLINK-32396 > Project: Flink > Issue Type: Improvement > Components: Table SQL / JDBC >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: Fang Yong >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > Support timestamp and timestamp_ltz data type for jdbc driver and sql-gateway -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32706) Add SPLIT(STRING) support in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-32706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32706:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Add SPLIT(STRING) support in SQL & Table API
>
>
> Key: FLINK-32706
> URL: https://issues.apache.org/jira/browse/FLINK-32706
> Project: Flink
> Issue Type: Improvement
>Reporter: Hanyu Zheng
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> SPLIT Function
> Description
> Splits a string into an array of substrings, based on a delimiter.
> Syntax
> The syntax for the SPLIT function is:
> {code:java}
> SPLIT(col1, delimiter){code}
> Splits a string into an array of substrings based on a delimiter. If the
> delimiter is not found, then the original string is returned as the only
> element in the array. If the delimiter is empty, then all characters in the
> string are split. If either, string or delimiter, are NULL, then a NULL value
> is returned.
> If the delimiter is found at the beginning or end of the string, or there are
> contiguous delimiters, then an empty space is added to the array.
> Example
> Let's look at some function examples and explore how to use the SPLIT
> function.
> For example:
>
> {code:java}
> SELECT SPLIT('abcdefg', 'c');
> Result: ['ab', 'defg']
> {code}
> see also:
> 1. ksqlDB Split function
> ksqlDB provides a scalar function named {{SPLIT}} which splits a string into
> an array of substrings based on a delimiter.
> Syntax: {{SPLIT(string, delimiter)}}
> For example: {{SPLIT('a,b,c', ',')}} will return {{{}['a', 'b', 'c']{}}}.
> [https://docs.ksqldb.io/en/0.8.1-ksqldb/developer-guide/ksqldb-reference/scalar-functions/#split]
> 2. Apache Hive Split function
> Hive offers a function named {{split}} which splits a string around a
> specified delimiter and returns an array of strings.
> Syntax: {{array split(string str, string pat)}}
> For example: {{split('a,b,c', ',')}} will return {{{}["a", "b", "c"]{}}}.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
> 3. Spark SQL Split function
> Spark SQL also offers a function named {{{}split{}}}, similar to the one in
> Hive.
> Syntax: {{split(str, pattern[, limit])}}
> Here, {{limit}} is an optional parameter to specify the maximum length of the
> returned array.
> For example: {{split('oneAtwoBthreeC', '[ABC]', 2)}} will return {{{}["one",
> "twoBthreeC"]{}}}.
> [https://spark.apache.org/docs/latest/api/sql/index.html#split]
> 4. Presto Split function
> Presto offers a function named {{split}} which splits a string around a
> regular expression and returns an array of strings.
> Syntax: {{array split(string str, string regex)}}
> For example: {{split('a.b.c', '\.')}} will return {{{}["a", "b", "c"]{}}}.
> [https://prestodb.io/docs/current/functions/string.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31335) using sql-gateway to submit job to yarn cluster, sql-gateway don't support kerberos
[ https://issues.apache.org/jira/browse/FLINK-31335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31335: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > using sql-gateway to submit job to yarn cluster, sql-gateway don't support > kerberos > --- > > Key: FLINK-31335 > URL: https://issues.apache.org/jira/browse/FLINK-31335 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Gateway >Affects Versions: 1.16.0 >Reporter: felixzh >Assignee: felixzh >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.17.2 > > > when submit job to yarn cluster, sql-gateway don't support kerberos. > 1. yarn-per-job mode > -Dexecution.target=yarn-per-job > 2. yarn-session mode > -Dexecution.target=yarn-session -Dyarn.application.id=yarnSessionAppID(eg: > application_1677479737242_0052) > sql-gateway need to use SecurityUtils Modules. > default use flink-conf.yaml(security.kerberos.login.principal and > security.kerberos.login.keytab), also support > -Dsecurity.kerberos.login.keytab and -Dsecurity.kerberos.login.principal (eg: > 1/2) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20628: --- Labels: auto-deprioritized-major pull-request-available stale-assigned (was: auto-deprioritized-major pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: RocMarshal >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32509) avoid using skip in InputStreamFSInputWrapper.seek
[
https://issues.apache.org/jira/browse/FLINK-32509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32509:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> avoid using skip in InputStreamFSInputWrapper.seek
> --
>
> Key: FLINK-32509
> URL: https://issues.apache.org/jira/browse/FLINK-32509
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.18.0
>Reporter: Libin Qin
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> The implementation of InputStream does not return -1 for eof.
> The java doc of InputStream said "The skip method may, for a variety of
> reasons, end up skipping over some smaller number of bytes, possibly 0."
> For FileInputStream, it allows skipping any number of bytes past the end of
> the file.
> So the method "seek" of InputStreamFSInputWrapper will cause infinite loop if
> desired exceed end of file
>
> I reproduced with following case
>
> {code:java}
> byte[] bytes = "flink".getBytes();
> try (InputStream inputStream = new ByteArrayInputStream(bytes)){
> InputStreamFSInputWrapper wrapper = new
> InputStreamFSInputWrapper(inputStream);
> wrapper.seek(20);
> } {code}
> I found an issue of commons-io talks about the problem of skip
> https://issues.apache.org/jira/browse/IO-203
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-12869) Add yarn acls capability to flink containers
[ https://issues.apache.org/jira/browse/FLINK-12869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-12869: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Add yarn acls capability to flink containers > > > Key: FLINK-12869 > URL: https://issues.apache.org/jira/browse/FLINK-12869 > Project: Flink > Issue Type: New Feature > Components: Deployment / YARN >Reporter: Nicolas Fraison >Assignee: Archit Goyal >Priority: Minor > Labels: pull-request-available, stale-assigned > Time Spent: 20m > Remaining Estimate: 0h > > Yarn provide application acls mechanism to be able to provide specific rights > to other users than the one running the job (view logs through the > resourcemanager/job history, kill the application) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31238: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Use IngestDB to speed up Rocksdb rescaling recovery > > > Key: FLINK-31238 > URL: https://issues.apache.org/jira/browse/FLINK-31238 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.16.1 >Reporter: Yue Ma >Assignee: Yue Ma >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: image-2023-02-27-16-41-18-552.png, > image-2023-02-27-16-57-18-435.png, image-2023-03-07-14-27-10-260.png, > image-2023-03-09-15-23-30-581.png, image-2023-03-09-15-26-12-314.png, > image-2023-03-09-15-28-32-363.png, image-2023-03-09-15-41-03-074.png, > image-2023-03-09-15-41-08-379.png, image-2023-03-09-15-45-56-081.png, > image-2023-03-09-15-46-01-176.png, image-2023-03-09-15-50-04-281.png, > image-2023-03-29-15-25-21-868.png, image-2023-07-17-14-37-38-864.png, > image-2023-07-17-14-38-56-946.png, image-2023-07-22-14-16-31-856.png, > image-2023-07-22-14-19-01-390.png, image-2023-08-08-21-32-43-783.png, > image-2023-08-08-21-34-39-008.png, image-2023-08-08-21-39-39-135.png, > screenshot-1.png > > > （The detailed design is in this document > [https://docs.google.com/document/d/10MNVytTsyiDLZQSR89kDkVdmK_YjbM6jh0teerfDFfI|https://docs.google.com/document/d/10MNVytTsyiDLZQSR89kDkVdmK_YjbM6jh0teerfDFfI]） > There have been many discussions and optimizations in the community about > optimizing rocksdb scaling and recovery. > https://issues.apache.org/jira/browse/FLINK-17971 > https://issues.apache.org/jira/browse/FLINK-8845 > https://issues.apache.org/jira/browse/FLINK-21321 > We hope to discuss some of our explorations under this ticket > The process of scaling and recovering in rocksdb simply requires two steps > # Insert the valid keyGroup data of the new task. > # Delete the invalid data in the old stateHandle. > The current method for data writing is to specify the main Db first and then > insert data using writeBatch.In addition, the method of deleteRange is > currently used to speed up the ClipDB. But in our production environment, we > found that the speed of rescaling is still very slow, especially when the > state of a single Task is large. > > We hope that the previous sst file can be reused directly when restoring > state, instead of retraversing the data. So we made some attempts to optimize > it in our internal version of flink and frocksdb. > > We added two APIs *ClipDb* and *IngestDb* in frocksdb. > * ClipDB is used to clip the data of a DB. Different from db.DeteleRange and > db.Delete, DeleteValue and RangeTombstone will not be generated for parts > beyond the key range. We will iterate over the FileMetaData of db. Process > each sst file. There are three situations here. > If all the keys of a file are required, we will keep the sst file and do > nothing > If all the keys of the sst file exceed the specified range, we will delete > the file directly. > If we only need some part of the sst file, we will rewrite the required keys > to generate a new sst file。 > All sst file changes will be placed in a VersionEdit, and the current > versions will LogAndApply this edit to ensure that these changes can take > effect > * IngestDb is used to directly ingest all sst files of one DB into another > DB. But it is necessary to strictly ensure that the keys of the two DBs do > not overlap, which is easy to do in the Flink scenario. The hard link method > will be used in the process of ingesting files, so it will be very fast. At > the same time, the file number of the main DB will be incremented > sequentially, and the SequenceNumber of the main DB will be updated to the > larger SequenceNumber of the two DBs. > When IngestDb and ClipDb are supported, the state restoration logic is as > follows > * Open the first StateHandle as the main DB and pause the compaction. > * Clip the main DB according to the KeyGroup range of the Task with ClipDB
[jira] [Updated] (FLINK-27402) Unexpected boolean expression simplification for AND expression
[
https://issues.apache.org/jira/browse/FLINK-27402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-27402:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Unexpected boolean expression simplification for AND expression
> -
>
> Key: FLINK-27402
> URL: https://issues.apache.org/jira/browse/FLINK-27402
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: luoyuxia
>Assignee: Yunhong Zheng
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> Flink supports to compare between string and boolean, so the following sql
> can work fine
>
> {code:java}
> create table (c1 int, c2 string);
> select * from c2 = true;
> {code}
> But the following sql will throw excpetion
>
> {code:java}
> select * from c1 = 1 and c2 = true; {code}
> The reason it that Flink will try to simplify BOOLEAN expressions if possible
> in
> "c1 = 1 and c2 = true".
> So "c2 = true" will be simplified to "c2" by the following code in Flink:
>
> {code:java}
> RexSimplify#simplifyAnd2ForUnknownAsFalse
> // Simplify BOOLEAN expressions if possible
> while (term.getKind() == SqlKind.EQUALS) {
> RexCall call = (RexCall) term;
> if (call.getOperands().get(0).isAlwaysTrue()) {
> term = call.getOperands().get(1);
> terms.set(i, term);
> continue;
> } else if (call.getOperands().get(1).isAlwaysTrue()) {
> term = call.getOperands().get(0);
> terms.set(i, term);
> continue;
> }
> break;
> } {code}
> So the expression will be reduced to ""c1 = 1 and c2". But AND requries both
> sides are boolean expression and c2 is not a boolean expression for it
> actually is a string.
> Then the exception "Boolean expression type expected" is thrown.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
