[jira] [Created] (FLINK-36316) Move PortRange util to flink-core
Ferenc Csaky created FLINK-36316: Summary: Move PortRange util to flink-core Key: FLINK-36316 URL: https://issues.apache.org/jira/browse/FLINK-36316 Project: Flink Issue Type: Improvement Components: API / Core Reporter: Ferenc Csaky Fix For: 2.0-preview The only thing what the {{PortRange}} utility does is wrapping a {{NetUtils#getPortRangeFromString}} to help keeping track of the original config string. Considering this, it makes sense to move it to the same module, as currently {{PortRange}} is in {{flink-runtime}}, which limits its usage to that module, or requiring to add it as a dependency, which is not a very good option. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35939) Do not set empty config values via ConfigUtils#encodeCollectionToConfig
Ferenc Csaky created FLINK-35939: Summary: Do not set empty config values via ConfigUtils#encodeCollectionToConfig Key: FLINK-35939 URL: https://issues.apache.org/jira/browse/FLINK-35939 Project: Flink Issue Type: Improvement Affects Versions: 1.19.1 Reporter: Ferenc Csaky Fix For: 2.0.0 The {{ConfigUtils#encodeCollectionToConfig}} function only skips to set a given {{ConfigOption}} value, if that value is null. If the passed collection is empty, it will set that empty collection. I think this behavior is less logical and can cause more undesired situations, when we only set a value if it is not empty AND not null. Furthermore, the method's [javadoc|https://github.com/apache/flink/blob/82b628d4730eef32b2f7a022e3b73cb18f950e6e/flink-core/src/main/java/org/apache/flink/configuration/ConfigUtils.java#L73] describes the logic I just mentioned above, which is in conflict with the actual implementation and tests, which sets an empty collection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35699) The flink-kubernetes artifact shades Jackson 2.15.3 from fabric8
Ferenc Csaky created FLINK-35699: Summary: The flink-kubernetes artifact shades Jackson 2.15.3 from fabric8 Key: FLINK-35699 URL: https://issues.apache.org/jira/browse/FLINK-35699 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.19.1 Reporter: Ferenc Csaky Fix For: 1.20.0, 1.19.2 The {{flink-kubernetes}} artifact shades Jackson classes coming through fabric8, but since Jackson 2.15, Jackson is a [multi-release JAR|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15#jar-changes], which requires some additional relocations for correct shading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35695) Release Testing: Verify FLINK-32315: Support local file upload in K8s mode
Ferenc Csaky created FLINK-35695: Summary: Release Testing: Verify FLINK-32315: Support local file upload in K8s mode Key: FLINK-35695 URL: https://issues.apache.org/jira/browse/FLINK-35695 Project: Flink Issue Type: Sub-task Components: Runtime / Network Reporter: Ferenc Csaky Fix For: 1.20.0 Follow up the test for https://issues.apache.org/jira/browse/FLINK-35533 In Flink 1.20, we proposed integrating Flink's Hybrid Shuffle with Apache Celeborn through a pluggable remote tier interface. To verify this feature, you should reference these main two steps. 1. Implement Celeborn tier. * Implement a new tier factory and tier for Celeborn, including these APIs, including TierFactory/TierMasterAgent/TierProducerAgent/TierConsumerAgent. * The implementations should support granular data management at the Segment level for both client and server sides. 2. Use the implemented tier to shuffle data. * Compile Flink and Celeborn. * Deploy Celeborn service ** Deploy a new Celeborn service with the new compiled packages. You can reference the doc ([https://celeborn.apache.org/docs/latest/]) to deploy the cluster. * Add the compiled flink plugin jar (celeborn-client-flink-xxx.jar) to Flink classpath. * Configure the options to enable the feature. ** Configure the option taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class to the new Celeborn tier classes. Except for this option, the following options should also be added. {code:java} execution.batch-shuffle-mode: ALL_EXCHANGES_HYBRID_FULL celeborn.master.endpoints: celeborn.client.shuffle.partition.type: MAP{code} * Run some test examples(e.g., WordCount) to verify the feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35662) Use maven batch mode in k8s-operator CI
Ferenc Csaky created FLINK-35662: Summary: Use maven batch mode in k8s-operator CI Key: FLINK-35662 URL: https://issues.apache.org/jira/browse/FLINK-35662 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Ferenc Csaky Fix For: kubernetes-operator-1.10.0 Currently, the GitHub workflows do not use batch mode in the k8s-operator repo, so there are a lot of lines in the log like this: {code} Progress (1): 4.1/14 kB Progress (1): 8.2/14 kB Progress (1): 12/14 kB Progress (1): 14 kB {code} To produce logs that are for more easy to navigate, all {{mvn}} calls should apply the batch-mode option {{-B}}. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35649) Bump Flink version to 1.19.1 in k8s-operator
Ferenc Csaky created FLINK-35649: Summary: Bump Flink version to 1.19.1 in k8s-operator Key: FLINK-35649 URL: https://issues.apache.org/jira/browse/FLINK-35649 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Ferenc Csaky Fix For: kubernetes-operator-1.10.0 In FLINK-28915 it came up the the operator is not able to utilize the artifact fetching capabilities that was introduced in Flink 1.19 until it is not built on that version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35625) FLIP-464: Merge "flink run" and "flink run-application"
Ferenc Csaky created FLINK-35625: Summary: FLIP-464: Merge "flink run" and "flink run-application" Key: FLINK-35625 URL: https://issues.apache.org/jira/browse/FLINK-35625 Project: Flink Issue Type: Improvement Components: Client / Job Submission, Command Line Client Reporter: Ferenc Csaky Fix For: 1.20.0 Ticket to track [FLIP-464|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35114) Remove old Table API implementations
Ferenc Csaky created FLINK-35114: Summary: Remove old Table API implementations Key: FLINK-35114 URL: https://issues.apache.org/jira/browse/FLINK-35114 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky At the moment, the connector has both the old Table sink/source/catalog implementations and the matching Dynamic... implementations as well. Going forward, the deprecated old implementation should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34931) Update Kudu connector DataStream Source/Sink implementation
Ferenc Csaky created FLINK-34931: Summary: Update Kudu connector DataStream Source/Sink implementation Key: FLINK-34931 URL: https://issues.apache.org/jira/browse/FLINK-34931 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky Update the DataSource API classes to use the current interfaces. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34930) Move existing Kudu connector code from Bahir repo to dedicated repo
Ferenc Csaky created FLINK-34930: Summary: Move existing Kudu connector code from Bahir repo to dedicated repo Key: FLINK-34930 URL: https://issues.apache.org/jira/browse/FLINK-34930 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34929) Create "flink-connector-kudu" repository
Ferenc Csaky created FLINK-34929: Summary: Create "flink-connector-kudu" repository Key: FLINK-34929 URL: https://issues.apache.org/jira/browse/FLINK-34929 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky We should create a "flink-connector-kudu" repositry under the "apache" GitHub organization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34928) FLIP-439: Externalize Kudu Connector from Bahir
Ferenc Csaky created FLINK-34928: Summary: FLIP-439: Externalize Kudu Connector from Bahir Key: FLINK-34928 URL: https://issues.apache.org/jira/browse/FLINK-34928 Project: Flink Issue Type: Improvement Reporter: Ferenc Csaky Umbrella issue for: https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34580) Job run via REST erases "pipeline.classpaths" config
Ferenc Csaky created FLINK-34580: Summary: Job run via REST erases "pipeline.classpaths" config Key: FLINK-34580 URL: https://issues.apache.org/jira/browse/FLINK-34580 Project: Flink Issue Type: Bug Components: Runtime / REST Affects Versions: 1.18.1, 1.17.2, 1.19.0 Reporter: Ferenc Csaky Fix For: 1.20.0 The [{{JarHandlerContext#applyToConfiguration}}|https://github.com/apache/flink/blob/e0b6c121eaf7aeb2974a45d199e452b022f07d29/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/utils/JarHandlerUtils.java#L134] creates a {{PackagedProgram}} and then overwrites the {{pipeline.jars}} and {{pipeline.classpaths}} values according to that newly created {{{}PackagedProgram{}}}. Although that [{{PackagedProgram}} init|https://github.com/apache/flink/blob/e0b6c121eaf7aeb2974a45d199e452b022f07d29/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/utils/JarHandlerUtils.java#L185] does not set {{classpaths}} at all, so it will always overwrites the effective configuration with an empty value, even if it had something previously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34506) Do not copy "file://" schemed artifact in standalone application modes
Ferenc Csaky created FLINK-34506: Summary: Do not copy "file://" schemed artifact in standalone application modes Key: FLINK-34506 URL: https://issues.apache.org/jira/browse/FLINK-34506 Project: Flink Issue Type: Bug Components: Client / Job Submission Affects Versions: 1.19.0 Reporter: Ferenc Csaky In standalone application mode, if an artifact is passed via a path witohut prefix, the file will be copied to `user.artifacts.base-dir`, although it should not be, as it can accessable locally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34388) Release Testing: Verify FLINK-28915 Support artifact fetching in Standalone and native K8s application mode
Ferenc Csaky created FLINK-34388: Summary: Release Testing: Verify FLINK-28915 Support artifact fetching in Standalone and native K8s application mode Key: FLINK-34388 URL: https://issues.apache.org/jira/browse/FLINK-34388 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Affects Versions: 1.19.0 Reporter: Ferenc Csaky Fix For: 1.19.0 This ticket covers testing three related features: FLINK-33695, FLINK-33735 and FLINK-33696. Instructions: # Configure Flink to use [Slf4jTraceReporter|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/trace_reporters/#slf4j] and with enabled *INFO* level logging (can be to console or to a file, doesn't matter). # Start a streaming job with enabled checkpointing. # Let it run for a couple of checkpoints. # Verify presence of a single *JobInitialization* [1] trace logged just after job start up. # Verify presence of a couple of *Checkpoint* [1] traces logged after each successful or failed checkpoint. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/traces/#checkpointing-and-initialization -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33542) Update HBase connector tests to JUnit5
Ferenc Csaky created FLINK-33542: Summary: Update HBase connector tests to JUnit5 Key: FLINK-33542 URL: https://issues.apache.org/jira/browse/FLINK-33542 Project: Flink Issue Type: Improvement Components: Connectors / HBase Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33440) Bump flink version on flink-connectors-hbase
Ferenc Csaky created FLINK-33440: Summary: Bump flink version on flink-connectors-hbase Key: FLINK-33440 URL: https://issues.apache.org/jira/browse/FLINK-33440 Project: Flink Issue Type: Improvement Components: Connectors / HBase Reporter: Ferenc Csaky Follow-up the 1.18 release in the connector repo as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33353) SQL fails because "TimestampType.kind" is not serialized
Ferenc Csaky created FLINK-33353: Summary: SQL fails because "TimestampType.kind" is not serialized Key: FLINK-33353 URL: https://issues.apache.org/jira/browse/FLINK-33353 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.18.0 Reporter: Ferenc Csaky We have a custom persistent catalog store, which stores tables, views etc. in a DB. In our application, it is required to utilize the serialized formats of entities, but the same applies to the Hive, as it functions as a persistent catalog. Take the following example SQL: {code:sql} CREATE TABLE IF NOT EXISTS `txn_gen` ( `txn_id` INT, `amount` INT, `ts` TIMESTAMP(3), WATERMARK FOR `ts` AS `ts` - INTERVAL '1' SECOND ) WITH ( 'connector' = 'datagen', 'fields.txn_id.min' = '1', 'fields.txn_id.max' = '5', 'rows-per-second' = '1' ); CREATE VIEW IF NOT EXISTS aggr_ten_sec AS SELECT txn_id, TUMBLE_ROWTIME(`ts`, INTERVAL '10' SECOND) AS w_row_time, COUNT(txn_id) AS txn_count FROM txn_gen GROUP BY txn_id, TUMBLE(`ts`, INTERVAL '10' SECOND); SELECT txn_id, SUM(txn_count), TUMBLE_START(w_row_time, INTERVAL '20' SECOND) AS total_txn_count FROM aggr_ten_sec GROUP BY txn_id, TUMBLE(w_row_time, INTERVAL '20' SECOND); {code} This will work without any problems when we simply execute it in a {{TableEnvironment}}, but it fails with the below error when we try to execute the query based on the serialized table metadata. {code} org.apache.flink.table.api.TableException: Window aggregate can only be defined over a time attribute column, but TIMESTAMP(3) encountered. {code} If there is a view which would require to use ROWTIME, it will be lost and we cannot recreate the same query from the serialized entites. Currently in {{TimestampType}} the "kind" field is deliberatly annotated as {{@Internal}} and is not serialized, although it breaks this functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32811) Add port range support for taskmanager.data.bind-port
Ferenc Csaky created FLINK-32811: Summary: Add port range support for taskmanager.data.bind-port Key: FLINK-32811 URL: https://issues.apache.org/jira/browse/FLINK-32811 Project: Flink Issue Type: Improvement Components: Runtime / Configuration, Runtime / Coordination Reporter: Ferenc Csaky Fix For: 1.19.0 Adding this feature could be helpful for installation in a restrictive network setup. The "port range" support is already available for some other port config options anyway. Right now, it is possible to specify a {{taskmanager.data.port}} and {{taskmanager.data.bind-port}} to be able to support NAT-like setups, although {{taskmanager.data.port}} is not bound to anything itself, so supporting a port range there is not an option according to my understanding. Although, supporting a port range only for {{taskmanager.data.bind-port}} can be still helpful for anyone who does not require a NAT capability, because if {{taskmanager.data.bind-port}} is set and {{taskmanager.data.port}} is set to *0*, then the bound port will be used everywhere. This change should keep the already possible setups working as is. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32660) Support external file systems in FileCatalogStore
Ferenc Csaky created FLINK-32660: Summary: Support external file systems in FileCatalogStore Key: FLINK-32660 URL: https://issues.apache.org/jira/browse/FLINK-32660 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky Fix For: 1.18.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32174) Update Cloudera product and link in doc page
Ferenc Csaky created FLINK-32174: Summary: Update Cloudera product and link in doc page Key: FLINK-32174 URL: https://issues.apache.org/jira/browse/FLINK-32174 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31085) Add schema option to confluent registry avro formats
Ferenc Csaky created FLINK-31085: Summary: Add schema option to confluent registry avro formats Key: FLINK-31085 URL: https://issues.apache.org/jira/browse/FLINK-31085 Project: Flink Issue Type: Improvement Reporter: Ferenc Csaky Fix For: 1.17.0 When using {{avro-confluent}} and {{debezium-avro-confluent}} formats with schemas already defined in the Confluent Schema Registry, serialization fails, because Flink uses a default name `record` when converting row types to avro schema. So if the predefined schema has a different name, the serialization schema will be incompatible with the registered schema due to name mismatch. Check [this|https://lists.apache.org/thread/5xppmnqjqwfzxqo4gvd3lzz8wzs566zp] thread about reproducing the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29707) Fix possible comparator violation for "flink list"
Ferenc Csaky created FLINK-29707: Summary: Fix possible comparator violation for "flink list" Key: FLINK-29707 URL: https://issues.apache.org/jira/browse/FLINK-29707 Project: Flink Issue Type: Bug Components: Command Line Client Affects Versions: 1.16.0 Reporter: Ferenc Csaky For the {{list}} CLI option, the code that prints the jobs, there is a {{startTimeComparator}} definition, which orders the jobs and it is done this way: {code:java} Comparator startTimeComparator = (o1, o2) -> (int) (o1.getStartTime() - o2.getStartTime()); {code} In some rare situation this can lead to this: {code:java} 2022-10-19 09:58:11,690 ERROR org.apache.flink.client.cli.CliFrontend [] - Error while running the command. java.lang.IllegalArgumentException: Comparison method violates its general contract! at java.util.TimSort.mergeLo(TimSort.java:777) ~[?:1.8.0_312] at java.util.TimSort.mergeAt(TimSort.java:514) ~[?:1.8.0_312] at java.util.TimSort.mergeForceCollapse(TimSort.java:457) ~[?:1.8.0_312] at java.util.TimSort.sort(TimSort.java:254) ~[?:1.8.0_312] at java.util.Arrays.sort(Arrays.java:1512) ~[?:1.8.0_312] at java.util.ArrayList.sort(ArrayList.java:1464) ~[?:1.8.0_312] at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:392) ~[?:1.8.0_312] at java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[?:1.8.0_312] at java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[?:1.8.0_312] at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:363) ~[?:1.8.0_312] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) ~[?:1.8.0_312] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_312] at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:1.8.0_312] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:1.8.0_312] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_312] at java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:490) ~[?:1.8.0_312] at org.apache.flink.client.cli.CliFrontend.printJobStatusMessages(CliFrontend.java:574) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-27441) Scrollbar is missing for particular UI elements (Accumulators, Backpressure, Watermarks)
Ferenc Csaky created FLINK-27441: Summary: Scrollbar is missing for particular UI elements (Accumulators, Backpressure, Watermarks) Key: FLINK-27441 URL: https://issues.apache.org/jira/browse/FLINK-27441 Project: Flink Issue Type: Bug Components: Runtime / Web Frontend Affects Versions: 1.14.3, 1.15.0 Reporter: Ferenc Csaky The angular version bump introduced a bug, where for {{nzScroll}} does not support percentage in CSS calc, so the scrollbar will be invisible. There is an easy workaround, the linked Angular discussion covers it. Angular issue: https://github.com/NG-ZORRO/ng-zorro-antd/issues/3090 -- This message was sent by Atlassian Jira (v8.20.7#820007)