Re: [PR] [FLINK-32082][docs] Documentation of checkpoint file-merging [flink]
masteryhx commented on code in PR #24766: URL: https://github.com/apache/flink/pull/24766#discussion_r1597905886 ## docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md: ## @@ -292,4 +292,25 @@ The final checkpoint would be triggered immediately after all operators have rea without waiting for periodic triggering, but the job will need to wait for this final checkpoint to be completed. +## Unify file merging mechanism for checkpoints + +Flink 1.20 introduces a unified file merging mechanism for checkpointing, which allows scattered small checkpoint files Review Comment: Is it a mvp or production ready in Flink 1.20 ? We'd better to describe it if it's a mvp. ## docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md: ## @@ -292,4 +292,25 @@ The final checkpoint would be triggered immediately after all operators have rea without waiting for periodic triggering, but the job will need to wait for this final checkpoint to be completed. +## Unify file merging mechanism for checkpoints + +Flink 1.20 introduces a unified file merging mechanism for checkpointing, which allows scattered small checkpoint files +to be written into a single file, reducing the number of file creations and file deletions, helping to alleviate the pressure +of file system metadata management and file flooding problem. The unified fie merging mechanism can be enabled by setting +the property `state.checkpoints.file-merging.enabled` to `true`. **Note** that enabling this mechanism may lead to space amplification, +that is, the actual occupation on the file system will be larger than the checkpoint size. `state.checkpoints.file-merging.max-space-amplification` Review Comment: IIUC, the metric of checkpoint size should be consistent, right ? If It's compared with before or acutal state, let's adjust it as `before checkpoint size` or `actual state size`. ## docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md: ## @@ -292,4 +292,25 @@ The final checkpoint would be triggered immediately after all operators have rea without waiting for periodic triggering, but the job will need to wait for this final checkpoint to be completed. +## Unify file merging mechanism for checkpoints + +Flink 1.20 introduces a unified file merging mechanism for checkpointing, which allows scattered small checkpoint files +to be written into a single file, reducing the number of file creations and file deletions, helping to alleviate the pressure +of file system metadata management and file flooding problem. The unified fie merging mechanism can be enabled by setting +the property `state.checkpoints.file-merging.enabled` to `true`. **Note** that enabling this mechanism may lead to space amplification, +that is, the actual occupation on the file system will be larger than the checkpoint size. `state.checkpoints.file-merging.max-space-amplification` +can be used to limit the upper bound of space amplification. + +This mechanism is applicable to all types of state files in Flink, including keyed state, operator state, channel state +and changelog DSTL files. Subtask level granular merging is provided for shared scope state; TaskManager-level granular merging Review Comment: some unsupported types, e.g. DSTL files could be removed for now or listed as a limitation ? ## docs/content.zh/docs/dev/datastream/fault-tolerance/checkpointing.md: ## @@ -250,5 +250,20 @@ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironm 需要注意的是，这一行为可能会延长任务运行的时间，如果 checkpoint 周期比较大，这一延迟会非常明显。 极端情况下，如果 checkpoint 的周期被设置为 `Long.MAX_VALUE`，那么任务永远不会结束，因为下一次 checkpoint 不会进行。 +## 统一的 checkpoint 文件合并机制 + +Flink 1.20 引入了统一的 checkpoint 文件合并机制，该机制允许把分散的 checkpoint 文件写到同一个文件中，减少 checkpoint 文件创建删除的次数， +有助于减轻文件系统元数据管理的压力、 解决文件洪泛问题。可以通过将 `state.checkpoints.file-merging.enabled` 设置为 `true` 来开启该机制。 Review Comment: `文件洪泛问题` seems not a common description in chinese. How about just describing it more directly ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35030][runtime] Introduce Epoch Manager under async execution [flink]
yunfengzhou-hub commented on code in PR #24748:
URL: https://github.com/apache/flink/pull/24748#discussion_r1597912452
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/EpochManager.java:
##
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing;
+
+import javax.annotation.Nullable;
+
+import java.util.LinkedList;
+
+/**
+ * Epoch manager segments inputs into distinct epochs, marked by the arrival
of non-records(e.g.
+ * watermark, record attributes). Records are assigned to a unique epoch based
on their arrival,
+ * records within an epoch are allowed to be parallelized, while the
non-record of an epoch can only
+ * be executed when all records in this epoch have finished.
+ *
+ * For more details please refer to FLIP-425.
+ */
+public class EpochManager {
+
+/**
+ * This enum defines whether parallel execution between epochs is allowed.
We should keep this
+ * internal and away from API module for now, until we could see the
concrete need for {@link
+ * #PARALLEL_BETWEEN_EPOCH} from average users.
+ */
+public enum ParallelMode {
+/**
+ * Subsequent epochs must wait until the previous epoch is completed
before they can start.
+ */
+SERIAL_BETWEEN_EPOCH,
+/**
+ * Subsequent epochs can begin execution even if the previous epoch
has not yet completed.
+ * Usually performs better than {@link #SERIAL_BETWEEN_EPOCH}.
+ */
+PARALLEL_BETWEEN_EPOCH
+}
+
+/**
+ * The reference to the {@link AsyncExecutionController}, used for {@link
+ * ParallelMode#SERIAL_BETWEEN_EPOCH}. Can be null when testing.
+ */
+final AsyncExecutionController asyncExecutionController;
+
+/** The number of epochs that have arrived. */
+long epochNum;
+
+/** The output queue to hold ongoing epochs. */
+LinkedList outputQueue;
+
+/** Current active epoch, only one active epoch at the same time. */
+Epoch activeEpoch;
+
+public EpochManager(AsyncExecutionController aec) {
+this.epochNum = 0;
+this.outputQueue = new LinkedList<>();
+// preset an empty epoch, the epoch action will be updated when
non-record is received.
+this.activeEpoch = new Epoch(epochNum++);
+this.outputQueue.add(activeEpoch);
+this.asyncExecutionController = aec;
+}
+
+/**
+ * Add a record to the current epoch and return the current open epoch,
the epoch will be
+ * associated with the {@link RecordContext} of this record. Must be
invoked within task thread.
+ *
+ * @return the current open epoch.
+ */
+public Epoch onRecord() {
+activeEpoch.ongoingRecordCount++;
+return activeEpoch;
+}
+
+/**
+ * Add a non-record to the current epoch, close current epoch and open a
new epoch. Must be
+ * invoked within task thread.
+ *
+ * @param action the action associated with this non-record.
+ * @param parallelMode the parallel mode for this epoch.
+ */
+public void onNonRecord(Runnable action, ParallelMode parallelMode) {
Review Comment:
Got it. Given that we agreed to add
`AsyncExecutionController.processNonRecord` in another comment and that
`AsyncExecutionController` has `exceptionHandler`, how about keeping
EpochManager using Runnable, and change
`AsyncExecutionController.processNonRecord` to use ThrowingRunnable?
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -311,6 +324,22 @@ public void drainInflightRecords(int targetNum) {
}
}
+public void processNonRecord(Runnable action) {
+epochManager.onNonRecord(action, epochParallelMode);
+}
+
+/**
+ * Configure the parallel mode of epoch execution. We should keep this
method internal for now,
+ * until we could see the concrete need for {@link
ParallelMode#PARALLEL_BETWEEN_EPOCH} from
+ * average users.
+ *
+ * @param parallelMode the parallel mode to set.
+ */
+@Internal
+void configureEpochParallelMode(ParallelMode parallelMode)
Re: [PR] [FLINK-35168][State] Basic State Iterator for async processing [flink]
masteryhx commented on code in PR #24690:
URL: https://github.com/apache/flink/pull/24690#discussion_r1597852929
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AbstractStateIterator.java:
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing;
+
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.state.v2.StateIterator;
+import org.apache.flink.core.state.InternalStateFuture;
+import org.apache.flink.core.state.StateFutureUtils;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.function.Consumer;
+import java.util.function.Function;
+
+/**
+ * A {@link StateIterator} implementation to facilitate async data load of
iterator. Each state
+ * backend could override this class to maintain more variables in need. Any
subclass should
+ * implement two methods, {@link #hasNext()} and {@link
#nextPayloadForContinuousLoading()}. The
+ * philosophy behind this class is to carry some already loaded elements and
provide iterating right
+ * on the task thread, and load following ones if needed (determined by {@link
#hasNext()}) by
+ * creating **ANOTHER** iterating request. Thus, later it returns another
iterator instance, and we
+ * continue to apply the user iteration on that instance. The whole elements
will be iterated by
+ * recursive call of {@code #onNext()}.
+ */
+@SuppressWarnings("rawtypes")
+public abstract class AbstractStateIterator implements StateIterator {
+
+/** The state this iterator iterates on. */
+final State originalState;
+
+/** The request type that create this iterator. */
+final StateRequestType requestType;
+
+/** The controller that can receive further requests. */
+final AsyncExecutionController aec;
Review Comment:
Could rebase and be `StateRequestHandler`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [hot][doc-ci] Add release 3.1 and enable build check for pr [flink-cdc]
GOODBOY008 opened a new pull request, #3317: URL: https://github.com/apache/flink-cdc/pull/3317 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35047][state] Shutdown StateExecutors when ForStKeyedStateBackend is disposed [flink]
masteryhx commented on code in PR #24768:
URL: https://github.com/apache/flink/pull/24768#discussion_r1597840604
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStKeyedStateBackend.java:
##
@@ -147,15 +160,30 @@ public S createState(@Nonnull
StateDescriptor stateDes
@Override
@Nonnull
public StateExecutor createStateExecutor() {
-// TODO: Make io parallelism configurable
-return new ForStStateExecutor(4, db,
optionsContainer.getWriteOptions());
+synchronized (lock) {
Review Comment:
Also shutdown in the close method ?
`close` is used to close resources created internally.
`dispose` is used to dispose resources including native resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35237) Allow Sink to Choose HashFunction in PrePartitionOperator
[
https://issues.apache.org/jira/browse/FLINK-35237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangdingxin updated FLINK-35237:
-
Fix Version/s: cdc-3.2.0
> Allow Sink to Choose HashFunction in PrePartitionOperator
> -
>
> Key: FLINK-35237
> URL: https://issues.apache.org/jira/browse/FLINK-35237
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: zhangdingxin
>Priority: Major
> Fix For: cdc-3.2.0
>
>
> The {{PrePartitionOperator}} in its current implementation only supports a
> fixed {{HashFunction}}
> ({{{}org.apache.flink.cdc.runtime.partitioning.PrePartitionOperator.HashFunction{}}}).
> This limits the ability of Sink implementations to customize the
> partitioning logic for {{{}DataChangeEvent{}}}s. For example, in the case of
> partitioned tables, it would be advantageous to allow hashing based on
> partition keys, hashing according to table names, or using the database
> engine's internal primary key hash functions (such as with MaxCompute
> DataSink).
> When users require such custom partitioning logic, they are compelled to
> implement their PartitionOperator, which undermines the utility of
> {{{}PrePartitionOperator{}}}.
> To address this limitation, it would be highly desirable to enable the
> {{PrePartitionOperator}} to support user-specified custom
> {{{}HashFunction{}}}s (Function). A possible
> solution could involve a mechanism analogous to the {{DataSink}} interface,
> allowing the specification of a {{HashFunctionProvider}} class path in the
> configuration file. This enhancement would greatly facilitate users in
> tailoring partition strategies to meet their specific application needs.
> In this case, I want to create new class {{HashFunctionProvider}} and
> {{{}HashFunction{}}}:
> {code:java}
> public interface HashFunctionProvider {
> HashFunction getHashFunction(Schema schema);
> }
> public interface HashFunction extends Function {
> Integer apply(DataChangeEvent event);
> } {code}
> add {{getHashFunctionProvider}} method to {{DataSink}}
>
> {code:java}
> public interface DataSink {
> /** Get the {@link EventSinkProvider} for writing changed data to
> external systems. */
> EventSinkProvider getEventSinkProvider();
> /** Get the {@link MetadataApplier} for applying metadata changes to
> external systems. */
> MetadataApplier getMetadataApplier();
> default HashFunctionProvider getHashFunctionProvider() {
> return new DefaultHashFunctionProvider();
> }
> } {code}
> and re-implement {{PrePartitionOperator}} {{recreateHashFunction}} method.
> {code:java}
> private HashFunction recreateHashFunction(TableId tableId) {
> return
> hashFunctionProvider.getHashFunction(loadLatestSchemaFromRegistry(tableId));
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35328] AutoScale supports setting the maximum floating parallelism by the number of Pulsar partitions [flink-kubernetes-operator]
wenbingshen commented on PR #827: URL: https://github.com/apache/flink-kubernetes-operator/pull/827#issuecomment-2106527073 > Thanks for the improvement! > > LGTM > > cc @gyfora @1996fanrui Can anyone else review this PR? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34615]Split `ExternalizedCheckpointCleanup` out of `Checkpoint… [flink]
spoon-lz commented on PR #24461: URL: https://github.com/apache/flink/pull/24461#issuecomment-2106510219 @Zakelly @masteryhx A new `ExternalizedCheckpointRetention.py` is created, the old one is marked as deprecated, and a separate commit for the python part is split. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33986][runtime] Extend ShuffleMaster to support snapshot and restore state. [flink]
flinkbot commented on PR #24774: URL: https://github.com/apache/flink/pull/24774#issuecomment-2106499628 ## CI report: * 93334c7356336561bc6a97c1eb8ceb7509f647a7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33986) Extend shuffleMaster to support batch snapshot.
[ https://issues.apache.org/jira/browse/FLINK-33986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33986: --- Labels: pull-request-available (was: ) > Extend shuffleMaster to support batch snapshot. > --- > > Key: FLINK-33986 > URL: https://issues.apache.org/jira/browse/FLINK-33986 > Project: Flink > Issue Type: Sub-task >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > > Extend shuffleMaster to support batch snapshot as follows: > # Add method supportsBatchSnapshot to identify whether the shuffle master > supports taking snapshot in batch scenarios > # Add method snapshotState and restoreState to snapshot and restore the > shuffle master's state. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33986][runtime] Extend ShuffleMaster to support snapshot and restore state. [flink]
JunRuiLee opened a new pull request, #24774: URL: https://github.com/apache/flink/pull/24774 ## What is the purpose of the change Extend ShuffleMaster to support snapshot and restore state. ## Brief change log Extend shuffleMaster to support batch snapshot as follows: 1. Add method supportsBatchSnapshot to identify whether the shuffle master supports taking snapshot in batch scenarios 2. Add method snapshotState and restoreState to snapshot and restore the shuffle master's state. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35197) Support the execution of suspend, resume materialized table in continuous refresh mode
[
https://issues.apache.org/jira/browse/FLINK-35197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845761#comment-17845761
]
dalongliu commented on FLINK-35197:
---
Merged in master: e4972c003f68da6dc4066459d4c6e5d981f07e96
> Support the execution of suspend, resume materialized table in continuous
> refresh mode
> --
>
> Key: FLINK-35197
> URL: https://issues.apache.org/jira/browse/FLINK-35197
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Gateway
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: Feng Jin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> In continuous refresh mode, support suspend, resume the background refresh
> job of materialized table.
> {code:SQL}
> // suspend
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name SUSPEND
> // resume
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name RESUME
> [WITH('key1' = 'val1', 'key2' = 'val2')]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-35197) Support the execution of suspend, resume materialized table in continuous refresh mode
[
https://issues.apache.org/jira/browse/FLINK-35197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dalongliu resolved FLINK-35197.
---
Resolution: Fixed
> Support the execution of suspend, resume materialized table in continuous
> refresh mode
> --
>
> Key: FLINK-35197
> URL: https://issues.apache.org/jira/browse/FLINK-35197
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Gateway
>Affects Versions: 1.20.0
>Reporter: dalongliu
>Assignee: Feng Jin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> In continuous refresh mode, support suspend, resume the background refresh
> job of materialized table.
> {code:SQL}
> // suspend
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name SUSPEND
> // resume
> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name RESUME
> [WITH('key1' = 'val1', 'key2' = 'val2')]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35197][table] Support the execution of supsend materialized table in continuous refresh mode [flink]
lsyldliu closed pull request #24765: [FLINK-35197][table] Support the execution of supsend materialized table in continuous refresh mode URL: https://github.com/apache/flink/pull/24765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-24379][Formats] Avro Glue Schema Registry table format [flink-connector-aws]
nicusX commented on code in PR #122:
URL:
https://github.com/apache/flink-connector-aws/pull/122#discussion_r1597692785
##
flink-formats-aws/flink-sql-avro-glue-schema-registry/pom.xml:
##
@@ -0,0 +1,163 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+4.0.0
+
+
+org.apache.flink
+flink-formats-aws-parent
+4.3-SNAPSHOT
+
+
+flink-sql-avro-glue-schema-registry
+Flink : Formats : AWS : SQL : Avro Glue Schema Registry
+jar
+
+
+
+org.apache.flink
+flink-avro-glue-schema-registry
+${project.version}
+
+
+org.apache.flink
+flink-test-utils
+${flink.version}
+test
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+
+
+shade-flink
+package
+
+shade
+
+
+
+
+
org.apache.flink:flink-connector-aws-base
+
org.apache.flink:flink-avro-glue-schema-registry
+com.amazonaws:*
+software.amazon.awssdk:*
+software.amazon.glue:*
+
+
org.apache.flink:flink-avro
+org.apache.avro:avro
+
com.fasterxml.jackson.core:*
+
com.fasterxml.jackson.dataformat:*
+
org.apache.commons:commons-compress
+
org.reactivestreams:reactive-streams
+com.google.guava:guava
+
com.google.guava:failureaccess
+
+
+
software.amazon.glue:schema-registry-build-tools
+
+
com.google.guava:listenablefuture
+
org.checkerframework:checker-qual
+
com.google.errorprone:error_prone_annotations
+
com.google.j2objc:j2objc-annotations
+
com.google.code.findbugs:jsr305
+
+
+
+
+software.amazon
+
org.apache.flink.avro.registry.glue.shaded.software.amazon
+
+
+com.amazonaws
+
org.apache.flink.avro.registry.glue.shaded.com.amazonaws
+
+
+org.apache.kafka
+
org.apache.flink.avro.registry.glue.shaded.org.apache.kafka
+
+
+org.reactivestreams
+
org.apache.flink.avro.registry.glue.shaded.org.reactivestreams
+
Review Comment:
They don't seems to be actually used by the AVRO SerDe: excluded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-31228] Update kafka and zookeeper docker image [flink-playgrounds]
robertkosz commented on PR #41: URL: https://github.com/apache/flink-playgrounds/pull/41#issuecomment-2106323518 Since all wurstmeister images are gone on docker, this is kind of a must now, docker compose up fails on operations-playground. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
superdiaodiao commented on PR #24773: URL: https://github.com/apache/flink/pull/24773#issuecomment-2106260978 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-34108) Add URL_ENCODE and URL_DECODE function
[ https://issues.apache.org/jira/browse/FLINK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845659#comment-17845659 ] chesterxu edited comment on FLINK-34108 at 5/12/24 1:42 PM: Hey, there~ If no one was assigned, please check this PR: [https://github.com/apache/flink/pull/24773] was (Author: JIRAUSER302535): https://github.com/apache/flink/pull/24773 > Add URL_ENCODE and URL_DECODE function > -- > > Key: FLINK-34108 > URL: https://issues.apache.org/jira/browse/FLINK-34108 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Priority: Major > Labels: pull-request-available > > Add URL_ENCODE and URL_DECODE function > URL_ENCODE(str) - Translates a string into > 'application/x-www-form-urlencoded' format using a specific encoding scheme. > URL_DECODE(str) - Decodes a string in 'application/x-www-form-urlencoded' > format using a specific encoding scheme. > Related ticket from Calcite: CALCITE-5825 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
superdiaodiao commented on PR #24773: URL: https://github.com/apache/flink/pull/24773#issuecomment-2106248298 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
flinkbot commented on PR #24773: URL: https://github.com/apache/flink/pull/24773#issuecomment-2106247014 ## CI report: * d7b8727596cda8b1feb6edb3c465043ed764fa4c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-34108) Add URL_ENCODE and URL_DECODE function
[ https://issues.apache.org/jira/browse/FLINK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845659#comment-17845659 ] chesterxu edited comment on FLINK-34108 at 5/12/24 1:24 PM: https://github.com/apache/flink/pull/24773 was (Author: JIRAUSER302535): I would like to support it, please assign to me. > Add URL_ENCODE and URL_DECODE function > -- > > Key: FLINK-34108 > URL: https://issues.apache.org/jira/browse/FLINK-34108 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Priority: Major > Labels: pull-request-available > > Add URL_ENCODE and URL_DECODE function > URL_ENCODE(str) - Translates a string into > 'application/x-www-form-urlencoded' format using a specific encoding scheme. > URL_DECODE(str) - Decodes a string in 'application/x-www-form-urlencoded' > format using a specific encoding scheme. > Related ticket from Calcite: CALCITE-5825 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34108) Add URL_ENCODE and URL_DECODE function
[ https://issues.apache.org/jira/browse/FLINK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34108: --- Labels: pull-request-available (was: ) > Add URL_ENCODE and URL_DECODE function > -- > > Key: FLINK-34108 > URL: https://issues.apache.org/jira/browse/FLINK-34108 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Priority: Major > Labels: pull-request-available > > Add URL_ENCODE and URL_DECODE function > URL_ENCODE(str) - Translates a string into > 'application/x-www-form-urlencoded' format using a specific encoding scheme. > URL_DECODE(str) - Decodes a string in 'application/x-www-form-urlencoded' > format using a specific encoding scheme. > Related ticket from Calcite: CALCITE-5825 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
superdiaodiao opened a new pull request, #24773: URL: https://github.com/apache/flink/pull/24773 ## What is the purpose of the change This is an implementation of URL_ENCODE and URL_DECODE 1. URL_ENCODE: Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8). 2. URL_DECODE: Decodes a string in 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8). ## Brief change log 1. **URL_ENCODE** - Syntax: url_encode(url) - Arguments: url: a string represents a URL - Returns: translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8), will be null if input is null. - Examples: ``` url = 'https://flink.apache.org/' SQL: url_encode(url) TableAPI: url.urlEncode() output: 'https%3A%2F%2Fflink.apache.org%2F' ``` 2. **URL_DECODE** - Syntax: url_decode(value) - Arguments: value: a URL encoded - Returns: decodes a string in 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8), will be null if input is null. - Examples: ``` value = 'https%3A%2F%2Fflink.apache.org%2F' SQL: url_decode(value) TableAPI: value.urlDecode() output: 'https://flink.apache.org/' ``` ## Verifying this change - This change added tests in UrlFunctionITCase. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35322) PubSub Connector Weekly build fails
[ https://issues.apache.org/jira/browse/FLINK-35322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845682#comment-17845682 ] Ahmed Hamdy commented on FLINK-35322: - [~Sergey Nuyanzin] yes, let's keep it till after the release as we are not sure about the fix version at the moment. Also this is not a blocker for the ongoing release since the weekly tests are only triggered from main branch so hence there should be no issue from the current release in my opinion. We just need to follow up with a ticket to re-enable 1.19 tests for 3.1 after the release. > PubSub Connector Weekly build fails > > > Key: FLINK-35322 > URL: https://issues.apache.org/jira/browse/FLINK-35322 > Project: Flink > Issue Type: Bug > Components: Connectors / Google Cloud PubSub >Affects Versions: 3.1.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available, test-stability > > Weekly builds for GCP pubSub connector is failing for 1.19 due to compilation > error in tests. > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8768752932/job/24063472769 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8863605354 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8954270618 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35322) PubSub Connector Weekly build fails
[ https://issues.apache.org/jira/browse/FLINK-35322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845676#comment-17845676 ] Sergey Nuyanzin commented on FLINK-35322: - Currently not sure whether we can close this task, since 3.1.0 is under voting phase and in case it passes this issue will not be a part of 3.1.0... let's keep it open so far... or, WDYT, [~danny.cranmer] since you are a RM for this release > PubSub Connector Weekly build fails > > > Key: FLINK-35322 > URL: https://issues.apache.org/jira/browse/FLINK-35322 > Project: Flink > Issue Type: Bug > Components: Connectors / Google Cloud PubSub >Affects Versions: 3.1.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available, test-stability > > Weekly builds for GCP pubSub connector is failing for 1.19 due to compilation > error in tests. > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8768752932/job/24063472769 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8863605354 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8954270618 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35322) PubSub Connector Weekly build fails
[ https://issues.apache.org/jira/browse/FLINK-35322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845675#comment-17845675 ] Sergey Nuyanzin commented on FLINK-35322: - Merged as [725f3d66e8065e457acab332216a63184db16e0b|https://github.com/apache/flink-connector-gcp-pubsub/commit/725f3d66e8065e457acab332216a63184db16e0b] Thanks for the fix [~chalixar] > PubSub Connector Weekly build fails > > > Key: FLINK-35322 > URL: https://issues.apache.org/jira/browse/FLINK-35322 > Project: Flink > Issue Type: Bug > Components: Connectors / Google Cloud PubSub >Affects Versions: 3.1.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available, test-stability > > Weekly builds for GCP pubSub connector is failing for 1.19 due to compilation > error in tests. > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8768752932/job/24063472769 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8863605354 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8954270618 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35322][Connectors/Google PubSub] Remove weekly tests of 1.19 for unsupporting version v3.0 [flink-connector-gcp-pubsub]
boring-cyborg[bot] commented on PR #26: URL: https://github.com/apache/flink-connector-gcp-pubsub/pull/26#issuecomment-2106180994 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35322][Connectors/Google PubSub] Remove weekly tests of 1.19 for unsupporting version v3.0 [flink-connector-gcp-pubsub]
snuyanzin merged PR #26: URL: https://github.com/apache/flink-connector-gcp-pubsub/pull/26 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35322) PubSub Connector Weekly build fails
[ https://issues.apache.org/jira/browse/FLINK-35322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin reassigned FLINK-35322: --- Assignee: Ahmed Hamdy > PubSub Connector Weekly build fails > > > Key: FLINK-35322 > URL: https://issues.apache.org/jira/browse/FLINK-35322 > Project: Flink > Issue Type: Bug > Components: Connectors / Google Cloud PubSub >Affects Versions: 3.1.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available, test-stability > > Weekly builds for GCP pubSub connector is failing for 1.19 due to compilation > error in tests. > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8768752932/job/24063472769 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8863605354 > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8954270618 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35322][Connectors/Google PubSub] Remove weekly tests of 1.19 for unsupporting version v3.0 [flink-connector-gcp-pubsub]
vahmed-hamdy commented on PR #26: URL: https://github.com/apache/flink-connector-gcp-pubsub/pull/26#issuecomment-2106175939 @snuyanzin yes, I added the rub link to the PR description for the same [commit](https://github.com/vahmed-hamdy/flink-connector-gcp-pubsub/commit/02344066377b396fd9265c9ab0fd792e0b1f50b9) changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35305](https://issues.apache.org/jira/browse/FLINK-35305)Amazon SQS Sink Connector [flink-connector-aws]
vahmed-hamdy commented on code in PR #141:
URL:
https://github.com/apache/flink-connector-aws/pull/141#discussion_r1597507484
##
flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/testutils/SqsTestUtils.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.sqs.sink.testutils;
+
+import org.apache.flink.connector.aws.testutils.AWSServicesTestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import software.amazon.awssdk.http.SdkHttpClient;
+import software.amazon.awssdk.services.sqs.SqsClient;
+import software.amazon.awssdk.utils.ImmutableMap;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A set of static methods that can be used to call common AWS services on the
Localstack container.
+ */
+public class SqsTestUtils {
+
+private static final ObjectMapper MAPPER = createObjectMapper();
+
+public static SqsClient createSqsClient(String endpoint, SdkHttpClient
httpClient) {
+return AWSServicesTestUtils.createAwsSyncClient(endpoint, httpClient,
SqsClient.builder());
Review Comment:
nit: Is there a reason this is async as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-33859) Support OpenSearch v2
[ https://issues.apache.org/jira/browse/FLINK-33859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin updated FLINK-33859: Affects Version/s: (was: opensearch-1.2.0) > Support OpenSearch v2 > - > > Key: FLINK-33859 > URL: https://issues.apache.org/jira/browse/FLINK-33859 > Project: Flink > Issue Type: Improvement > Components: Connectors / Opensearch >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > > The main issue is that in OpenSearch v2 there were several breaking changes > like > [https://github.com/opensearch-project/OpenSearch/pull/9082] > [https://github.com/opensearch-project/OpenSearch/pull/5902] > which made current connector version failing while communicating with v2 > > Also it would make sense to add integration and e2e tests to test against v2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33859) Support OpenSearch v2
[ https://issues.apache.org/jira/browse/FLINK-33859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin updated FLINK-33859: Affects Version/s: opensearch-1.1.0 > Support OpenSearch v2 > - > > Key: FLINK-33859 > URL: https://issues.apache.org/jira/browse/FLINK-33859 > Project: Flink > Issue Type: Improvement > Components: Connectors / Opensearch >Affects Versions: opensearch-1.1.0 >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > > The main issue is that in OpenSearch v2 there were several breaking changes > like > [https://github.com/opensearch-project/OpenSearch/pull/9082] > [https://github.com/opensearch-project/OpenSearch/pull/5902] > which made current connector version failing while communicating with v2 > > Also it would make sense to add integration and e2e tests to test against v2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
