Re: [PR] [FLINK-34410][ci] Disables the nightly trigger workflow for forks [flink]
flinkbot commented on PR #24291: URL: https://github.com/apache/flink/pull/24291#issuecomment-1933536761 ## CI report: * e46a0011b4cacac0b793307c4e58b17dbb109a41 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34267][CI] Update miniconda install script to fix build on MacOS [flink-connector-shared-utils]
gaborgsomogyi commented on PR #34: URL: https://github.com/apache/flink-connector-shared-utils/pull/34#issuecomment-1933528414 @pvary do you remember why we added the old conda version here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34410) Disable nightly trigger in forks
[
https://issues.apache.org/jira/browse/FLINK-34410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-34410:
--
Description:
We can disable the automatic triggering of the nightly trigger workflow in fork
(see [GHA
docs|https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions]s:
{code}
if: github.repository == 'octo-org/octo-repo-prod'
{code}
No backport is needed because the schedule triggers will on fire for {{master}}.
was:
We can disable the automatic triggering of the nightly trigger workflow in fork
(see [GHA
docs|https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions]s:
{code}
if: github.repository == 'octo-org/octo-repo-prod'
{code}
> Disable nightly trigger in forks
>
>
> Key: FLINK-34410
> URL: https://issues.apache.org/jira/browse/FLINK-34410
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System / CI
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> We can disable the automatic triggering of the nightly trigger workflow in
> fork (see [GHA
> docs|https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions]s:
> {code}
> if: github.repository == 'octo-org/octo-repo-prod'
> {code}
> No backport is needed because the schedule triggers will on fire for
> {{master}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34410) Disable nightly trigger in forks
[
https://issues.apache.org/jira/browse/FLINK-34410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-34410:
---
Labels: pull-request-available (was: )
> Disable nightly trigger in forks
>
>
> Key: FLINK-34410
> URL: https://issues.apache.org/jira/browse/FLINK-34410
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System / CI
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> We can disable the automatic triggering of the nightly trigger workflow in
> fork (see [GHA
> docs|https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions]s:
> {code}
> if: github.repository == 'octo-org/octo-repo-prod'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-34410][ci] Disables the nightly trigger workflow for forks [flink]
XComp opened a new pull request, #24291: URL: https://github.com/apache/flink/pull/24291 ## What is the purpose of the change Disables the nightly trigger for the forks. The nightly builds can be still triggered manually through the [`workflow_dispatch` trigger of the nightly workflow](https://github.com/apache/flink/blob/master/.github/workflows/nightly.yml#L22) ## Brief change log * Added repo check to trigger workflow ## Verifying this change * [Test run](https://github.com/XComp/flink/actions/runs/7826269858) in `XComp/flink` fork ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34267][CI] Update miniconda install script to fix build on MacOS [flink-connector-shared-utils]
gaborgsomogyi commented on PR #34: URL: https://github.com/apache/flink-connector-shared-utils/pull/34#issuecomment-1933522906 I've just taken a small look at the release date of the mentioned version and seems like it's released on `2019-10-25`. Is it possible to use a more recent version to catch up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-34410) Disable nightly trigger in forks
[
https://issues.apache.org/jira/browse/FLINK-34410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-34410:
-
Assignee: Matthias Pohl
> Disable nightly trigger in forks
>
>
> Key: FLINK-34410
> URL: https://issues.apache.org/jira/browse/FLINK-34410
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System / CI
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> We can disable the automatic triggering of the nightly trigger workflow in
> fork (see [GHA
> docs|https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions]s:
> {code}
> if: github.repository == 'octo-org/octo-repo-prod'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33936][table] Outputting Identical Results in Mini-Batch Aggregation with Set TTL [flink]
flinkbot commented on PR #24290: URL: https://github.com/apache/flink/pull/24290#issuecomment-1933490545 ## CI report: * 096179ded432aefd53274cc26402c82a274b4865 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34410) Disable nightly trigger in forks
Matthias Pohl created FLINK-34410:
-
Summary: Disable nightly trigger in forks
Key: FLINK-34410
URL: https://issues.apache.org/jira/browse/FLINK-34410
Project: Flink
Issue Type: Technical Debt
Components: Build System / CI
Affects Versions: 1.20.0
Reporter: Matthias Pohl
We can disable the automatic triggering of the nightly trigger workflow in fork
(see [GHA
docs|https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions]s:
{code}
if: github.repository == 'octo-org/octo-repo-prod'
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33936) Outputting Identical Results in Mini-Batch Aggregation with Set TTL
[
https://issues.apache.org/jira/browse/FLINK-33936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33936:
---
Labels: pull-request-available (was: )
> Outputting Identical Results in Mini-Batch Aggregation with Set TTL
> ---
>
> Key: FLINK-33936
> URL: https://issues.apache.org/jira/browse/FLINK-33936
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.18.0
>Reporter: Feng Jin
>Priority: Major
> Labels: pull-request-available
>
> If mini-batch is enabled currently, and if the aggregated result is the same
> as the previous output, this current aggregation result will not be sent
> downstream. This will cause downstream nodes to not receive updated data. If
> there is a TTL set for states at this time, the TTL of downstream will not be
> updated either.
> The specific logic is as follows.
> https://github.com/apache/flink/blob/a18c0cd3f0cdfd7e0acb53283f40cd2033a86472/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/MiniBatchGroupAggFunction.java#L224
> {code:java}
> if (!equaliser.equals(prevAggValue, newAggValue)) {
> // new row is not same with prev row
> if (generateUpdateBefore) {
> // prepare UPDATE_BEFORE message for previous row
> resultRow
> .replace(currentKey, prevAggValue)
> .setRowKind(RowKind.UPDATE_BEFORE);
> out.collect(resultRow);
> }
> // prepare UPDATE_AFTER message for new row
> resultRow.replace(currentKey,
> newAggValue).setRowKind(RowKind.UPDATE_AFTER);
> out.collect(resultRow);
> }
> // new row is same with prev row, no need to output
> {code}
> When mini-batch is not enabled, even if the aggregation result of this time
> is the same as last time, new results will still be sent if TTL is set.
> https://github.com/apache/flink/blob/e9353319ad625baa5b2c20fa709ab5b23f83c0f4/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/GroupAggFunction.java#L170
> {code:java}
> if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue,
> newAggValue)) {
> // newRow is the same as before and state cleaning is not
> enabled.
> // We do not emit retraction and acc message.
> // If state cleaning is enabled, we have to emit messages
> to prevent too early
> // state eviction of downstream operators.
> return;
> } else {
> // retract previous result
> if (generateUpdateBefore) {
> // prepare UPDATE_BEFORE message for previous row
> resultRow
> .replace(currentKey, prevAggValue)
> .setRowKind(RowKind.UPDATE_BEFORE);
> out.collect(resultRow);
> }
> // prepare UPDATE_AFTER message for new row
> resultRow.replace(currentKey,
> newAggValue).setRowKind(RowKind.UPDATE_AFTER);
> }
> {code}
> Therefore, based on the consideration of TTL scenarios, I believe that when
> mini-batch aggregation is enabled, new results should also output when the
> aggregated result is the same as the previous one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-33936][table] Outputting Identical Results in Mini-Batch Aggregation with Set TTL [flink]
hackergin opened a new pull request, #24290: URL: https://github.com/apache/flink/pull/24290 ## What is the purpose of the change *Outputting Identical Results in Mini-Batch Aggregation with Set TTL* ## Brief change log - Outputting Identical Results in Mini-Batch Aggregation with Set TTL including `MiniBatchGlobalGroupAggFunction` and `MiniBatchGroupAggFunction` ## Verifying this change This change added tests and can be verified as follows: - *Extended unit test case in GroupAggregateHarnessTest#testAggregateWithRetraction that the aggregation results are the last result.* - *Add the unit test GroupAggregateHarnessTest#testGlobalAggregateWithRetraction including MiniBatchGlobalGroupAggFunction* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33936) Outputting Identical Results in Mini-Batch Aggregation with Set TTL
[
https://issues.apache.org/jira/browse/FLINK-33936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Feng Jin updated FLINK-33936:
-
Summary: Outputting Identical Results in Mini-Batch Aggregation with Set
TTL (was: The aggregation of mini-batches should output the result even if the
result is the same as before when TTL is configured.)
> Outputting Identical Results in Mini-Batch Aggregation with Set TTL
> ---
>
> Key: FLINK-33936
> URL: https://issues.apache.org/jira/browse/FLINK-33936
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.18.0
>Reporter: Feng Jin
>Priority: Major
>
> If mini-batch is enabled currently, and if the aggregated result is the same
> as the previous output, this current aggregation result will not be sent
> downstream. This will cause downstream nodes to not receive updated data. If
> there is a TTL set for states at this time, the TTL of downstream will not be
> updated either.
> The specific logic is as follows.
> https://github.com/apache/flink/blob/a18c0cd3f0cdfd7e0acb53283f40cd2033a86472/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/MiniBatchGroupAggFunction.java#L224
> {code:java}
> if (!equaliser.equals(prevAggValue, newAggValue)) {
> // new row is not same with prev row
> if (generateUpdateBefore) {
> // prepare UPDATE_BEFORE message for previous row
> resultRow
> .replace(currentKey, prevAggValue)
> .setRowKind(RowKind.UPDATE_BEFORE);
> out.collect(resultRow);
> }
> // prepare UPDATE_AFTER message for new row
> resultRow.replace(currentKey,
> newAggValue).setRowKind(RowKind.UPDATE_AFTER);
> out.collect(resultRow);
> }
> // new row is same with prev row, no need to output
> {code}
> When mini-batch is not enabled, even if the aggregation result of this time
> is the same as last time, new results will still be sent if TTL is set.
> https://github.com/apache/flink/blob/e9353319ad625baa5b2c20fa709ab5b23f83c0f4/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/GroupAggFunction.java#L170
> {code:java}
> if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue,
> newAggValue)) {
> // newRow is the same as before and state cleaning is not
> enabled.
> // We do not emit retraction and acc message.
> // If state cleaning is enabled, we have to emit messages
> to prevent too early
> // state eviction of downstream operators.
> return;
> } else {
> // retract previous result
> if (generateUpdateBefore) {
> // prepare UPDATE_BEFORE message for previous row
> resultRow
> .replace(currentKey, prevAggValue)
> .setRowKind(RowKind.UPDATE_BEFORE);
> out.collect(resultRow);
> }
> // prepare UPDATE_AFTER message for new row
> resultRow.replace(currentKey,
> newAggValue).setRowKind(RowKind.UPDATE_AFTER);
> }
> {code}
> Therefore, based on the consideration of TTL scenarios, I believe that when
> mini-batch aggregation is enabled, new results should also output when the
> aggregated result is the same as the previous one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34152] Tune heap memory of autoscaled jobs [flink-kubernetes-operator]
1996fanrui commented on code in PR #762:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/762#discussion_r1482469765
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/utils/MemoryTuningUtils.java:
##
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.utils;
+
+import org.apache.flink.autoscaler.JobAutoScalerContext;
+import org.apache.flink.autoscaler.config.AutoScalerOptions;
+import org.apache.flink.autoscaler.event.AutoScalerEventHandler;
+import org.apache.flink.autoscaler.metrics.EvaluatedMetrics;
+import org.apache.flink.autoscaler.metrics.ScalingMetric;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.configuration.UnmodifiableConfiguration;
+import org.apache.flink.runtime.util.config.memory.CommonProcessMemorySpec;
+import
org.apache.flink.runtime.util.config.memory.JvmMetaspaceAndOverheadOptions;
+import org.apache.flink.runtime.util.config.memory.ProcessMemoryOptions;
+import org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils;
+import
org.apache.flink.runtime.util.config.memory.taskmanager.TaskExecutorFlinkMemory;
+import
org.apache.flink.runtime.util.config.memory.taskmanager.TaskExecutorFlinkMemoryUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Map;
+
+/** Tunes the TaskManager memory. */
+public class MemoryTuningUtils {
+
+private static final Logger LOG =
LoggerFactory.getLogger(MemoryTuningUtils.class);
+public static final ProcessMemoryUtils
FLINK_MEMORY_UTILS =
+new ProcessMemoryUtils<>(getMemoryOptions(), new
TaskExecutorFlinkMemoryUtils());
+
+private static final Configuration EMPTY_CONFIG = new Configuration();
+
+/**
+ * Emits a Configuration which contains overrides for the current
configuration. We are not
+ * modifying the config directly, but we are emitting a new configuration
which contains any
+ * overrides. This config is persisted separately and applied by the
autoscaler. That way we can
+ * clear any applied overrides if auto-tuning is disabled.
+ */
+public static Configuration tuneTaskManagerHeapMemory(
+JobAutoScalerContext context,
+EvaluatedMetrics evaluatedMetrics,
+AutoScalerEventHandler eventHandler) {
+
+// Please note that this config is the original configuration created
from the user spec.
+// It does not contain any already applied overrides.
+var config = new UnmodifiableConfiguration(context.getConfiguration());
+
+// Gather original memory configuration from the user spec
+CommonProcessMemorySpec memSpecs;
+try {
+memSpecs = FLINK_MEMORY_UTILS.memoryProcessSpecFromConfig(config);
+} catch (IllegalConfigurationException e) {
+LOG.warn("Current memory configuration is not valid. Aborting
memory tuning.");
+return EMPTY_CONFIG;
+}
+
+var maxHeapSize = memSpecs.getFlinkMemory().getJvmHeapMemorySize();
+LOG.info("Current configured heap size: {}", maxHeapSize);
+
+MemorySize avgHeapSize = getAverageMemorySize(evaluatedMetrics);
+
+// Apply min/max heap size limits
+MemorySize newHeapSize =
+new MemorySize(
+Math.min(
+// Upper limit is the original max heap size
in the spec
+maxHeapSize.getBytes(),
+Math.max(
+// Lower limit is the minimum
configured heap size
+
config.get(AutoScalerOptions.MEMORY_TUNING_MIN_HEAP)
+.getBytes(),
+avgHeapSize.getBytes(;
+LOG.info("New TM heap memory {}",
Re: [PR] [FLINK-34152] Tune heap memory of autoscaled jobs [flink-kubernetes-operator]
1996fanrui commented on code in PR #762:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/762#discussion_r1482469765
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/utils/MemoryTuningUtils.java:
##
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.utils;
+
+import org.apache.flink.autoscaler.JobAutoScalerContext;
+import org.apache.flink.autoscaler.config.AutoScalerOptions;
+import org.apache.flink.autoscaler.event.AutoScalerEventHandler;
+import org.apache.flink.autoscaler.metrics.EvaluatedMetrics;
+import org.apache.flink.autoscaler.metrics.ScalingMetric;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.configuration.UnmodifiableConfiguration;
+import org.apache.flink.runtime.util.config.memory.CommonProcessMemorySpec;
+import
org.apache.flink.runtime.util.config.memory.JvmMetaspaceAndOverheadOptions;
+import org.apache.flink.runtime.util.config.memory.ProcessMemoryOptions;
+import org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils;
+import
org.apache.flink.runtime.util.config.memory.taskmanager.TaskExecutorFlinkMemory;
+import
org.apache.flink.runtime.util.config.memory.taskmanager.TaskExecutorFlinkMemoryUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Map;
+
+/** Tunes the TaskManager memory. */
+public class MemoryTuningUtils {
+
+private static final Logger LOG =
LoggerFactory.getLogger(MemoryTuningUtils.class);
+public static final ProcessMemoryUtils
FLINK_MEMORY_UTILS =
+new ProcessMemoryUtils<>(getMemoryOptions(), new
TaskExecutorFlinkMemoryUtils());
+
+private static final Configuration EMPTY_CONFIG = new Configuration();
+
+/**
+ * Emits a Configuration which contains overrides for the current
configuration. We are not
+ * modifying the config directly, but we are emitting a new configuration
which contains any
+ * overrides. This config is persisted separately and applied by the
autoscaler. That way we can
+ * clear any applied overrides if auto-tuning is disabled.
+ */
+public static Configuration tuneTaskManagerHeapMemory(
+JobAutoScalerContext context,
+EvaluatedMetrics evaluatedMetrics,
+AutoScalerEventHandler eventHandler) {
+
+// Please note that this config is the original configuration created
from the user spec.
+// It does not contain any already applied overrides.
+var config = new UnmodifiableConfiguration(context.getConfiguration());
+
+// Gather original memory configuration from the user spec
+CommonProcessMemorySpec memSpecs;
+try {
+memSpecs = FLINK_MEMORY_UTILS.memoryProcessSpecFromConfig(config);
+} catch (IllegalConfigurationException e) {
+LOG.warn("Current memory configuration is not valid. Aborting
memory tuning.");
+return EMPTY_CONFIG;
+}
+
+var maxHeapSize = memSpecs.getFlinkMemory().getJvmHeapMemorySize();
+LOG.info("Current configured heap size: {}", maxHeapSize);
+
+MemorySize avgHeapSize = getAverageMemorySize(evaluatedMetrics);
+
+// Apply min/max heap size limits
+MemorySize newHeapSize =
+new MemorySize(
+Math.min(
+// Upper limit is the original max heap size
in the spec
+maxHeapSize.getBytes(),
+Math.max(
+// Lower limit is the minimum
configured heap size
+
config.get(AutoScalerOptions.MEMORY_TUNING_MIN_HEAP)
+.getBytes(),
+avgHeapSize.getBytes(;
+LOG.info("New TM heap memory {}",
[jira] [Comment Edited] (FLINK-34289) Release Testing Instructions: Verify FLINK-33695 Introduce TraceReporter and use it to create checkpointing and recovery traces
[
https://issues.apache.org/jira/browse/FLINK-34289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815532#comment-17815532
]
Piotr Nowojski edited comment on FLINK-34289 at 2/8/24 6:33 AM:
!Screenshot 2024-02-08 at 07.22.19.png|width=600!
This part was confusing to me. My thought process was:
# Ok first box says, I should create a *new* ticket.
# But following the arrow, I should *remove* "Instructions". That's slightly
suggested to rename the ticket.
# The truly confusing part. The box on the right, shows an arrow from "X
Instructions: Verify" to "X: Verify". I've interpreted it as a subtask issue
first, supporting "create new ticket" interpretation. But I couldn't create
sub-task. So maybe that arrow is supposed to represent action of renaming thet
ticket?
# I've checked that in FLINK-34285 someone else has already renamed ticket by
removing "Instructions" so that further supported my interpretation that I
should rename it as well.
Rephrasing the instruction to clarify the meaning of {{remove "Instructions"}}
and labeling the arrow between "Instructions" and non "Instructions" tickets,
to state that for example it represents a linked ticket would help me avoid
this mistake.
was (Author: pnowojski):
!Screenshot 2024-02-08 at 07.22.19.png!
This part was confusing to me. My thought process was:
# Ok first box says, I should create a *new* ticket.
# But following the arrow, I should *remove* "Instructions". That's slightly
suggested to rename the ticket.
# The truly confusing part. The box on the right, shows an arrow from "X
Instructions: Verify" to "X: Verify". I've interpreted it as a subtask issue
first, supporting "create new ticket" interpretation. But I couldn't create
sub-task. So maybe that arrow is supposed to represent action of renaming thet
ticket?
# I've checked that in FLINK-34285 someone else has already renamed ticket by
removing "Instructions" so that further supported my interpretation that I
should rename it as well.
Rephrasing the instruction to clarify the meaning of {{remove "Instructions"}}
and labeling the arrow between "Instructions" and non "Instructions" tickets,
to state that for example it represents a linked ticket would help me avoid
this mistake.
> Release Testing Instructions: Verify FLINK-33695 Introduce TraceReporter and
> use it to create checkpointing and recovery traces
>
>
> Key: FLINK-34289
> URL: https://issues.apache.org/jira/browse/FLINK-34289
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Metrics
>Affects Versions: 1.19.0
>Reporter: lincoln lee
>Assignee: Piotr Nowojski
>Priority: Blocker
> Fix For: 1.19.0
>
> Attachments: Screenshot 2024-02-08 at 07.22.19.png, screenshot-1.png
>
>
> This ticket covers testing three related features: FLINK-33695, FLINK-33735
> and FLINK-33696.
> Instructions:
> # Configure Flink to use
> [Slf4jTraceReporter|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/trace_reporters/#slf4j]
> and with enabled *INFO* level logging (can be to console or to a file,
> doesn't matter).
> # Start a streaming job with enabled checkpointing.
> # Let it run for a couple of checkpoints.
> # Verify presence of a single *JobInitialization* [1] trace logged just after
> job start up.
> # Verify presence of a couple of *Checkpoint* [1] traces logged after each
> successful or failed checkpoint.
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/traces/#checkpointing-and-initialization
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34289) Release Testing Instructions: Verify FLINK-33695 Introduce TraceReporter and use it to create checkpointing and recovery traces
[
https://issues.apache.org/jira/browse/FLINK-34289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815532#comment-17815532
]
Piotr Nowojski commented on FLINK-34289:
!Screenshot 2024-02-08 at 07.22.19.png!
This part was confusing to me. My thought process was:
# Ok first box says, I should create a *new* ticket.
# But following the arrow, I should *remove* "Instructions". That's slightly
suggested to rename the ticket.
# The truly confusing part. The box on the right, shows an arrow from "X
Instructions: Verify" to "X: Verify". I've interpreted it as a subtask issue
first, supporting "create new ticket" interpretation. But I couldn't create
sub-task. So maybe that arrow is supposed to represent action of renaming thet
ticket?
# I've checked that in FLINK-34285 someone else has already renamed ticket by
removing "Instructions" so that further supported my interpretation that I
should rename it as well.
Rephrasing the instruction to clarify the meaning of {{remove "Instructions"}}
and labeling the arrow between "Instructions" and non "Instructions" tickets,
to state that for example it represents a linked ticket would help me avoid
this mistake.
> Release Testing Instructions: Verify FLINK-33695 Introduce TraceReporter and
> use it to create checkpointing and recovery traces
>
>
> Key: FLINK-34289
> URL: https://issues.apache.org/jira/browse/FLINK-34289
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Metrics
>Affects Versions: 1.19.0
>Reporter: lincoln lee
>Assignee: Piotr Nowojski
>Priority: Blocker
> Fix For: 1.19.0
>
> Attachments: Screenshot 2024-02-08 at 07.22.19.png, screenshot-1.png
>
>
> This ticket covers testing three related features: FLINK-33695, FLINK-33735
> and FLINK-33696.
> Instructions:
> # Configure Flink to use
> [Slf4jTraceReporter|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/trace_reporters/#slf4j]
> and with enabled *INFO* level logging (can be to console or to a file,
> doesn't matter).
> # Start a streaming job with enabled checkpointing.
> # Let it run for a couple of checkpoints.
> # Verify presence of a single *JobInitialization* [1] trace logged just after
> job start up.
> # Verify presence of a couple of *Checkpoint* [1] traces logged after each
> successful or failed checkpoint.
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/traces/#checkpointing-and-initialization
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34405) RightOuterJoinTaskTest#testCancelOuterJoinTaskWhileSort2 fails due to an interruption of the RightOuterJoinDriver#prepare method
[
https://issues.apache.org/jira/browse/FLINK-34405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815531#comment-17815531
]
Rui Fan commented on FLINK-34405:
-
org.apache.flink.runtime.operators.RightOuterJoinTaskTest.testCancelOuterJoinTaskWhileSort1
fails as well.
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57395=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=9113
> RightOuterJoinTaskTest#testCancelOuterJoinTaskWhileSort2 fails due to an
> interruption of the RightOuterJoinDriver#prepare method
>
>
> Key: FLINK-34405
> URL: https://issues.apache.org/jira/browse/FLINK-34405
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.17.2, 1.19.0, 1.18.1, 1.20.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: starter, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57357=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=be5a4b15-4b23-56b1-7582-795f58a645a2=9027
> {code}
> Feb 07 03:20:16 03:20:16.223 [ERROR] Failures:
> Feb 07 03:20:16 03:20:16.223 [ERROR]
> org.apache.flink.runtime.operators.RightOuterJoinTaskTest.testCancelOuterJoinTaskWhileSort2
> Feb 07 03:20:16 03:20:16.223 [ERROR] Run 1:
> RightOuterJoinTaskTest>AbstractOuterJoinTaskTest.testCancelOuterJoinTaskWhileSort2:435
>
> Feb 07 03:20:16 expected:
> Feb 07 03:20:16 null
> Feb 07 03:20:16 but was:
> Feb 07 03:20:16 java.lang.Exception: The data preparation caused an error:
> Interrupted
> Feb 07 03:20:16 at
> org.apache.flink.runtime.operators.testutils.BinaryOperatorTestBase.testDriverInternal(BinaryOperatorTestBase.java:209)
> Feb 07 03:20:16 at
> org.apache.flink.runtime.operators.testutils.BinaryOperatorTestBase.testDriver(BinaryOperatorTestBase.java:189)
> Feb 07 03:20:16 at
> org.apache.flink.runtime.operators.AbstractOuterJoinTaskTest.access$100(AbstractOuterJoinTaskTest.java:48)
> Feb 07 03:20:16 ...(1 remaining lines not displayed - this can be
> changed with Assertions.setMaxStackTraceElementsDisplayed)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34152] Tune heap memory of autoscaled jobs [flink-kubernetes-operator]
1996fanrui commented on code in PR #762:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/762#discussion_r1482361348
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/config/AutoScalerOptions.java:
##
@@ -250,6 +251,31 @@ private static ConfigOptions.OptionBuilder
autoScalerConfig(String key) {
.withDescription(
"Max allowed percentage of heap usage during
scaling operations. Autoscaling will be paused if the heap usage exceeds this
threshold.");
+public static final ConfigOption MEMORY_TUNING_ENABLED =
+autoScalerConfig("memory.tuning.enabled")
+.booleanType()
+.defaultValue(false)
+
.withFallbackKeys(oldOperatorConfigKey("memory.tuning.enabled"))
+.withDescription(
+"If enabled, the initial amount of memory
specified for TaskManagers will be reduced according to the observed needs.");
+
+public static final ConfigOption MEMORY_TUNING_MIN_HEAP =
+autoScalerConfig("memory.tuning.heap.min")
+.memoryType()
+.defaultValue(MemorySize.ofMebiBytes(512L))
+
.withFallbackKeys(oldOperatorConfigKey("memory.tuning.heap.min"))
+.withDescription(
+"The minimum amount of TaskManager memory, if
memory tuning is enabled.");
+
+public static final ConfigOption
MEMORY_TUNING_TRANSFER_HEAP_TO_MANAGED =
+autoScalerConfig("memory.tuning.heap.transfer-to-managed")
+.booleanType()
+.defaultValue(false)
+.withFallbackKeys(
+
oldOperatorConfigKey("memory.tuning.heap.transfer-to-managed"))
+.withDescription(
+"If enabled, any reduction of heap memory will
increase the managed memory for RocksDB. RocksDB needs to be enabled.");
Review Comment:
```suggestion
"If enabled, any reduction of heap memory will
increase the managed memory for RocksDB when rocksdb state backend is used.");
```
How about this?
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerContext.java:
##
@@ -52,19 +55,25 @@ public class JobAutoScalerContext {
@Nullable @Getter private final JobStatus jobStatus;
+/** The configuration derived from the current spec. */
Review Comment:
```suggestion
/** The configuration derived from the user-specified spec instead of
actual spec. */
```
##
flink-autoscaler/src/test/java/org/apache/flink/autoscaler/utils/ResourceCheckUtilsTest.java:
##
@@ -32,7 +32,7 @@
class ResourceCheckUtilsTest {
@Test
-void testEstimateNumTaskSlotsAfterRescale() {
+void estimateNumTaskSlotsAfterRescale() {
Review Comment:
Why remove the test prefix? In general, the prefix of all test methods are
`test`.
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkResourceContext.java:
##
@@ -65,7 +65,7 @@ public KubernetesJobAutoScalerContext
getJobAutoScalerContext() {
}
private KubernetesJobAutoScalerContext createJobAutoScalerContext() {
-Configuration conf = new Configuration(getObserveConfig());
+Configuration conf = new
Configuration(getDeployConfig(resource.getSpec()));
Review Comment:
Using `getDeployConfig` here makes sense to me. And I left a comment about
the code comment of `JobAutoScalerContext#configuration`.
It's better to remind others it's user-specified spec instead of actual spec.
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/KubernetesScalingRealizer.java:
##
@@ -43,6 +53,34 @@ public void realize(
getOverrideString(context, parallelismOverrides));
}
+@Override
+public void realizeConfigOverrides(
+KubernetesJobAutoScalerContext context, Configuration
configOverrides) {
+if (!(context.getResource() instanceof FlinkDeployment)) {
+// We can't adjust the configuration of non-job deployments.
+return;
+}
+FlinkDeployment flinkDeployment = ((FlinkDeployment)
context.getResource());
+// Apply config overrides
+
flinkDeployment.getSpec().getFlinkConfiguration().putAll(configOverrides.toMap());
+
+// Update total memory in spec
+var totalMemoryOverride =
MemoryTuningUtils.getTotalMemory(configOverrides, context);
+if (totalMemoryOverride.compareTo(MemorySize.ZERO) <= 0) {
+LOG.warn("Memory override {} is not valid", totalMemoryOverride);
Review Comment:
nit:
```suggestion
LOG.warn("Total memory override {} is not valid",
totalMemoryOverride);
```
--
[jira] [Updated] (FLINK-34289) Release Testing Instructions: Verify FLINK-33695 Introduce TraceReporter and use it to create checkpointing and recovery traces
[ https://issues.apache.org/jira/browse/FLINK-34289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-34289: --- Attachment: Screenshot 2024-02-08 at 07.22.19.png > Release Testing Instructions: Verify FLINK-33695 Introduce TraceReporter and > use it to create checkpointing and recovery traces > > > Key: FLINK-34289 > URL: https://issues.apache.org/jira/browse/FLINK-34289 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics >Affects Versions: 1.19.0 >Reporter: lincoln lee >Assignee: Piotr Nowojski >Priority: Blocker > Fix For: 1.19.0 > > Attachments: Screenshot 2024-02-08 at 07.22.19.png, screenshot-1.png > > > This ticket covers testing three related features: FLINK-33695, FLINK-33735 > and FLINK-33696. > Instructions: > # Configure Flink to use > [Slf4jTraceReporter|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/trace_reporters/#slf4j] > and with enabled *INFO* level logging (can be to console or to a file, > doesn't matter). > # Start a streaming job with enabled checkpointing. > # Let it run for a couple of checkpoints. > # Verify presence of a single *JobInitialization* [1] trace logged just after > job start up. > # Verify presence of a couple of *Checkpoint* [1] traces logged after each > successful or failed checkpoint. > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/traces/#checkpointing-and-initialization -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34386][state] Add RocksDB bloom filter metrics [flink]
hejufang commented on PR #24274: URL: https://github.com/apache/flink/pull/24274#issuecomment-1933419178 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Draft][bugfix] Move the deserialization of shuffleDescriptor to a separate … [flink]
caodizhou commented on PR #24115: URL: https://github.com/apache/flink/pull/24115#issuecomment-1933323089 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34060][table] Migrate UserDefinedTableAggFunctions to JavaUserDefinedTableAggFunctions [flink]
flinkbot commented on PR #24289: URL: https://github.com/apache/flink/pull/24289#issuecomment-1933319116 ## CI report: * ebc29752d43c83eef334c29498202c888142893f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-33489) LISTAGG with generating partial-final agg will cause wrong result
[
https://issues.apache.org/jira/browse/FLINK-33489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xuyang resolved FLINK-33489.
Resolution: Resolved
> LISTAGG with generating partial-final agg will cause wrong result
> -
>
> Key: FLINK-33489
> URL: https://issues.apache.org/jira/browse/FLINK-33489
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0,
> 1.16.0, 1.17.0, 1.18.0
>Reporter: xuyang
>Assignee: xuyang
>Priority: Major
> Labels: pull-request-available
>
> Adding the following test cases in SplitAggregateITCase will reproduce this
> bug:
>
> {code:java}
> // code placeholder
> @Test
> def testListAggWithDistinctMultiArgs(): Unit = {
> val t1 = tEnv.sqlQuery(s"""
> |SELECT
> | a,
> | LISTAGG(DISTINCT c, '#')
> |FROM T
> |GROUP BY a
> """.stripMargin)
> val sink = new TestingRetractSink
> t1.toRetractStream[Row].addSink(sink)
> env.execute()
> val expected = Map[String, List[String]](
> "1" -> List("Hello 0", "Hello 1"),
> "2" -> List("Hello 0", "Hello 1", "Hello 2", "Hello 3", "Hello 4"),
> "3" -> List("Hello 0", "Hello 1"),
> "4" -> List("Hello 1", "Hello 2", "Hello 3")
> )
> val actualData = sink.getRetractResults.sorted
> println(actualData)
> } {code}
> The `actualData` is `List(1,Hello 0,Hello 1, 2,Hello 2,Hello 4,Hello 3,Hello
> 1,Hello 0, 3,Hello 1,Hello 0, 4,Hello 2,Hello 3,Hello 1)`, and the delimiter
> `#` will be ignored.
> Let's take its plan:
> {code:java}
> // code placeholder
> LegacySink(name=[DataStreamTableSink], fields=[a, EXPR$1])
> +- GroupAggregate(groupBy=[a], partialFinalType=[FINAL], select=[a,
> LISTAGG_RETRACT($f3_0) AS $f1])
> +- Exchange(distribution=[hash[a]])
> +- GroupAggregate(groupBy=[a, $f3, $f4], partialFinalType=[PARTIAL],
> select=[a, $f3, $f4, LISTAGG(DISTINCT c, $f2) AS $f3_0])
> +- Exchange(distribution=[hash[a, $f3, $f4]])
> +- Calc(select=[a, c, _UTF-16LE'#' AS $f2, MOD(HASH_CODE(c),
> 1024) AS $f3, MOD(HASH_CODE(_UTF-16LE'#'), 1024) AS $f4])
> +- MiniBatchAssigner(interval=[1000ms], mode=[ProcTime])
> +- DataStreamScan(table=[[default_catalog,
> default_database, T]], fields=[a, b, c]) {code}
> The final `GroupAggregate` missing the delimiter args, and the default
> delimiter `,` will be used.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34060) Migrate UserDefinedTableAggFunctions to JavaUserDefinedTableAggFunctions
[ https://issues.apache.org/jira/browse/FLINK-34060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34060: --- Labels: pull-request-available (was: ) > Migrate UserDefinedTableAggFunctions to JavaUserDefinedTableAggFunctions > > > Key: FLINK-34060 > URL: https://issues.apache.org/jira/browse/FLINK-34060 > Project: Flink > Issue Type: Technical Debt > Components: Table SQL / Runtime >Affects Versions: 1.19.0 >Reporter: Jane Chan >Assignee: xuyang >Priority: Not a Priority > Labels: pull-request-available > > The issue is discovered when testing FLINK-31788. > The Top3 function emits a tuple of (entry.getKey, entry.getKey) see > [UserDefinedTableAggFunctions.scala#L127|https://github.com/apache/flink/blob/907d0f32126b9f8acfc80f3f4098e71cb37f0e37/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/UserDefinedTableAggFunctions.scala#L127], > which is peculiar. > Meanwhile, consider getting the scala-free goal; it's time to migrate this > class to the `JavaUserDefinedTableAggFunctions`, and revisit the > implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34060][table] Migrate UserDefinedTableAggFunctions to JavaUserDefinedTableAggFunctions [flink]
xuyangzhong opened a new pull request, #24289: URL: https://github.com/apache/flink/pull/24289 ## What is the purpose of the change The Top3 function emits a tuple of (entry.getKey, entry.getKey) see [UserDefinedTableAggFunctions.scala#L127](https://github.com/apache/flink/blob/907d0f32126b9f8acfc80f3f4098e71cb37f0e37/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/UserDefinedTableAggFunctions.scala#L127), which is peculiar. Meanwhile, consider getting the scala-free goal; it's time to migrate this class to the `JavaUserDefinedTableAggFunctions`, and revisit the implementation. ## Brief change log - *Fix peculiar logic about Top3, Top3WithMapViewAccum and Top3WithRetractInputAcc used for test* - *Move all udx in UserDefinedTableAggFunctions to JavaUserDefinedTableAggFunctions* ## Verifying this change Existent tests can cover these changes. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34192] Update to be compatible with updated SinkV2 interfaces [flink-connector-kafka]
Jiabao-Sun commented on PR #84: URL: https://github.com/apache/flink-connector-kafka/pull/84#issuecomment-1933308568 By the way, this PR was blocked by https://github.com/apache/flink/pull/24249 before is that InitContextWrapper does not implement the metadataConsumer (introduced by FLINK-25696) method, but we use it in KafkaSinkITCase, which causes the behavior of metadataConsumer to not take effect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34192] Update to be compatible with updated SinkV2 interfaces [flink-connector-kafka]
Jiabao-Sun commented on PR #84: URL: https://github.com/apache/flink-connector-kafka/pull/84#issuecomment-1933305498 Thanks @mas-chen for the review. The main reason for this PR is in version 1.19, due to the introduction of WriterInitContext by FLINK-33973. In terms of code changes, KafkaSink.createWriter is used to ensure that TestSinkInitContext can correctly create KafkaWriter regardless of whether it inherits from WriterInitContext or Sink.InitContext. https://github.com/apache/flink/blob/1b95b191922829fd8e7a76e5c9d8de68bb57ae7d/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java#L66-L78 https://github.com/apache/flink/pull/24180/files#r1467200868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33634] Add Conditions to Flink CRD's Status field [flink-kubernetes-operator]
lajith2006 commented on PR #749: URL: https://github.com/apache/flink-kubernetes-operator/pull/749#issuecomment-1933293529 > > > > @gyfora yes, I am having JIRA account using which I can login to https://issues.apache.org/jira/projects/FLINK/. > > > > > > > > > okay then can you please tell me the account name? :D > > > > > > account name : **lajithk** > > It seems like you need to create a confluence account (cwiki.apache.org) once you have that I can give you permissions to create a FLIP page I have been checking on to create confluence account , https://cwiki.apache.org/confluence , it says for register go to Log in page, but don't see any option to register there in login page. On further digging noticed some thing like https://cwiki.apache.org/confluence/display/DIRxTRIPLESEC/User+Registration . Is that something I have to follow up ?. or are there any path I can look for registration?. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34390) Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java profiler
[ https://issues.apache.org/jira/browse/FLINK-34390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815489#comment-17815489 ] junzhong qin commented on FLINK-34390: -- Hi [~yunta] I have tested it and it meets expectations. Detailed test results are shown in the following screenshots. h3. 1. Disable rest.profiling.enabled !image-2024-02-08-10-43-27-679.png! h3. 2. Enable rest.profiling.enabled Cannot create another profiling while one is running. (Default profiling duration 30 s) !image-2024-02-08-10-44-55-401.png! 11 snapshots were triggered, but only 10 will be retained in the end. (CPU mode failed because I tested it on my Mac) !image-2024-02-08-10-45-13-951.png! !image-2024-02-08-10-45-31-564.png! > Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java > profiler > -- > > Key: FLINK-34390 > URL: https://issues.apache.org/jira/browse/FLINK-34390 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 >Reporter: Yun Tang >Assignee: junzhong qin >Priority: Major > Labels: release-testing > Fix For: 1.19.0 > > Attachments: image-2024-02-08-10-43-27-679.png, > image-2024-02-08-10-44-55-401.png, image-2024-02-08-10-45-13-951.png, > image-2024-02-08-10-45-31-564.png > > > See https://issues.apache.org/jira/browse/FLINK-34310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34239) Introduce a deep copy method of SerializerConfig for merging with Table configs in org.apache.flink.table.catalog.DataTypeFactoryImpl
[ https://issues.apache.org/jira/browse/FLINK-34239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815486#comment-17815486 ] Fang Yong commented on FLINK-34239: --- [~mallikarjuna] DONE > Introduce a deep copy method of SerializerConfig for merging with Table > configs in org.apache.flink.table.catalog.DataTypeFactoryImpl > -- > > Key: FLINK-34239 > URL: https://issues.apache.org/jira/browse/FLINK-34239 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Affects Versions: 1.19.0 >Reporter: Zhanghao Chen >Assignee: Kumar Mallikarjuna >Priority: Major > > *Problem* > Currently, > org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig > will create a deep-copy of the SerializerConfig and merge Table config into > it. However, the deep copy is done by manully calling the getter and setter > methods of SerializerConfig, and is prone to human errors, e.g. missing > copying a newly added field in SerializerConfig. > *Proposal* > Introduce a deep copy method for SerializerConfig and replace the curr impl > in > org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34239) Introduce a deep copy method of SerializerConfig for merging with Table configs in org.apache.flink.table.catalog.DataTypeFactoryImpl
[ https://issues.apache.org/jira/browse/FLINK-34239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fang Yong reassigned FLINK-34239: - Assignee: Kumar Mallikarjuna > Introduce a deep copy method of SerializerConfig for merging with Table > configs in org.apache.flink.table.catalog.DataTypeFactoryImpl > -- > > Key: FLINK-34239 > URL: https://issues.apache.org/jira/browse/FLINK-34239 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Affects Versions: 1.19.0 >Reporter: Zhanghao Chen >Assignee: Kumar Mallikarjuna >Priority: Major > > *Problem* > Currently, > org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig > will create a deep-copy of the SerializerConfig and merge Table config into > it. However, the deep copy is done by manully calling the getter and setter > methods of SerializerConfig, and is prone to human errors, e.g. missing > copying a newly added field in SerializerConfig. > *Proposal* > Introduce a deep copy method for SerializerConfig and replace the curr impl > in > org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34390) Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java profiler
[ https://issues.apache.org/jira/browse/FLINK-34390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junzhong qin updated FLINK-34390: - Attachment: image-2024-02-08-10-45-13-951.png > Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java > profiler > -- > > Key: FLINK-34390 > URL: https://issues.apache.org/jira/browse/FLINK-34390 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 >Reporter: Yun Tang >Assignee: junzhong qin >Priority: Major > Labels: release-testing > Fix For: 1.19.0 > > Attachments: image-2024-02-08-10-43-27-679.png, > image-2024-02-08-10-44-55-401.png, image-2024-02-08-10-45-13-951.png, > image-2024-02-08-10-45-31-564.png > > > See https://issues.apache.org/jira/browse/FLINK-34310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34390) Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java profiler
[ https://issues.apache.org/jira/browse/FLINK-34390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junzhong qin updated FLINK-34390: - Attachment: image-2024-02-08-10-45-31-564.png > Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java > profiler > -- > > Key: FLINK-34390 > URL: https://issues.apache.org/jira/browse/FLINK-34390 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 >Reporter: Yun Tang >Assignee: junzhong qin >Priority: Major > Labels: release-testing > Fix For: 1.19.0 > > Attachments: image-2024-02-08-10-43-27-679.png, > image-2024-02-08-10-44-55-401.png, image-2024-02-08-10-45-13-951.png, > image-2024-02-08-10-45-31-564.png > > > See https://issues.apache.org/jira/browse/FLINK-34310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34336][test] Fix the bug that AutoRescalingITCase may hang sometimes [flink]
1996fanrui commented on PR #24248: URL: https://github.com/apache/flink/pull/24248#issuecomment-1933278564 https://github.com/apache/flink/pull/24246 is merged so far, I guess the CI of master branch may be stuck. This PR can fix it if CI is stuck on `waitForRunningTasks` of `AutoRescalingITCase`. I start vacation since this afternoon, feel free to merge it if the change make sense. BTW, I will check the github email every day on my vacation, so I still can address the comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34390) Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java profiler
[ https://issues.apache.org/jira/browse/FLINK-34390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junzhong qin updated FLINK-34390: - Attachment: image-2024-02-08-10-44-55-401.png > Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java > profiler > -- > > Key: FLINK-34390 > URL: https://issues.apache.org/jira/browse/FLINK-34390 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 >Reporter: Yun Tang >Assignee: junzhong qin >Priority: Major > Labels: release-testing > Fix For: 1.19.0 > > Attachments: image-2024-02-08-10-43-27-679.png, > image-2024-02-08-10-44-55-401.png > > > See https://issues.apache.org/jira/browse/FLINK-34310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34390) Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java profiler
[ https://issues.apache.org/jira/browse/FLINK-34390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junzhong qin updated FLINK-34390: - Attachment: (was: image-2024-02-08-10-42-34-802.png) > Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java > profiler > -- > > Key: FLINK-34390 > URL: https://issues.apache.org/jira/browse/FLINK-34390 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 >Reporter: Yun Tang >Assignee: junzhong qin >Priority: Major > Labels: release-testing > Fix For: 1.19.0 > > Attachments: image-2024-02-08-10-43-27-679.png > > > See https://issues.apache.org/jira/browse/FLINK-34310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34390) Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java profiler
[ https://issues.apache.org/jira/browse/FLINK-34390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junzhong qin updated FLINK-34390: - Attachment: image-2024-02-08-10-43-27-679.png > Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java > profiler > -- > > Key: FLINK-34390 > URL: https://issues.apache.org/jira/browse/FLINK-34390 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 >Reporter: Yun Tang >Assignee: junzhong qin >Priority: Major > Labels: release-testing > Fix For: 1.19.0 > > Attachments: image-2024-02-08-10-43-27-679.png > > > See https://issues.apache.org/jira/browse/FLINK-34310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34390) Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java profiler
[ https://issues.apache.org/jira/browse/FLINK-34390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junzhong qin updated FLINK-34390: - Attachment: image-2024-02-08-10-42-34-802.png > Release Testing: Verify FLINK-33325 Built-in cross-platform powerful java > profiler > -- > > Key: FLINK-34390 > URL: https://issues.apache.org/jira/browse/FLINK-34390 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 >Reporter: Yun Tang >Assignee: junzhong qin >Priority: Major > Labels: release-testing > Fix For: 1.19.0 > > Attachments: image-2024-02-08-10-42-34-802.png > > > See https://issues.apache.org/jira/browse/FLINK-34310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-28693][table] Fix janino compile failed because the code generated refers the class in table-planner [flink]
xuyangzhong commented on PR #24280: URL: https://github.com/apache/flink/pull/24280#issuecomment-1933276374 Hi, @snuyanzin I think regular IT tests and UT tests cannot help. If some tests must be added, I guess it can be added to the `flink-stream-sql-test` module in the `flink-end-to-end-tests` module. But I'm not sure that this module only depends on table-planner-loader but not table-planner. Meanwhile, we need to mock a source that supports SupportWatermarkPushDown -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33960][Scheduler] Fix the bug that Adaptive Scheduler doesn't respect the lowerBound when one flink job has more than 1 tasks [flink]
flinkbot commented on PR #24288: URL: https://github.com/apache/flink/pull/24288#issuecomment-1933274806 ## CI report: * c34660e4be25e533cf52c1e21c97291cfa11b4b5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33960][Scheduler] Fix the bug that Adaptive Scheduler doesn't respect the lowerBound when one flink job has more than 1 tasks [flink]
flinkbot commented on PR #24287: URL: https://github.com/apache/flink/pull/24287#issuecomment-1933274217 ## CI report: * 437473a4ae810c1542cc3c5628fee0c201228c36 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34348) Release Testing: Verify FLINK-20281 Window aggregation supports changelog stream input
[ https://issues.apache.org/jira/browse/FLINK-34348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815480#comment-17815480 ] xuyang commented on FLINK-34348: Hi, [~hackergin]. Thanks for your detailed testing. Overall, it seems that nothing is missing, regarding the plan test and IT test. > Release Testing: Verify FLINK-20281 Window aggregation supports changelog > stream input > -- > > Key: FLINK-34348 > URL: https://issues.apache.org/jira/browse/FLINK-34348 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: xuyang >Assignee: Feng Jin >Priority: Blocker > Labels: release-testing > Fix For: 1.19.0 > > Attachments: 截屏2024-02-07 16.21.37.png, 截屏2024-02-07 16.21.55.png, > 截屏2024-02-07 16.22.24.png, 截屏2024-02-07 16.23.12.png, 截屏2024-02-07 > 16.23.27.png, 截屏2024-02-07 16.23.38.png, 截屏2024-02-07 16.29.09.png, > 截屏2024-02-07 16.29.21.png, 截屏2024-02-07 16.29.34.png, 截屏2024-02-07 > 16.46.12.png, 截屏2024-02-07 16.46.23.png, 截屏2024-02-07 16.46.37.png, > 截屏2024-02-07 16.53.37.png, 截屏2024-02-07 16.53.47.png, 截屏2024-02-07 > 16.54.01.png, 截屏2024-02-07 16.59.22.png, 截屏2024-02-07 16.59.33.png, > 截屏2024-02-07 16.59.42.png > > > Window TVF aggregation supports changelog stream is ready for testing. User > can add a window tvf aggregation as a down stream after CDC source or some > nodes that will produce cdc records. > Someone can verify this feature with: > # Prepare a mysql table, and insert some data at first. > # Start sql-client and prepare ddl for this mysql table as a cdc source. > # You can verify the plan by `EXPLAIN PLAN_ADVICE` to check if there is a > window aggregate node and the changelog contains "UA" or "UB" or "D" in its > upstream. > # Use different kinds of window tvf to test window tvf aggregation while > updating the source data to check the data correctness. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34200) AutoRescalingITCase#testCheckpointRescalingInKeyedState fails
[
https://issues.apache.org/jira/browse/FLINK-34200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan updated FLINK-34200:
Fix Version/s: 1.19.0
1.20.0
> AutoRescalingITCase#testCheckpointRescalingInKeyedState fails
> -
>
> Key: FLINK-34200
> URL: https://issues.apache.org/jira/browse/FLINK-34200
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Rui Fan
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0, 1.20.0
>
> Attachments: FLINK-34200.failure.log.gz, debug-34200.log
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56601=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=ea7cf968-e585-52cb-e0fc-f48de023a7ca=8200]
> {code:java}
> Jan 19 02:31:53 02:31:53.954 [ERROR] Tests run: 32, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 1050 s <<< FAILURE! -- in
> org.apache.flink.test.checkpointing.AutoRescalingITCase
> Jan 19 02:31:53 02:31:53.954 [ERROR]
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState[backend
> = rocksdb, buffersPerChannel = 2] -- Time elapsed: 59.10 s <<< FAILURE!
> Jan 19 02:31:53 java.lang.AssertionError: expected:<[(0,8000), (0,32000),
> (0,48000), (0,72000), (1,78000), (1,3), (1,54000), (0,2000), (0,1),
> (0,5), (0,66000), (0,74000), (0,82000), (1,8), (1,0), (1,16000),
> (1,24000), (1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000),
> (0,6), (0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000),
> (1,58000), (0,6000), (0,14000), (0,22000), (0,38000), (0,46000), (0,62000),
> (0,7), (1,4000), (1,2), (1,36000), (1,44000)]> but was:<[(0,8000),
> (0,32000), (0,48000), (0,72000), (1,78000), (1,3), (1,54000), (0,2000),
> (0,1), (0,5), (0,66000), (0,74000), (0,82000), (1,8), (1,0),
> (1,16000), (1,24000), (1,4), (1,56000), (1,64000), (0,12000), (0,28000),
> (0,52000), (0,6), (0,68000), (0,76000), (0,1000), (0,25000), (0,33000),
> (0,41000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000), (0,6000),
> (0,14000), (0,22000), (0,38000), (0,46000), (0,62000), (0,7), (1,4000),
> (1,2), (1,36000), (1,44000)]>
> Jan 19 02:31:53 at org.junit.Assert.fail(Assert.java:89)
> Jan 19 02:31:53 at org.junit.Assert.failNotEquals(Assert.java:835)
> Jan 19 02:31:53 at org.junit.Assert.assertEquals(Assert.java:120)
> Jan 19 02:31:53 at org.junit.Assert.assertEquals(Assert.java:146)
> Jan 19 02:31:53 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingKeyedState(AutoRescalingITCase.java:296)
> Jan 19 02:31:53 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState(AutoRescalingITCase.java:196)
> Jan 19 02:31:53 at java.lang.reflect.Method.invoke(Method.java:498)
> Jan 19 02:31:53 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34200][test] Fix the bug that AutoRescalingITCase.testCheckpointRescalingInKeyedState fails occasionally [flink]
flinkbot commented on PR #24286: URL: https://github.com/apache/flink/pull/24286#issuecomment-1933267143 ## CI report: * 3106d453263661e7a08972882a2aabbdf3a7b20d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-34200][test] Fix the bug that AutoRescalingITCase.testCheckpointRescalingInKeyedState fails occasionally [flink]
1996fanrui opened a new pull request, #24286: URL: https://github.com/apache/flink/pull/24286 Backporting FLINK-34200 to 1.19 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34200) AutoRescalingITCase#testCheckpointRescalingInKeyedState fails
[
https://issues.apache.org/jira/browse/FLINK-34200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815477#comment-17815477
]
Rui Fan commented on FLINK-34200:
-
Merged to:
master(1.20) via: 1b95b191922829fd8e7a76e5c9d8de68bb57ae7d and
9d7de680f4cde3e97368a18d58c2b70646a6f242
> AutoRescalingITCase#testCheckpointRescalingInKeyedState fails
> -
>
> Key: FLINK-34200
> URL: https://issues.apache.org/jira/browse/FLINK-34200
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Rui Fan
>Priority: Critical
> Labels: pull-request-available, test-stability
> Attachments: FLINK-34200.failure.log.gz, debug-34200.log
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56601=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=ea7cf968-e585-52cb-e0fc-f48de023a7ca=8200]
> {code:java}
> Jan 19 02:31:53 02:31:53.954 [ERROR] Tests run: 32, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 1050 s <<< FAILURE! -- in
> org.apache.flink.test.checkpointing.AutoRescalingITCase
> Jan 19 02:31:53 02:31:53.954 [ERROR]
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState[backend
> = rocksdb, buffersPerChannel = 2] -- Time elapsed: 59.10 s <<< FAILURE!
> Jan 19 02:31:53 java.lang.AssertionError: expected:<[(0,8000), (0,32000),
> (0,48000), (0,72000), (1,78000), (1,3), (1,54000), (0,2000), (0,1),
> (0,5), (0,66000), (0,74000), (0,82000), (1,8), (1,0), (1,16000),
> (1,24000), (1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000),
> (0,6), (0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000),
> (1,58000), (0,6000), (0,14000), (0,22000), (0,38000), (0,46000), (0,62000),
> (0,7), (1,4000), (1,2), (1,36000), (1,44000)]> but was:<[(0,8000),
> (0,32000), (0,48000), (0,72000), (1,78000), (1,3), (1,54000), (0,2000),
> (0,1), (0,5), (0,66000), (0,74000), (0,82000), (1,8), (1,0),
> (1,16000), (1,24000), (1,4), (1,56000), (1,64000), (0,12000), (0,28000),
> (0,52000), (0,6), (0,68000), (0,76000), (0,1000), (0,25000), (0,33000),
> (0,41000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000), (0,6000),
> (0,14000), (0,22000), (0,38000), (0,46000), (0,62000), (0,7), (1,4000),
> (1,2), (1,36000), (1,44000)]>
> Jan 19 02:31:53 at org.junit.Assert.fail(Assert.java:89)
> Jan 19 02:31:53 at org.junit.Assert.failNotEquals(Assert.java:835)
> Jan 19 02:31:53 at org.junit.Assert.assertEquals(Assert.java:120)
> Jan 19 02:31:53 at org.junit.Assert.assertEquals(Assert.java:146)
> Jan 19 02:31:53 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingKeyedState(AutoRescalingITCase.java:296)
> Jan 19 02:31:53 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState(AutoRescalingITCase.java:196)
> Jan 19 02:31:53 at java.lang.reflect.Method.invoke(Method.java:498)
> Jan 19 02:31:53 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34200][test] Fix the bug that AutoRescalingITCase.testCheckpointRescalingInKeyedState fails occasionally [flink]
1996fanrui merged PR #24246: URL: https://github.com/apache/flink/pull/24246 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34200][test] Fix the bug that AutoRescalingITCase.testCheckpointRescalingInKeyedState fails occasionally [flink]
1996fanrui commented on PR #24246: URL: https://github.com/apache/flink/pull/24246#issuecomment-1933257619 Thanks @XComp @StefanRRichter for the review, merging~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33960) Adaptive Scheduler doesn't respect the lowerBound for tasks
[
https://issues.apache.org/jira/browse/FLINK-33960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815475#comment-17815475
]
Rui Fan commented on FLINK-33960:
-
Merged to:
master(1.20) via: 777e96f0fbd90e5a45366c0fd54bda85dc813b94 and
71336d6c874cd3e4da3b694e22df132dff51a6a8
> Adaptive Scheduler doesn't respect the lowerBound for tasks
> ---
>
> Key: FLINK-33960
> URL: https://issues.apache.org/jira/browse/FLINK-33960
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.2, 1.18.1
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Major
> Labels: pull-request-available
>
> Adaptive Scheduler doesn't respect the lowerBound for tasks when one flink
> job has more than 1 tasks.
>
> When we using the adaptive scheduler and the rescale api, users will set the
> lowerBound and upperBound for each job vertices. And users expect the
> parallelism of all vertices between lowerBound and upperBound.
> But when one flink job has more than 1 vertex, and resource isn't enough.
> Some of lowerBound won't be respect.
> h2. How to reproduce this bug:
> One job has 2 job vertices, we set the resource requirements are:
> * Vertex1: lowerBound=2, upperBound=2
> * Vertex2: lowerBound=8, upperBound=8
> They are same slotSharingGroup, and we only 5 available slots. This job
> shouldn't run due to the slots cannot meets the resource requiremnt for
> vertex2.
> But the job can runs, and the parallelism of vertex2 is 5.
>
> h2. Why does this bug happen?
> Flink calculates the minimumRequiredSlots for each slot sharing group, it
> should be the {color:#FF}max{color} lowerBound for all vertices of
> current slot sharing group.
> But it's using the on the {color:#FF}minimum{color} lowerBound.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33960][Scheduler] Fix the bug that Adaptive Scheduler doesn't respect the lowerBound when one flink job has more than 1 tasks [flink]
1996fanrui merged PR #24012: URL: https://github.com/apache/flink/pull/24012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34192] Update to be compatible with updated SinkV2 interfaces [flink-connector-kafka]
mas-chen commented on code in PR #84: URL: https://github.com/apache/flink-connector-kafka/pull/84#discussion_r1482311723 ## .github/workflows/push_pr.yml: ## @@ -30,6 +30,8 @@ jobs: include: - flink: 1.18.1 jdk: '8, 11, 17' + - flink: 1.19-SNAPSHOT Review Comment: Are the rest of the changes due to removed internal APIs from Flink (e.g. the metrics stuff)? I would rename this PR as "Add build option for 1.19 to verify SinkV2 backward compatibility" or similar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34192] Update to be compatible with updated SinkV2 interfaces [flink-connector-kafka]
mas-chen commented on code in PR #84: URL: https://github.com/apache/flink-connector-kafka/pull/84#discussion_r1482310252 ## .github/workflows/push_pr.yml: ## @@ -30,6 +30,8 @@ jobs: include: - flink: 1.18.1 jdk: '8, 11, 17' + - flink: 1.19-SNAPSHOT Review Comment: Is this the only relevant thing in this PR? Everything else looks like refactoring... After https://github.com/apache/flink/pull/24249 was merged, this compiles which confirms no breaking changes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34282] Update dev-master point to 1.20 [flink-docker]
lincoln-lil merged PR #178: URL: https://github.com/apache/flink-docker/pull/178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
mtfelisb commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-1933166244 Hi, @MartijnVisser. I have a question regarding compatibility. This pull request is working on every Flink version but `1.19-SNAPSHOT`. I've checked [this issue](https://issues.apache.org/jira/browse/FLINK-34113?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=17811151#comment-17811151), however, I'm already using `TestSinkInitContext` here. It's unclear if I should consider `1.19-SNAPSHOT` and upward only. Am I missing something here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-34400) Kafka sources with watermark alignment sporadically stop consuming
[ https://issues.apache.org/jira/browse/FLINK-34400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815253#comment-17815253 ] Alexis Sarda-Espinosa edited comment on FLINK-34400 at 2/7/24 10:50 PM: Ah, let me clarify. The setup is roughly like this: * Source 1 with parallelism 2 consuming from topic A that continuously receives messages ** I have 2 Task Managers, so each source reader should be consuming approximately 15 partitions in each TM. * Source 2 with parallelism 1 consuming from topic B that rarely receives messages and has stayed mostly empty during my experiments Both sources were assigned to the same alignment group. So, what I meant is that Source 1 is showing lag in only _one_ of its readers, and the corresponding error logs only show in 1 TM, the other reader and its TM never show errors. This is why the graph I posted earlier shows the lag is slightly reduced at one step, but then increases a lot more in the next step (instead of showing increasing steps all the time). On the other hand, why would lag start showing up only after 15 minutes or so? I will probably enable idleness anyway, but I was testing both scenarios and I find these inconsistencies kind of unexpected. was (Author: asardaes): Ah, let me clarify. The setup is roughly like this: * Source 1 with parallelism 2 consuming from topic A that continuously receives messages ** I have 2 Task Managers, so each source reader should be consuming approximately 15 partitions in each TM. * Source 2 with parallelism 1 consuming from topic B that rarely receives messages and has stayed mostly empty during my experiments Both sources were assigned to the same alignment group. So, what I meant is that Source 1 is showing lag in only _one_ of its readers, and the corresponding error logs only show in 1 TM, the other reader and its TM never show errors. On the other hand, why would lag start showing up only after 15 minutes or so? I will probably enable idleness anyway, but I was testing both scenarios and I find these inconsistencies kind of unexpected. > Kafka sources with watermark alignment sporadically stop consuming > -- > > Key: FLINK-34400 > URL: https://issues.apache.org/jira/browse/FLINK-34400 > Project: Flink > Issue Type: Bug >Affects Versions: 1.18.1 >Reporter: Alexis Sarda-Espinosa >Priority: Major > Attachments: alignment_lags.png, logs.txt > > > I have 2 Kafka sources that read from different topics. I have assigned them > to the same watermark alignment group, and I have _not_ enabled idleness > explicitly in their watermark strategies. One topic remains pretty much empty > most of the time, while the other receives a few events per second all the > time. Parallelism of the active source is 2, for the other one it's 1, and > checkpoints are once every minute. > This works correctly for some time (10 - 15 minutes in my case) but then 1 of > the active sources stops consuming, which causes lag to increase. Weirdly, > after another 15 minutes or so, all the backlog is consumed at once, and then > everything stops again. > I'm attaching some logs from the Task Manager where the issue appears. You > will notice that the Kafka network client reports disconnections (a long time > after the deserializer stopped reporting that events were being consumed), > I'm not sure if this is related. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34404) GroupWindowAggregateProcTimeRestoreTest#testRestore times out
[
https://issues.apache.org/jira/browse/FLINK-34404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815429#comment-17815429
]
Bonnie Varghese commented on FLINK-34404:
-
I ran this in repeat mode (200 times) on master branch as well as release-1.19
branch and the test passed.
> GroupWindowAggregateProcTimeRestoreTest#testRestore times out
> -
>
> Key: FLINK-34404
> URL: https://issues.apache.org/jira/browse/FLINK-34404
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57357=logs=32715a4c-21b8-59a3-4171-744e5ab107eb=ff64056b-5320-5afe-c22c-6fa339e59586=11603
> {code}
> Feb 07 02:17:40 "ForkJoinPool-74-worker-1" #382 daemon prio=5 os_prio=0
> cpu=282.22ms elapsed=961.78s tid=0x7f880a485c00 nid=0x6745 waiting on
> condition [0x7f878a6f9000]
> Feb 07 02:17:40java.lang.Thread.State: WAITING (parking)
> Feb 07 02:17:40 at
> jdk.internal.misc.Unsafe.park(java.base@17.0.7/Native Method)
> Feb 07 02:17:40 - parking to wait for <0xff73d060> (a
> java.util.concurrent.CompletableFuture$Signaller)
> Feb 07 02:17:40 at
> java.util.concurrent.locks.LockSupport.park(java.base@17.0.7/LockSupport.java:211)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.7/CompletableFuture.java:1864)
> Feb 07 02:17:40 at
> java.util.concurrent.ForkJoinPool.compensatedBlock(java.base@17.0.7/ForkJoinPool.java:3449)
> Feb 07 02:17:40 at
> java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.7/ForkJoinPool.java:3432)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.7/CompletableFuture.java:1898)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture.get(java.base@17.0.7/CompletableFuture.java:2072)
> Feb 07 02:17:40 at
> org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase.testRestore(RestoreTestBase.java:292)
> Feb 07 02:17:40 at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native
> Method)
> Feb 07 02:17:40 at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
> Feb 07 02:17:40 at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
> Feb 07 02:17:40 at
> java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
> Feb 07 02:17:40 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34403) VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
[
https://issues.apache.org/jira/browse/FLINK-34403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815422#comment-17815422
]
Sai Sharath Dandi commented on FLINK-34403:
---
The test was added for an extreme case that would fail without the changes made
in the PR. If we reduce the test data size, the test case would pass without
the changes in the PR. I've tried hard to make such a test that would meet all
the requirements and also pass the Azure pipelines heap size requirement but
I'm not sure why it has started failing after passing earlier. Fwiw, the test
passes comfortably in my local environment at a much larger size than what was
merged into the codebase. I can remove this test altogether if it is not
possible to increase the heap size for the tests as the PR already achieved
it's goal
> VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
> -
>
> Key: FLINK-34403
> URL: https://issues.apache.org/jira/browse/FLINK-34403
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.20.0
>Reporter: Benchao Li
>Priority: Major
> Labels: test-stability
>
> After FLINK-33611 merged, the misc test on GHA cannot pass due to out of
> memory error, throwing following exceptions:
> {code:java}
> Error: 05:43:21 05:43:21.768 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 40.98 s <<< FAILURE! -- in
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest
> Error: 05:43:21 05:43:21.773 [ERROR]
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple -- Time
> elapsed: 40.97 s <<< ERROR!
> Feb 07 05:43:21 org.apache.flink.util.FlinkRuntimeException: Error in
> serialization.
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:327)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:162)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1007)
> Feb 07 05:43:21 at
> org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56)
> Feb 07 05:43:21 at
> org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:45)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollectWithClient(DataStream.java:1495)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1382)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1367)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.validateRow(ProtobufTestHelper.java:66)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:89)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:76)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:71)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple(VeryBigPbRowToProtoTest.java:37)
> Feb 07 05:43:21 at java.lang.reflect.Method.invoke(Method.java:498)
> Feb 07 05:43:21 Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: Self-suppression not permitted
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:323)
> Feb 07 05:43:21 ... 18 more
> Feb 07 05:43:21 Caused by: java.lang.IllegalArgumentException:
>
[jira] [Commented] (FLINK-34404) GroupWindowAggregateProcTimeRestoreTest#testRestore times out
[
https://issues.apache.org/jira/browse/FLINK-34404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815396#comment-17815396
]
Bonnie Varghese commented on FLINK-34404:
-
Unsure how this is related to FLINK-34095. Will see if I can reproduce this
locally.
> GroupWindowAggregateProcTimeRestoreTest#testRestore times out
> -
>
> Key: FLINK-34404
> URL: https://issues.apache.org/jira/browse/FLINK-34404
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57357=logs=32715a4c-21b8-59a3-4171-744e5ab107eb=ff64056b-5320-5afe-c22c-6fa339e59586=11603
> {code}
> Feb 07 02:17:40 "ForkJoinPool-74-worker-1" #382 daemon prio=5 os_prio=0
> cpu=282.22ms elapsed=961.78s tid=0x7f880a485c00 nid=0x6745 waiting on
> condition [0x7f878a6f9000]
> Feb 07 02:17:40java.lang.Thread.State: WAITING (parking)
> Feb 07 02:17:40 at
> jdk.internal.misc.Unsafe.park(java.base@17.0.7/Native Method)
> Feb 07 02:17:40 - parking to wait for <0xff73d060> (a
> java.util.concurrent.CompletableFuture$Signaller)
> Feb 07 02:17:40 at
> java.util.concurrent.locks.LockSupport.park(java.base@17.0.7/LockSupport.java:211)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.7/CompletableFuture.java:1864)
> Feb 07 02:17:40 at
> java.util.concurrent.ForkJoinPool.compensatedBlock(java.base@17.0.7/ForkJoinPool.java:3449)
> Feb 07 02:17:40 at
> java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.7/ForkJoinPool.java:3432)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.7/CompletableFuture.java:1898)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture.get(java.base@17.0.7/CompletableFuture.java:2072)
> Feb 07 02:17:40 at
> org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase.testRestore(RestoreTestBase.java:292)
> Feb 07 02:17:40 at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native
> Method)
> Feb 07 02:17:40 at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
> Feb 07 02:17:40 at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
> Feb 07 02:17:40 at
> java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
> Feb 07 02:17:40 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34234) Apply ShadeOptionalChecker for flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-34234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815379#comment-17815379
]
Sergey Nuyanzin commented on FLINK-34234:
-
This is a good question
>From one side just to fix perf regression on flink-shaded side we don't need
>to add this check
>From another side it is still not clear whether we need to do it in order to
>be able to release with maven 3.8.6 or not...
I asked this question in PR however still no answer.
I think we can continue with fix of pref regression (on flink-shaded side)
first and have another task for movement towards maven 3.8.6
> Apply ShadeOptionalChecker for flink-shaded
> ---
>
> Key: FLINK-34234
> URL: https://issues.apache.org/jira/browse/FLINK-34234
> Project: Flink
> Issue Type: Bug
> Components: BuildSystem / Shaded
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Blocker
> Labels: pull-request-available
>
> As it was found within FLINK-34148
> that newer version of shade plugin breaks previous behavior and non shaded
> artifacts are started being added to flink-shaded deps.
>
> The tasks is to apply same check for flink-shaded with help of
> {{ShadeOptionalChecker}} which is already applied for Flink
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34267][CI] Update miniconda install script to fix build on MacOS [flink-connector-shared-utils]
z3d1k commented on PR #34:
URL:
https://github.com/apache/flink-connector-shared-utils/pull/34#issuecomment-1932525296
Thanks for the feedback @gaborgsomogyi, I see that main repo has newer
miniconda versions in the script:
`Miniconda3-py310_23.5.2-0-{OS}` in main vs `Miniconda3-4.7.12.1-{OS}` here.
https://github.com/apache/flink/blame/master/flink-python/dev/lint-python.sh#L200-L203
I will verify that version in main repo works and update this PR to align
versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34267][CI] Update miniconda install script to fix build on MacOS [flink-connector-shared-utils]
gaborgsomogyi commented on PR #34: URL: https://github.com/apache/flink-connector-shared-utils/pull/34#issuecomment-1932446087 It would be good to change in Flink main repo too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
dawidwys commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481739305
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -323,15 +306,47 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
}
val builder = call.builder()
-val condition = builder.and(joinCondition, temporalCondition)
-
-builder.push(leftInput)
-builder.push(snapshot)
-builder.join(correlate.getJoinType, condition)
-val temporalJoin = builder.build()
+val condition = builder.and(actualJoin.getCondition, temporalCondition)
+
+val joinWithTemporalCondition = actualJoin.copy(
+ actualJoin.getTraitSet,
+ condition,
+ actualJoin.getLeft,
+ actualJoin.getRight,
+ actualJoin.getJoinType,
+ actualJoin.isSemiJoinDone)
+
+val temporalJoin = if (actualJoin != rewriteJoin) {
+ rewriteJoin.replaceInput(0, joinWithTemporalCondition)
Review Comment:
I admit I had to check this again, but we add the projection if we pushed
anything down. If we don't we don't have the extra projection. We have cases
for that (basically all tests before this PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
jnh5y commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481698481
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -323,15 +306,47 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
}
val builder = call.builder()
-val condition = builder.and(joinCondition, temporalCondition)
-
-builder.push(leftInput)
-builder.push(snapshot)
-builder.join(correlate.getJoinType, condition)
-val temporalJoin = builder.build()
+val condition = builder.and(actualJoin.getCondition, temporalCondition)
+
+val joinWithTemporalCondition = actualJoin.copy(
+ actualJoin.getTraitSet,
+ condition,
+ actualJoin.getLeft,
+ actualJoin.getRight,
+ actualJoin.getJoinType,
+ actualJoin.isSemiJoinDone)
+
+val temporalJoin = if (actualJoin != rewriteJoin) {
+ rewriteJoin.replaceInput(0, joinWithTemporalCondition)
Review Comment:
Do we have a test cases for both when the `actualJoin` is equal and not
equal to the `rewriteJoin`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
dawidwys commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481687197
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -323,15 +306,47 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
}
val builder = call.builder()
-val condition = builder.and(joinCondition, temporalCondition)
-
-builder.push(leftInput)
-builder.push(snapshot)
-builder.join(correlate.getJoinType, condition)
-val temporalJoin = builder.build()
+val condition = builder.and(actualJoin.getCondition, temporalCondition)
+
+val joinWithTemporalCondition = actualJoin.copy(
+ actualJoin.getTraitSet,
+ condition,
+ actualJoin.getLeft,
+ actualJoin.getRight,
+ actualJoin.getJoinType,
+ actualJoin.isSemiJoinDone)
+
+val temporalJoin = if (actualJoin != rewriteJoin) {
+ rewriteJoin.replaceInput(0, joinWithTemporalCondition)
Review Comment:
What test cases do you have in mind? As far as I can tell we don't have a
pushdown through a correlate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
dawidwys commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481689031
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/TemporalJoinTestPrograms.java:
##
@@ -84,6 +130,36 @@ public class TemporalJoinTestPrograms {
+ "ON o.currency = r.currency ")
.build();
+static final TableTestProgram TEMPORAL_JOIN_TABLE_JOIN_NESTED_KEY =
+TableTestProgram.of(
+"temporal-join-table-join-nested-key",
+"validates temporal join with a table when the
join keys comes from a nested row")
+.setupTableSource(ORDERS_WITH_NESTED_ID)
+.setupTableSource(RATES)
+.setupTableSink(AMOUNTS)
+.runSql(
+"INSERT INTO MySink "
++ "SELECT amount * r.rate "
++ "FROM OrdersNestedId AS o "
++ "JOIN RatesHistory FOR SYSTEM_TIME AS OF
o.rowtime AS r "
++ "ON o.nested_row.currency = r.currency ")
+.build();
+
+static final TableTestProgram TEMPORAL_JOIN_TABLE_JOIN_KEY_FROM_MAP =
+TableTestProgram.of(
+"temporal-join-table-join-key-from-map",
+"validates temporal join with a table when the
join key comes from a map value")
+.setupTableSource(ORDERS_WITH_NESTED_ID)
+.setupTableSource(RATES)
+.setupTableSink(AMOUNTS)
+.runSql(
+"INSERT INTO MySink "
++ "SELECT amount * r.rate "
++ "FROM OrdersNestedId AS o "
++ "JOIN RatesHistory FOR SYSTEM_TIME AS OF
o.rowtime AS r "
++ "ON o.nested_map['currency'] =
r.currency ")
Review Comment:
I'll add those cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
dawidwys commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481688174
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -237,57 +245,32 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
val snapshot = getLogicalSnapshot(call)
val leftRowType = leftInput.getRowType
-val joinCondition = filterCondition.accept(new RexShuttle() {
- // change correlate variable expression to normal RexInputRef (which is
from left side)
- override def visitFieldAccess(fieldAccess: RexFieldAccess): RexNode = {
-fieldAccess.getReferenceExpr match {
- case corVar: RexCorrelVariable =>
-require(correlate.getCorrelationId.equals(corVar.id))
-val index = leftRowType.getFieldList.indexOf(fieldAccess.getField)
-RexInputRef.of(index, leftRowType)
- case _ => super.visitFieldAccess(fieldAccess)
-}
- }
-
- // update the field index from right side
- override def visitInputRef(inputRef: RexInputRef): RexNode = {
-val rightIndex = leftRowType.getFieldCount + inputRef.getIndex
-new RexInputRef(rightIndex, inputRef.getType)
- }
-})
+val joinCondition = decorrelate(filterCondition, leftRowType,
correlate.getCorrelationId)
validateSnapshotInCorrelate(snapshot, correlate)
val rexBuilder = correlate.getCluster.getRexBuilder
-val (leftJoinKey, rightJoinKey) = {
- val relBuilder = call.builder()
- relBuilder.push(leftInput)
- relBuilder.push(snapshot)
- val rewriteJoin = relBuilder.join(correlate.getJoinType,
joinCondition).build()
- val joinInfo = rewriteJoin.asInstanceOf[LogicalJoin].analyzeCondition()
- val leftJoinKey = joinInfo.leftKeys.map(i =>
rexBuilder.makeInputRef(leftInput, i))
- val leftFieldCnt = leftInput.getRowType.getFieldCount
- val rightJoinKey = joinInfo.rightKeys.map(
-i => {
- val leftKeyType = snapshot.getRowType.getFieldList.get(i).getType
- rexBuilder.makeInputRef(leftKeyType, leftFieldCnt + i)
-})
- if (leftJoinKey.length == 0 || rightJoinKey.length == 0) {
-throw new ValidationException(
- "Currently the join key in Temporal Table Join " +
-"can not be empty.")
- }
- (leftJoinKey, rightJoinKey)
+val relBuilder = call.builder()
+relBuilder.push(leftInput)
+relBuilder.push(snapshot)
+val nonPushedJoin =
+ relBuilder.join(correlate.getJoinType,
joinCondition).build().asInstanceOf[LogicalJoin]
+val rewriteJoin = RelOptUtil.pushDownJoinConditions(nonPushedJoin,
relBuilder)
Review Comment:
If they cannot they won't. If that results in that we won't have an equi
join condition, this will fail as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
dawidwys commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481688174
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -237,57 +245,32 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
val snapshot = getLogicalSnapshot(call)
val leftRowType = leftInput.getRowType
-val joinCondition = filterCondition.accept(new RexShuttle() {
- // change correlate variable expression to normal RexInputRef (which is
from left side)
- override def visitFieldAccess(fieldAccess: RexFieldAccess): RexNode = {
-fieldAccess.getReferenceExpr match {
- case corVar: RexCorrelVariable =>
-require(correlate.getCorrelationId.equals(corVar.id))
-val index = leftRowType.getFieldList.indexOf(fieldAccess.getField)
-RexInputRef.of(index, leftRowType)
- case _ => super.visitFieldAccess(fieldAccess)
-}
- }
-
- // update the field index from right side
- override def visitInputRef(inputRef: RexInputRef): RexNode = {
-val rightIndex = leftRowType.getFieldCount + inputRef.getIndex
-new RexInputRef(rightIndex, inputRef.getType)
- }
-})
+val joinCondition = decorrelate(filterCondition, leftRowType,
correlate.getCorrelationId)
validateSnapshotInCorrelate(snapshot, correlate)
val rexBuilder = correlate.getCluster.getRexBuilder
-val (leftJoinKey, rightJoinKey) = {
- val relBuilder = call.builder()
- relBuilder.push(leftInput)
- relBuilder.push(snapshot)
- val rewriteJoin = relBuilder.join(correlate.getJoinType,
joinCondition).build()
- val joinInfo = rewriteJoin.asInstanceOf[LogicalJoin].analyzeCondition()
- val leftJoinKey = joinInfo.leftKeys.map(i =>
rexBuilder.makeInputRef(leftInput, i))
- val leftFieldCnt = leftInput.getRowType.getFieldCount
- val rightJoinKey = joinInfo.rightKeys.map(
-i => {
- val leftKeyType = snapshot.getRowType.getFieldList.get(i).getType
- rexBuilder.makeInputRef(leftKeyType, leftFieldCnt + i)
-})
- if (leftJoinKey.length == 0 || rightJoinKey.length == 0) {
-throw new ValidationException(
- "Currently the join key in Temporal Table Join " +
-"can not be empty.")
- }
- (leftJoinKey, rightJoinKey)
+val relBuilder = call.builder()
+relBuilder.push(leftInput)
+relBuilder.push(snapshot)
+val nonPushedJoin =
+ relBuilder.join(correlate.getJoinType,
joinCondition).build().asInstanceOf[LogicalJoin]
+val rewriteJoin = RelOptUtil.pushDownJoinConditions(nonPushedJoin,
relBuilder)
Review Comment:
If they cannot they won't. If that results we won't have equi join
condition, this will fail as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
dawidwys commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481687197
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -323,15 +306,47 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
}
val builder = call.builder()
-val condition = builder.and(joinCondition, temporalCondition)
-
-builder.push(leftInput)
-builder.push(snapshot)
-builder.join(correlate.getJoinType, condition)
-val temporalJoin = builder.build()
+val condition = builder.and(actualJoin.getCondition, temporalCondition)
+
+val joinWithTemporalCondition = actualJoin.copy(
+ actualJoin.getTraitSet,
+ condition,
+ actualJoin.getLeft,
+ actualJoin.getRight,
+ actualJoin.getJoinType,
+ actualJoin.isSemiJoinDone)
+
+val temporalJoin = if (actualJoin != rewriteJoin) {
+ rewriteJoin.replaceInput(0, joinWithTemporalCondition)
Review Comment:
What cases do you have in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
jnh5y commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481681792
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -237,57 +245,32 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
val snapshot = getLogicalSnapshot(call)
val leftRowType = leftInput.getRowType
-val joinCondition = filterCondition.accept(new RexShuttle() {
- // change correlate variable expression to normal RexInputRef (which is
from left side)
- override def visitFieldAccess(fieldAccess: RexFieldAccess): RexNode = {
-fieldAccess.getReferenceExpr match {
- case corVar: RexCorrelVariable =>
-require(correlate.getCorrelationId.equals(corVar.id))
-val index = leftRowType.getFieldList.indexOf(fieldAccess.getField)
-RexInputRef.of(index, leftRowType)
- case _ => super.visitFieldAccess(fieldAccess)
-}
- }
-
- // update the field index from right side
- override def visitInputRef(inputRef: RexInputRef): RexNode = {
-val rightIndex = leftRowType.getFieldCount + inputRef.getIndex
-new RexInputRef(rightIndex, inputRef.getType)
- }
-})
+val joinCondition = decorrelate(filterCondition, leftRowType,
correlate.getCorrelationId)
validateSnapshotInCorrelate(snapshot, correlate)
val rexBuilder = correlate.getCluster.getRexBuilder
-val (leftJoinKey, rightJoinKey) = {
- val relBuilder = call.builder()
- relBuilder.push(leftInput)
- relBuilder.push(snapshot)
- val rewriteJoin = relBuilder.join(correlate.getJoinType,
joinCondition).build()
- val joinInfo = rewriteJoin.asInstanceOf[LogicalJoin].analyzeCondition()
- val leftJoinKey = joinInfo.leftKeys.map(i =>
rexBuilder.makeInputRef(leftInput, i))
- val leftFieldCnt = leftInput.getRowType.getFieldCount
- val rightJoinKey = joinInfo.rightKeys.map(
-i => {
- val leftKeyType = snapshot.getRowType.getFieldList.get(i).getType
- rexBuilder.makeInputRef(leftKeyType, leftFieldCnt + i)
-})
- if (leftJoinKey.length == 0 || rightJoinKey.length == 0) {
-throw new ValidationException(
- "Currently the join key in Temporal Table Join " +
-"can not be empty.")
- }
- (leftJoinKey, rightJoinKey)
+val relBuilder = call.builder()
+relBuilder.push(leftInput)
+relBuilder.push(snapshot)
+val nonPushedJoin =
+ relBuilder.join(correlate.getJoinType,
joinCondition).build().asInstanceOf[LogicalJoin]
+val rewriteJoin = RelOptUtil.pushDownJoinConditions(nonPushedJoin,
relBuilder)
Review Comment:
In what situations can the join conditions not be pushed down?
We need some validation for those situations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
jnh5y commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481679233
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala:
##
@@ -323,15 +306,47 @@ abstract class
LogicalCorrelateToJoinFromGeneralTemporalTableRule(
}
val builder = call.builder()
-val condition = builder.and(joinCondition, temporalCondition)
-
-builder.push(leftInput)
-builder.push(snapshot)
-builder.join(correlate.getJoinType, condition)
-val temporalJoin = builder.build()
+val condition = builder.and(actualJoin.getCondition, temporalCondition)
+
+val joinWithTemporalCondition = actualJoin.copy(
+ actualJoin.getTraitSet,
+ condition,
+ actualJoin.getLeft,
+ actualJoin.getRight,
+ actualJoin.getJoinType,
+ actualJoin.isSemiJoinDone)
+
+val temporalJoin = if (actualJoin != rewriteJoin) {
+ rewriteJoin.replaceInput(0, joinWithTemporalCondition)
Review Comment:
Are there test cases which cover adding the projection?
Also, does this run into any issues if there are multiple pushdowns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-31220] Replace Pod with PodTemplateSpec for the pod template properties [flink-kubernetes-operator]
gyfora commented on PR #770: URL: https://github.com/apache/flink-kubernetes-operator/pull/770#issuecomment-1932314211 > My understanding is that in the main Flink project people are encouraged to use the Jira Release Note field in case something special needs to be mentioned for the respective ticket being closed. AFAIK release managers were previously scanning those when preparing Flink release announcements. Not a must, obviously, but generally seems like a good practice we could also follow for the operator. Makes sense, I actually already added a release note to the ticket after your previous comment, I just forgot to mention it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34409][ci] Enables (most) tests that were disabled for the AdaptiveScheduler due to missing features [flink]
XComp commented on PR #24285: URL: https://github.com/apache/flink/pull/24285#issuecomment-1932301482 > Should we check if any on the other tests that are currently disabled for AdaptiveScheduler would now work, e.g. ? Yikes, that was actually my intention. But looks like I forgot to do another pass over the e2e test scripts. :-D Thanks. I will enable those as well. We should add a comment with the reason if we notice that they still fail. :+1: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-24239] Event time temporal join should support values from array, map, row, etc. as join key [flink]
jnh5y commented on code in PR #24253:
URL: https://github.com/apache/flink/pull/24253#discussion_r1481662098
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/TemporalJoinTestPrograms.java:
##
@@ -84,6 +130,36 @@ public class TemporalJoinTestPrograms {
+ "ON o.currency = r.currency ")
.build();
+static final TableTestProgram TEMPORAL_JOIN_TABLE_JOIN_NESTED_KEY =
+TableTestProgram.of(
+"temporal-join-table-join-nested-key",
+"validates temporal join with a table when the
join keys comes from a nested row")
+.setupTableSource(ORDERS_WITH_NESTED_ID)
+.setupTableSource(RATES)
+.setupTableSink(AMOUNTS)
+.runSql(
+"INSERT INTO MySink "
++ "SELECT amount * r.rate "
++ "FROM OrdersNestedId AS o "
++ "JOIN RatesHistory FOR SYSTEM_TIME AS OF
o.rowtime AS r "
++ "ON o.nested_row.currency = r.currency ")
+.build();
+
+static final TableTestProgram TEMPORAL_JOIN_TABLE_JOIN_KEY_FROM_MAP =
+TableTestProgram.of(
+"temporal-join-table-join-key-from-map",
+"validates temporal join with a table when the
join key comes from a map value")
+.setupTableSource(ORDERS_WITH_NESTED_ID)
+.setupTableSource(RATES)
+.setupTableSink(AMOUNTS)
+.runSql(
+"INSERT INTO MySink "
++ "SELECT amount * r.rate "
++ "FROM OrdersNestedId AS o "
++ "JOIN RatesHistory FOR SYSTEM_TIME AS OF
o.rowtime AS r "
++ "ON o.nested_map['currency'] =
r.currency ")
Review Comment:
Tests are added for covering what happens with the left-hand side of the
join.
Do we need to verify anything for the right-hand side and/or both sides?
Also, can join on the output of a function call? (E.g., `ON
UPPERCASE(o.nested_map['currency']) = r.currency`?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-31220] Replace Pod with PodTemplateSpec for the pod template properties [flink-kubernetes-operator]
afedulov commented on PR #770: URL: https://github.com/apache/flink-kubernetes-operator/pull/770#issuecomment-1932291404 My understanding is that in the main Flink project people are encouraged to use the Jira Release Note field in case something special needs to be mentioned for the respective ticket being closed. AFAIK release managers were previously scanning those when preparing Flink release announcements. Not a must, obviously, but generally seems like a good practice we could also follow for the operator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34409][ci] Enables (most) tests that were disabled for the AdaptiveScheduler due to missing features [flink]
MartijnVisser commented on code in PR #24285: URL: https://github.com/apache/flink/pull/24285#discussion_r1481652226 ## tools/azure-pipelines/build-apache-repo.yml: ## @@ -69,7 +69,7 @@ stages: name: Default e2e_pool_definition: vmImage: 'ubuntu-20.04' - environment: PROFILE="-Dflink.hadoop.version=2.10.2" + environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Penable-adaptive-scheduler" Review Comment: Oh wow I completely missed the debug commit 臘 - Thanks :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34409][ci] Enables (most) tests that were disabled for the AdaptiveScheduler due to missing features [flink]
XComp commented on code in PR #24285: URL: https://github.com/apache/flink/pull/24285#discussion_r1481649353 ## tools/azure-pipelines/build-apache-repo.yml: ## @@ -69,7 +69,7 @@ stages: name: Default e2e_pool_definition: vmImage: 'ubuntu-20.04' - environment: PROFILE="-Dflink.hadoop.version=2.10.2" + environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Penable-adaptive-scheduler" Review Comment: That's a debug commit which I added to the PR to have a CI run with the AdaptiveScheduler enabled. I'm not planning to merge this commit. The goal is to have a CI run that actually contains the changes of this PR. That way, we can verify that the tests are actually enabled now (even before merging the PR). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34409][ci] Enables (most) tests that were disabled for the AdaptiveScheduler due to missing features [flink]
MartijnVisser commented on code in PR #24285: URL: https://github.com/apache/flink/pull/24285#discussion_r1481645434 ## tools/azure-pipelines/build-apache-repo.yml: ## @@ -69,7 +69,7 @@ stages: name: Default e2e_pool_definition: vmImage: 'ubuntu-20.04' - environment: PROFILE="-Dflink.hadoop.version=2.10.2" + environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Penable-adaptive-scheduler" Review Comment: Why do we want to activate the adaptive-scheduler profile here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34409][ci] Enables (most) tests that were disabled for the AdaptiveScheduler due to missing features [flink]
flinkbot commented on PR #24285: URL: https://github.com/apache/flink/pull/24285#issuecomment-1932271167 ## CI report: * 5ccb2b93fcc79079d28293d9a72394236aa412f5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34234) Apply ShadeOptionalChecker for flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-34234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815338#comment-17815338
]
Matthias Pohl commented on FLINK-34234:
---
[~Sergey Nuyanzin] what's the state of this issue now? Shall we close it as
{{Not A Problem}}? Or is it still worth adding the check?
> Apply ShadeOptionalChecker for flink-shaded
> ---
>
> Key: FLINK-34234
> URL: https://issues.apache.org/jira/browse/FLINK-34234
> Project: Flink
> Issue Type: Bug
> Components: BuildSystem / Shaded
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Blocker
> Labels: pull-request-available
>
> As it was found within FLINK-34148
> that newer version of shade plugin breaks previous behavior and non shaded
> artifacts are started being added to flink-shaded deps.
>
> The tasks is to apply same check for flink-shaded with help of
> {{ShadeOptionalChecker}} which is already applied for Flink
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34403) VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
[
https://issues.apache.org/jira/browse/FLINK-34403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815314#comment-17815314
]
Matthias Pohl edited comment on FLINK-34403 at 2/7/24 3:16 PM:
---
[~libenchao] Maybe, I don't fully understand the test case. But can't we reduce
the test data, instead?
was (Author: mapohl):
[~libenchao] Maybe, I'm not fully aware of the test's purpose. But can't we
reduce the test data, instead?
> VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
> -
>
> Key: FLINK-34403
> URL: https://issues.apache.org/jira/browse/FLINK-34403
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.20.0
>Reporter: Benchao Li
>Priority: Major
> Labels: test-stability
>
> After FLINK-33611 merged, the misc test on GHA cannot pass due to out of
> memory error, throwing following exceptions:
> {code:java}
> Error: 05:43:21 05:43:21.768 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 40.98 s <<< FAILURE! -- in
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest
> Error: 05:43:21 05:43:21.773 [ERROR]
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple -- Time
> elapsed: 40.97 s <<< ERROR!
> Feb 07 05:43:21 org.apache.flink.util.FlinkRuntimeException: Error in
> serialization.
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:327)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:162)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1007)
> Feb 07 05:43:21 at
> org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56)
> Feb 07 05:43:21 at
> org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:45)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollectWithClient(DataStream.java:1495)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1382)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1367)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.validateRow(ProtobufTestHelper.java:66)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:89)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:76)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:71)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple(VeryBigPbRowToProtoTest.java:37)
> Feb 07 05:43:21 at java.lang.reflect.Method.invoke(Method.java:498)
> Feb 07 05:43:21 Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: Self-suppression not permitted
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:323)
> Feb 07 05:43:21 ... 18 more
> Feb 07 05:43:21 Caused by: java.lang.IllegalArgumentException:
> Self-suppression not permitted
> Feb 07 05:43:21 at
> java.lang.Throwable.addSuppressed(Throwable.java:1072)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:556)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:486)
> Feb 07 05:43:21 at
[jira] [Comment Edited] (FLINK-34403) VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
[
https://issues.apache.org/jira/browse/FLINK-34403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815314#comment-17815314
]
Matthias Pohl edited comment on FLINK-34403 at 2/7/24 3:16 PM:
---
[~libenchao] Maybe, I don't fully understand the test case. But can't we reduce
the test data size, instead?
was (Author: mapohl):
[~libenchao] Maybe, I don't fully understand the test case. But can't we reduce
the test data, instead?
> VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
> -
>
> Key: FLINK-34403
> URL: https://issues.apache.org/jira/browse/FLINK-34403
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.20.0
>Reporter: Benchao Li
>Priority: Major
> Labels: test-stability
>
> After FLINK-33611 merged, the misc test on GHA cannot pass due to out of
> memory error, throwing following exceptions:
> {code:java}
> Error: 05:43:21 05:43:21.768 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 40.98 s <<< FAILURE! -- in
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest
> Error: 05:43:21 05:43:21.773 [ERROR]
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple -- Time
> elapsed: 40.97 s <<< ERROR!
> Feb 07 05:43:21 org.apache.flink.util.FlinkRuntimeException: Error in
> serialization.
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:327)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:162)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1007)
> Feb 07 05:43:21 at
> org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56)
> Feb 07 05:43:21 at
> org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:45)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollectWithClient(DataStream.java:1495)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1382)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1367)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.validateRow(ProtobufTestHelper.java:66)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:89)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:76)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:71)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple(VeryBigPbRowToProtoTest.java:37)
> Feb 07 05:43:21 at java.lang.reflect.Method.invoke(Method.java:498)
> Feb 07 05:43:21 Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: Self-suppression not permitted
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:323)
> Feb 07 05:43:21 ... 18 more
> Feb 07 05:43:21 Caused by: java.lang.IllegalArgumentException:
> Self-suppression not permitted
> Feb 07 05:43:21 at
> java.lang.Throwable.addSuppressed(Throwable.java:1072)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:556)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:486)
> Feb 07 05:43:21 at
[jira] [Updated] (FLINK-34403) VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
[
https://issues.apache.org/jira/browse/FLINK-34403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-34403:
--
Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
(was: Build System / CI)
> VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
> -
>
> Key: FLINK-34403
> URL: https://issues.apache.org/jira/browse/FLINK-34403
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.20.0
>Reporter: Benchao Li
>Priority: Major
> Labels: test-stability
>
> After FLINK-33611 merged, the misc test on GHA cannot pass due to out of
> memory error, throwing following exceptions:
> {code:java}
> Error: 05:43:21 05:43:21.768 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 40.98 s <<< FAILURE! -- in
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest
> Error: 05:43:21 05:43:21.773 [ERROR]
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple -- Time
> elapsed: 40.97 s <<< ERROR!
> Feb 07 05:43:21 org.apache.flink.util.FlinkRuntimeException: Error in
> serialization.
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:327)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:162)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1007)
> Feb 07 05:43:21 at
> org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56)
> Feb 07 05:43:21 at
> org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:45)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollectWithClient(DataStream.java:1495)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1382)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1367)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.validateRow(ProtobufTestHelper.java:66)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:89)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:76)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:71)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple(VeryBigPbRowToProtoTest.java:37)
> Feb 07 05:43:21 at java.lang.reflect.Method.invoke(Method.java:498)
> Feb 07 05:43:21 Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: Self-suppression not permitted
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:323)
> Feb 07 05:43:21 ... 18 more
> Feb 07 05:43:21 Caused by: java.lang.IllegalArgumentException:
> Self-suppression not permitted
> Feb 07 05:43:21 at
> java.lang.Throwable.addSuppressed(Throwable.java:1072)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:556)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:486)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamConfig.lambda$triggerSerializationAndReturnFuture$0(StreamConfig.java:182)
> Feb 07 05:43:21 at
>
[jira] [Updated] (FLINK-34403) VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
[
https://issues.apache.org/jira/browse/FLINK-34403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-34403:
--
Labels: test-stability (was: )
> VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM
> -
>
> Key: FLINK-34403
> URL: https://issues.apache.org/jira/browse/FLINK-34403
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Affects Versions: 1.20.0
>Reporter: Benchao Li
>Priority: Major
> Labels: test-stability
>
> After FLINK-33611 merged, the misc test on GHA cannot pass due to out of
> memory error, throwing following exceptions:
> {code:java}
> Error: 05:43:21 05:43:21.768 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 40.98 s <<< FAILURE! -- in
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest
> Error: 05:43:21 05:43:21.773 [ERROR]
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple -- Time
> elapsed: 40.97 s <<< ERROR!
> Feb 07 05:43:21 org.apache.flink.util.FlinkRuntimeException: Error in
> serialization.
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:327)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:162)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1007)
> Feb 07 05:43:21 at
> org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56)
> Feb 07 05:43:21 at
> org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:45)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
> Feb 07 05:43:21 at
> org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollectWithClient(DataStream.java:1495)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1382)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1367)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.validateRow(ProtobufTestHelper.java:66)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:89)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:76)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:71)
> Feb 07 05:43:21 at
> org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple(VeryBigPbRowToProtoTest.java:37)
> Feb 07 05:43:21 at java.lang.reflect.Method.invoke(Method.java:498)
> Feb 07 05:43:21 Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: Self-suppression not permitted
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:323)
> Feb 07 05:43:21 ... 18 more
> Feb 07 05:43:21 Caused by: java.lang.IllegalArgumentException:
> Self-suppression not permitted
> Feb 07 05:43:21 at
> java.lang.Throwable.addSuppressed(Throwable.java:1072)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:556)
> Feb 07 05:43:21 at
> org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:486)
> Feb 07 05:43:21 at
> org.apache.flink.streaming.api.graph.StreamConfig.lambda$triggerSerializationAndReturnFuture$0(StreamConfig.java:182)
> Feb 07 05:43:21 at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
> Feb 07 05:43:21 at
>
Re: [PR] [FLINK-21400][ci] Enables FileSink and Stateful stream job e2e test for the AdaptiveScheduler [flink]
XComp commented on PR #24284: URL: https://github.com/apache/flink/pull/24284#issuecomment-1932261297 Closing the issue in favor of FLINK-34409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-21400][ci] Enables FileSink and Stateful stream job e2e test for the AdaptiveScheduler [flink]
XComp closed pull request #24284: [FLINK-21400][ci] Enables FileSink and Stateful stream job e2e test for the AdaptiveScheduler URL: https://github.com/apache/flink/pull/24284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34409) Increase test coverage for AdaptiveScheduler
[
https://issues.apache.org/jira/browse/FLINK-34409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-34409:
---
Labels: pull-request-available (was: )
> Increase test coverage for AdaptiveScheduler
>
>
> Key: FLINK-34409
> URL: https://issues.apache.org/jira/browse/FLINK-34409
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.17.2, 1.19.0, 1.18.1, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> There are still several tests disabled for the {{AdaptiveScheduler}} which we
> can enable now. All the issues seem to have been fixed.
> We can even remove the annotation {{@FailsWithAdaptiveScheduler}} now. It's
> not needed anymore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-34409][ci] Enables (most) tests that were disabled for the AdaptiveScheduler due to missing features [flink]
XComp opened a new pull request, #24285: URL: https://github.com/apache/flink/pull/24285 ## What is the purpose of the change Increasing test coverage. ## Brief change log * Removes annotations ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. PR is temporarily enabling the AdaptiveScheduler ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34409) Increase test coverage for AdaptiveScheduler
[
https://issues.apache.org/jira/browse/FLINK-34409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815337#comment-17815337
]
Matthias Pohl commented on FLINK-34409:
---
We can't remove the annotation because of
{{MiniClusterITCase#testHandlingNotEnoughSlotsThroughEarlyAbortRequest}}.
> Increase test coverage for AdaptiveScheduler
>
>
> Key: FLINK-34409
> URL: https://issues.apache.org/jira/browse/FLINK-34409
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.17.2, 1.19.0, 1.18.1, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> There are still several tests disabled for the {{AdaptiveScheduler}} which we
> can enable now. All the issues seem to have been fixed.
> We can even remove the annotation {{@FailsWithAdaptiveScheduler}} now. It's
> not needed anymore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34409) Increase test coverage for AdaptiveScheduler
[
https://issues.apache.org/jira/browse/FLINK-34409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-34409:
--
Affects Version/s: 1.17.2
> Increase test coverage for AdaptiveScheduler
>
>
> Key: FLINK-34409
> URL: https://issues.apache.org/jira/browse/FLINK-34409
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.17.2, 1.19.0, 1.18.1, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> There are still several tests disabled for the {{AdaptiveScheduler}} which we
> can enable now. All the issues seem to have been fixed.
> We can even remove the annotation {{@FailsWithAdaptiveScheduler}} now. It's
> not needed anymore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34409) Increase test coverage for AdaptiveScheduler
[
https://issues.apache.org/jira/browse/FLINK-34409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815336#comment-17815336
]
Matthias Pohl commented on FLINK-34409:
---
Linking FLINK-21535, FLINK-21450 and FLINK-21400. These were the issues that
motivated disabling the tests. All of them are resolved now.
> Increase test coverage for AdaptiveScheduler
>
>
> Key: FLINK-34409
> URL: https://issues.apache.org/jira/browse/FLINK-34409
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.19.0, 1.18.1, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> There are still several tests disabled for the {{AdaptiveScheduler}} which we
> can enable now. All the issues seem to have been fixed.
> We can even remove the annotation {{@FailsWithAdaptiveScheduler}} now. It's
> not needed anymore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34409) Increase test coverage for AdaptiveScheduler
[
https://issues.apache.org/jira/browse/FLINK-34409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-34409:
-
Assignee: Matthias Pohl
> Increase test coverage for AdaptiveScheduler
>
>
> Key: FLINK-34409
> URL: https://issues.apache.org/jira/browse/FLINK-34409
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.19.0, 1.18.1, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> There are still several tests disabled for the {{AdaptiveScheduler}} which we
> can enable now. All the issues seem to have been fixed.
> We can even remove the annotation {{@FailsWithAdaptiveScheduler}} now. It's
> not needed anymore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-21949) Support ARRAY_AGG aggregate function
[ https://issues.apache.org/jira/browse/FLINK-21949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-21949. Fix Version/s: 1.20.0 (was: 1.19.0) Resolution: Implemented Implemented in 042a4d2d8a8cec10ea9c287c1ebf7769bd469b22 > Support ARRAY_AGG aggregate function > > > Key: FLINK-21949 > URL: https://issues.apache.org/jira/browse/FLINK-21949 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.12.0 >Reporter: Jiabao Sun >Assignee: Jiabao Sun >Priority: Minor > Labels: pull-request-available > Fix For: 1.20.0 > > > Some nosql databases like mongodb and elasticsearch support nested data types. > Aggregating multiple rows into ARRAY is a common requirement. > The CollectToArray function is similar to Collect, except that it returns > ARRAY instead of MULTISET. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34409) Increase test coverage for AdaptiveScheduler
Matthias Pohl created FLINK-34409:
-
Summary: Increase test coverage for AdaptiveScheduler
Key: FLINK-34409
URL: https://issues.apache.org/jira/browse/FLINK-34409
Project: Flink
Issue Type: Technical Debt
Components: Runtime / Coordination
Affects Versions: 1.18.1, 1.19.0, 1.20.0
Reporter: Matthias Pohl
There are still several tests disabled for the {{AdaptiveScheduler}} which we
can enable now. All the issues seem to have been fixed.
We can even remove the annotation {{@FailsWithAdaptiveScheduler}} now. It's not
needed anymore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
dawidwys merged PR #23411: URL: https://github.com/apache/flink/pull/23411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34314) Update CI Node Actions from NodeJS 16 to NodeJS 20
[
https://issues.apache.org/jira/browse/FLINK-34314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-34314:
---
Labels: pull-request-available (was: )
> Update CI Node Actions from NodeJS 16 to NodeJS 20
> --
>
> Key: FLINK-34314
> URL: https://issues.apache.org/jira/browse/FLINK-34314
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System / CI
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Node.js 16 actions are deprecated. Please update the following actions to use
> Node.js 20: actions/checkout@v3, actions/setup-java@v3,
> stCarolas/setup-maven@v4.5, actions/cache/restore@v3, actions/cache/save@v3.
> {code}
> For more information see:
> https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34314] Update CI Node Actions from NodeJS 16 to NodeJS 20 [flink-connector-shared-utils]
MartijnVisser commented on PR #35: URL: https://github.com/apache/flink-connector-shared-utils/pull/35#issuecomment-1932224534 Looking at another GHA run (say) https://github.com/apache/flink-connector-shared-utils/actions/runs/7702796322 shows that this run has no annotations about needing to update, so we should be good to go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-21400][ci] Enables FileSink and Stateful stream job e2e test for the AdaptiveScheduler [flink]
flinkbot commented on PR #24284: URL: https://github.com/apache/flink/pull/24284#issuecomment-1932211497 ## CI report: * bb877762a050476aac84f2adb60347cb4cf39185 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34282) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-34282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815329#comment-17815329
]
lincoln lee commented on FLINK-34282:
-
Record the related commits that without prs:
master branch version update: b8b2596a1e4cb726674949b031f577079832f42c
add 1.19 to docs.yml for doc build: eb1f7c8f998c4befa0bdee0743f81cc8dbac71d8
> Create a release branch
> ---
>
> Key: FLINK-34282
> URL: https://issues.apache.org/jira/browse/FLINK-34282
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.19.0
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: screenshot-1.png
>
>
> If you are doing a new minor release, you need to update Flink version in the
> following repositories and the [AzureCI project
> configuration|https://dev.azure.com/apache-flink/apache-flink/]:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Patch releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> tools $ releasing/create_snapshot_branch.sh
> tools $ git checkout master
> tools $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION
> NEW_VERSION=$NEXT_SNAPSHOT_VERSION releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{v1_16("1.16")}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> Additionally in master, update the branch list of the GitHub Actions nightly
> workflow (see
> [apache/flink:.github/workflows/nightly-trigger.yml#L31ff|https://github.com/apache/flink/blob/master/.github/workflows/nightly-trigger.yml#L31]):
> The two most-recent releases and master should be covered.
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new minor release branch, as the last step you should also
> update the documentation workflow to also build the documentation for the new
> release branch. Check [Managing
> Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
> on details on how to do that. You may also want to manually trigger a build
> to make the changes visible as soon as possible.
> h4. Flink Benchmark Repository
> First of all, checkout the {{master}} branch to {{dev-x.y}} branch in
> [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that
> we can have a branch named {{dev-x.y}} which could be built on top of
> (${{CURRENT_SNAPSHOT_VERSION}}).
> Then, inside the repository you need to manually update the {{flink.version}}
> property inside the parent *pom.xml* file. It should be pointing to the most
> recent snapshot version ($NEXT_SNAPSHOT_VERSION). For example:
> {code:xml}
> 1.18-SNAPSHOT
> {code}
> h4. AzureCI Project Configuration
> The new release branch needs to be configured within AzureCI to make azure
> aware of the new release branch. This matter can only be handled by Ververica
> employees since they are owning the AzureCI setup.
>
>
> h3. Expectations (Minor Version only if not stated otherwise)
> * Release branch has been created and pushed
> * Changes on the new release branch are picked up by [Azure
> CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
> * {{master}} branch has the version information updated to the new version
> (check pom.xml files and
> *
>
[PR] [FLINK-21400][ci] Enables FileSink and Stateful stream job e2e test for the AdaptiveScheduler [flink]
XComp opened a new pull request, #24284: URL: https://github.com/apache/flink/pull/24284 ## What is the purpose of the change Increases test coverage for `AdaptiveScheduler`. ## Brief change log * Removes if condition ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34192] Update to be compatible with updated SinkV2 interfaces [flink-connector-kafka]
MartijnVisser commented on PR #84: URL: https://github.com/apache/flink-connector-kafka/pull/84#issuecomment-1932188194 @pvary @mas-chen Can either of you want to take a look at this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
