[GitHub] [flink] flinkbot edited a comment on pull request #17030: [FLINK-24035][network] Notify the buffer listeners when the local buffer pool receives available notification from the global pool
flinkbot edited a comment on pull request #17030: URL: https://github.com/apache/flink/pull/17030#issuecomment-907558700 ## CI report: * 1ce0b8ec84514371769d20b0807ecc4ee45226d5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22990) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
flinkbot edited a comment on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-906989888 ## CI report: * bc7c93156dd9dbcee4184818b4b4f6e24c8ad234 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22989) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17009: [FLINK-23466][network] Fix the bug that buffer listeners may not be notified when recycling buffers
flinkbot edited a comment on pull request #17009: URL: https://github.com/apache/flink/pull/17009#issuecomment-906989802 ## CI report: * 1cda0c085f25dc1b8e9e36d23ac8bf30d85f58e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22987) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22969) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17011: [FLINK-23663][table-planner] Reduce state size of ChangelogNormalize
Airblader commented on a change in pull request #17011:
URL: https://github.com/apache/flink/pull/17011#discussion_r697805773
##
File path:
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TableFactoryHarness.java
##
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.factories;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.TableDescriptor;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import
org.apache.flink.table.connector.sink.DynamicTableSink.SinkRuntimeProvider;
+import org.apache.flink.table.connector.sink.SinkFunctionProvider;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.ScanTableSource;
+import
org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider;
+import org.apache.flink.table.connector.source.SourceFunctionProvider;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.DynamicTableSinkFactory;
+import org.apache.flink.table.factories.DynamicTableSourceFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Provides a flexible testing harness for table factories.
+ *
+ * This testing harness allows writing custom sources and sinks which can
be directly
+ * instantiated from the test. This avoids having to implement a factory, and
enables using the
+ * {@link SharedObjects} rule to get direct access to the underlying
source/sink from the test.
+ *
+ * Note that the underlying source/sink must be {@link Serializable}. It is
recommended to extend
+ * from {@link SourceBase} or {@link SinkBase} which provides default
implementations for most
+ * methods as well as some convenience methods.
+ *
+ * The harness provides a {@link Factory}. You can register a source / sink
through configuration
+ * by passing a base64-encoded serialization. The harness provides convenience
methods to make this
+ * process as simple as possible.
+ *
+ * Example:
+ *
+ * {@code
+ * public class CustomSourceTest {
+ * {@literal @}Rule public SharedObjects sharedObjects =
SharedObjects.create();
+ *
+ * {@literal @}Test
+ * public void test() {
+ * SharedReference> appliedLimits = sharedObjects.add(new
ArrayList<>());
+ *
+ * Schema schema = Schema.newBuilder().build();
+ * TableDescriptor sourceDescriptor =
TableFactoryHarness.forSource(schema,
+ * new CustomSource(appliedLimits));
+ *
+ * tEnv.createTable("T", sourceDescriptor);
+ * tEnv.explainSql("SELECT * FROM T LIMIT 42");
+ *
+ * assertEquals(1, appliedLimits.get().size());
+ * assertEquals((Long) 42L, appliedLimits.get().get(0));
+ * }
+ *
+ * private static class CustomSource extends SourceBase implements
SupportsLimitPushDown {
+ * private final SharedReference> appliedLimits;
+ *
+ * CustomSource(SharedReference> appliedLimits) {
+ * this.appliedLimits = appliedLimits;
+ * }
+ *
+ * {@literal @}Override
+ * public void applyLimit(long limit) {
+ * appliedLimits.get().add(limit);
+ * }
+ * }
+ * }
+ * }
+ */
+public class TableFactoryHarness {
+
+/** Factory identifier for {@link Factory}. */
+public static final String IDENTIFIER
[GitHub] [flink] Airblader commented on a change in pull request #17011: [FLINK-23663][table-planner] Reduce state size of ChangelogNormalize
Airblader commented on a change in pull request #17011:
URL: https://github.com/apache/flink/pull/17011#discussion_r697805629
##
File path:
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TableFactoryHarness.java
##
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.factories;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.TableDescriptor;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import
org.apache.flink.table.connector.sink.DynamicTableSink.SinkRuntimeProvider;
+import org.apache.flink.table.connector.sink.SinkFunctionProvider;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.ScanTableSource;
+import
org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider;
+import org.apache.flink.table.connector.source.SourceFunctionProvider;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.DynamicTableSinkFactory;
+import org.apache.flink.table.factories.DynamicTableSourceFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Provides a flexible testing harness for table factories.
+ *
+ * This testing harness allows writing custom sources and sinks which can
be directly
+ * instantiated from the test. This avoids having to implement a factory, and
enables using the
+ * {@link SharedObjects} rule to get direct access to the underlying
source/sink from the test.
+ *
+ * Note that the underlying source/sink must be {@link Serializable}. It is
recommended to extend
+ * from {@link SourceBase} or {@link SinkBase} which provides default
implementations for most
+ * methods as well as some convenience methods.
+ *
+ * The harness provides a {@link Factory}. You can register a source / sink
through configuration
+ * by passing a base64-encoded serialization. The harness provides convenience
methods to make this
+ * process as simple as possible.
+ *
+ * Example:
+ *
+ * {@code
+ * public class CustomSourceTest {
+ * {@literal @}Rule public SharedObjects sharedObjects =
SharedObjects.create();
+ *
+ * {@literal @}Test
+ * public void test() {
+ * SharedReference> appliedLimits = sharedObjects.add(new
ArrayList<>());
+ *
+ * Schema schema = Schema.newBuilder().build();
+ * TableDescriptor sourceDescriptor =
TableFactoryHarness.forSource(schema,
+ * new CustomSource(appliedLimits));
+ *
+ * tEnv.createTable("T", sourceDescriptor);
+ * tEnv.explainSql("SELECT * FROM T LIMIT 42");
+ *
+ * assertEquals(1, appliedLimits.get().size());
+ * assertEquals((Long) 42L, appliedLimits.get().get(0));
+ * }
+ *
+ * private static class CustomSource extends SourceBase implements
SupportsLimitPushDown {
+ * private final SharedReference> appliedLimits;
+ *
+ * CustomSource(SharedReference> appliedLimits) {
+ * this.appliedLimits = appliedLimits;
+ * }
+ *
+ * {@literal @}Override
+ * public void applyLimit(long limit) {
+ * appliedLimits.get().add(limit);
+ * }
+ * }
+ * }
+ * }
+ */
+public class TableFactoryHarness {
+
+/** Factory identifier for {@link Factory}. */
+public static final String IDENTIFIER
[GitHub] [flink] Airblader commented on a change in pull request #17011: [FLINK-23663][table-planner] Reduce state size of ChangelogNormalize
Airblader commented on a change in pull request #17011:
URL: https://github.com/apache/flink/pull/17011#discussion_r697805528
##
File path:
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TableFactoryHarness.java
##
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.factories;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.TableDescriptor;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import
org.apache.flink.table.connector.sink.DynamicTableSink.SinkRuntimeProvider;
+import org.apache.flink.table.connector.sink.SinkFunctionProvider;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.ScanTableSource;
+import
org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider;
+import org.apache.flink.table.connector.source.SourceFunctionProvider;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.DynamicTableSinkFactory;
+import org.apache.flink.table.factories.DynamicTableSourceFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Provides a flexible testing harness for table factories.
+ *
+ * This testing harness allows writing custom sources and sinks which can
be directly
+ * instantiated from the test. This avoids having to implement a factory, and
enables using the
+ * {@link SharedObjects} rule to get direct access to the underlying
source/sink from the test.
+ *
+ * Note that the underlying source/sink must be {@link Serializable}. It is
recommended to extend
+ * from {@link SourceBase} or {@link SinkBase} which provides default
implementations for most
+ * methods as well as some convenience methods.
+ *
+ * The harness provides a {@link Factory}. You can register a source / sink
through configuration
+ * by passing a base64-encoded serialization. The harness provides convenience
methods to make this
+ * process as simple as possible.
+ *
+ * Example:
+ *
+ * {@code
+ * public class CustomSourceTest {
+ * {@literal @}Rule public SharedObjects sharedObjects =
SharedObjects.create();
+ *
+ * {@literal @}Test
+ * public void test() {
+ * SharedReference> appliedLimits = sharedObjects.add(new
ArrayList<>());
Review comment:
Actually using SharedObjects here is entirely unnecessary. We
instantiate the source specifically for this test, so we can just keep state on
the source and assert on that. Will update and also the docs for the harness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17011: [FLINK-23663][table-planner] Reduce state size of ChangelogNormalize
Airblader commented on a change in pull request #17011:
URL: https://github.com/apache/flink/pull/17011#discussion_r697805051
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/PushFilterPastChangelogNormalizeRule.java
##
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.stream;
+
+import org.apache.flink.annotation.Internal;
+import
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalCalc;
+import
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalChangelogNormalize;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexLocalRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexProgramBuilder;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.Pair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static
org.apache.flink.table.planner.plan.utils.RexNodeExtractor.extractRefInputFields;
+
+/**
+ * Pushes primary key filters through a {@link
StreamPhysicalChangelogNormalize ChangelogNormalize}
+ * operator to reduce its state size.
+ *
+ * This rule looks for Calc → ChangelogNormalize where the {@link
StreamPhysicalCalc Calc}
+ * contains a filter condition. The condition is transformed into CNF and then
each conjunction is
+ * tested for whether it affects only primary key columns. If such conditions
exist, they are moved
+ * into a new, separate Calc and pushed through the ChangelogNormalize
operator. ChangelogNormalize
+ * keeps state for every unique key it encounters, thus pushing filters on the
primary key in front
+ * of it helps reduce the size of its state.
+ *
+ * Note that pushing primary key filters is safe to do, but pushing any
other filters would lead
Review comment:
Wonky sentence here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17030: [FLINK-24035][network] Notify the buffer listeners when the local buffer pool receives available notification from the global pool
flinkbot edited a comment on pull request #17030: URL: https://github.com/apache/flink/pull/17030#issuecomment-907558700 ## CI report: * 1ce0b8ec84514371769d20b0807ecc4ee45226d5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22990) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
flinkbot edited a comment on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-906989888 ## CI report: * 39d62e3e08149a884491b039c0cadfbe4bac9fa4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22988) * bc7c93156dd9dbcee4184818b4b4f6e24c8ad234 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22989) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wsry commented on pull request #17030: [FLINK-24035][network] Notify the buffer listeners when the local buffer pool receives available notification from the global pool
wsry commented on pull request #17030: URL: https://github.com/apache/flink/pull/17030#issuecomment-907561521 This fix is still under testing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17030: [FLINK-24035][network] Notify the buffer listeners when the local buffer pool receives available notification from the global pool
flinkbot commented on pull request #17030: URL: https://github.com/apache/flink/pull/17030#issuecomment-907558700 ## CI report: * 1ce0b8ec84514371769d20b0807ecc4ee45226d5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
flinkbot edited a comment on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-906989888 ## CI report: * Unknown: [CANCELED](TBD) * 39d62e3e08149a884491b039c0cadfbe4bac9fa4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22988) * bc7c93156dd9dbcee4184818b4b4f6e24c8ad234 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
flinkbot edited a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-893364145 ## CI report: * c1fceb63708e32d8927bc9350656754e4fb7f012 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22986) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17030: [FLINK-24035][network] Notify the buffer listeners when the local buffer pool receives available notification from the global pool
flinkbot commented on pull request #17030: URL: https://github.com/apache/flink/pull/17030#issuecomment-907557558 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1ce0b8ec84514371769d20b0807ecc4ee45226d5 (Sat Aug 28 03:07:07 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-24035).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24035) Notify the buffer listeners when the local buffer pool receives available notification from the global pool
[ https://issues.apache.org/jira/browse/FLINK-24035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-24035: --- Labels: pull-request-available (was: ) > Notify the buffer listeners when the local buffer pool receives available > notification from the global pool > --- > > Key: FLINK-24035 > URL: https://issues.apache.org/jira/browse/FLINK-24035 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.14.0 >Reporter: Yingjie Cao >Priority: Critical > Labels: pull-request-available > Fix For: 1.14.0 > > > The buffer listeners are not notified when the the local buffer pool receives > available notification from the global pool. This may cause potential > deadlock issue: > # A LocalBufferPool is created, but there is no available buffers in the > global NetworkBufferPool. > # The LocalBufferPool registers an available buffer listener to the global > NetworkBufferPool. > # The BufferManager requests buffers from the LocalBufferPool but no buffer > is available. As a result, it registers an available buffer listener to the > LocalBufferPool. > # A buffer is recycled to the global pool and the local buffer pool is > notified about the available buffer. > # The local buffer pool requests the available buffer from the global pool > but the registered available buffer listener of BufferManager is not notified > and it can never get a chance to be notified so deadlock occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wsry opened a new pull request #17030: [FLINK-24035][network] Notify the buffer listeners when the local buffer pool receives available notification from the global pool
wsry opened a new pull request #17030: URL: https://github.com/apache/flink/pull/17030 ## What is the purpose of the change Previously, The buffer listeners are not notified when the the local buffer pool receives available notification from the global pool. This may cause potential deadlock issue: 1. A LocalBufferPool is created but there is no available buffers in the global NetworkBufferPool. 2. The LocalBufferPool registers an available buffer listener to the global NetworkBufferPool. 3. The BufferManager requests buffers from the LocalBufferPool but no buffer is available. As a result, it registers an available buffer listener to the LocalBufferPool. 4. A buffer is recycled to the global NetworkBufferPool and the LocalBufferPool is notified about the available buffer. 5. The LocalBufferPool requests the available buffer from the global NetworkBufferPool but the registered available buffer listener of BufferManager is not notified and it can never get a chance to be notified so deadlock occurs. This patch fixes this issue by notifying the buffer listeners when the local buffer pool receives available notification from the global pool. ## Brief change log - Notify the buffer listeners when the local buffer pool receives available notification from the global pool. ## Verifying this change This change added tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24035) Notify the buffer listeners when the local buffer pool receives available notification from the global pool
[ https://issues.apache.org/jira/browse/FLINK-24035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-24035: Priority: Critical (was: Major) > Notify the buffer listeners when the local buffer pool receives available > notification from the global pool > --- > > Key: FLINK-24035 > URL: https://issues.apache.org/jira/browse/FLINK-24035 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.14.0 >Reporter: Yingjie Cao >Priority: Critical > Fix For: 1.14.0 > > > The buffer listeners are not notified when the the local buffer pool receives > available notification from the global pool. This may cause potential > deadlock issue: > # A LocalBufferPool is created, but there is no available buffers in the > global NetworkBufferPool. > # The LocalBufferPool registers an available buffer listener to the global > NetworkBufferPool. > # The BufferManager requests buffers from the LocalBufferPool but no buffer > is available. As a result, it registers an available buffer listener to the > LocalBufferPool. > # A buffer is recycled to the global pool and the local buffer pool is > notified about the available buffer. > # The local buffer pool requests the available buffer from the global pool > but the registered available buffer listener of BufferManager is not notified > and it can never get a chance to be notified so deadlock occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24035) Notify the buffer listeners when the local buffer pool receives available notification from the global pool
Yingjie Cao created FLINK-24035: --- Summary: Notify the buffer listeners when the local buffer pool receives available notification from the global pool Key: FLINK-24035 URL: https://issues.apache.org/jira/browse/FLINK-24035 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.14.0 Reporter: Yingjie Cao Fix For: 1.14.0 The buffer listeners are not notified when the the local buffer pool receives available notification from the global pool. This may cause potential deadlock issue: # A LocalBufferPool is created, but there is no available buffers in the global NetworkBufferPool. # The LocalBufferPool registers an available buffer listener to the global NetworkBufferPool. # The BufferManager requests buffers from the LocalBufferPool but no buffer is available. As a result, it registers an available buffer listener to the LocalBufferPool. # A buffer is recycled to the global pool and the local buffer pool is notified about the available buffer. # The local buffer pool requests the available buffer from the global pool but the registered available buffer listener of BufferManager is not notified and it can never get a chance to be notified so deadlock occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
flinkbot edited a comment on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-906989888 ## CI report: * Unknown: [CANCELED](TBD) * 39d62e3e08149a884491b039c0cadfbe4bac9fa4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22988) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17009: [FLINK-23466][network] Fix the bug that buffer listeners may not be notified when recycling buffers
flinkbot edited a comment on pull request #17009: URL: https://github.com/apache/flink/pull/17009#issuecomment-906989802 ## CI report: * 1cda0c085f25dc1b8e9e36d23ac8bf30d85f58e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22987) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22969) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
flinkbot edited a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-893364145 ## CI report: * c1fceb63708e32d8927bc9350656754e4fb7f012 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22986) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23719) Support switch WAL in Flink SQL DDL options for HBase sink
[
https://issues.apache.org/jira/browse/FLINK-23719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406091#comment-17406091
]
Liebing Yu commented on FLINK-23719:
Hi [~jark], I've done the work. Could you take a moment to review it?
> Support switch WAL in Flink SQL DDL options for HBase sink
> ---
>
> Key: FLINK-23719
> URL: https://issues.apache.org/jira/browse/FLINK-23719
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / HBase
>Affects Versions: 1.13.2
>Reporter: Carl
>Assignee: Liebing Yu
>Priority: Major
> Labels: pull-request-available
>
> My application scenario is like this:
> Use Flink SQL to write the batch data of hive table to a new HBase table
> every day.
> In this case, I think it is appropriate to disable WAL for the following
> reasons:
> 1. If an error occurs, i can re-run it. No high security is required.
> 2. The write speed will be greatly improved. No additional writing to the wal
> disk log
> 3. Save disk IO resources of the cluster
>
> However, the HBase official website is described below,
> h4. 70.6.10. Disabling the WAL
> It is possible to disable the WAL, to improve performance in certain specific
> situations. However, disabling the WAL puts your data at risk. The only
> situation where this is recommended is during a bulk load. This is because,
> in the event of a problem, the bulk load can be re-run with no risk of data
> loss.
> The WAL is disabled by calling the HBase client field
> {{Mutation.writeToWAL(false)}}. Use the
> {{Mutation.setDurability(Durability.SKIP_WAL)}} and Mutation.getDurability()
> methods to set and get the field’s value. There is no way to disable the WAL
> for only a specific table.
> [http://hbase.apache.org/2.3/book.html#hbase_default_configurations]
>
> So i think it needs API level support, not properties support.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23462) Translate the abfs documentation to chinese
[ https://issues.apache.org/jira/browse/FLINK-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406089#comment-17406089 ] Liebing Yu commented on FLINK-23462: Hi [~arvid]. I see that the corresponding English documents have been merged. Can you deal with this issue? > Translate the abfs documentation to chinese > --- > > Key: FLINK-23462 > URL: https://issues.apache.org/jira/browse/FLINK-23462 > Project: Flink > Issue Type: Bug > Components: chinese-translation, Documentation >Reporter: Srinivasulu Punuru >Assignee: Liebing Yu >Priority: Major > Labels: pull-request-available > > Translate the documentation changes that were made in this PR to chinese > https://github.com/apache/flink/pull/16559/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] camilesing commented on pull request #16629: [FLINK-23847][connectors][kafka] improve error message when valueDeseriali…
camilesing commented on pull request #16629: URL: https://github.com/apache/flink/pull/16629#issuecomment-907551909 @NicoK hi, can you review it? thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] camilesing removed a comment on pull request #16629: [FLINK-23847][connectors][kafka] improve error message when valueDeseriali…
camilesing removed a comment on pull request #16629: URL: https://github.com/apache/flink/pull/16629#issuecomment-901539082 @tsreaper hi, can you review it? thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
flinkbot edited a comment on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-906989888 ## CI report: * Unknown: [CANCELED](TBD) * 39d62e3e08149a884491b039c0cadfbe4bac9fa4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17009: [FLINK-23466][network] Fix the bug that buffer listeners may not be notified when recycling buffers
flinkbot edited a comment on pull request #17009: URL: https://github.com/apache/flink/pull/17009#issuecomment-906989802 ## CI report: * 1cda0c085f25dc1b8e9e36d23ac8bf30d85f58e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22969) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22987) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
dianfu commented on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-907546462 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wsry commented on pull request #17009: [FLINK-23466][network] Fix the bug that buffer listeners may not be notified when recycling buffers
wsry commented on pull request #17009: URL: https://github.com/apache/flink/pull/17009#issuecomment-907545877 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24025) The components on which Flink depends may contain vulnerabilities. If yes, fix them.
[ https://issues.apache.org/jira/browse/FLINK-24025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406085#comment-17406085 ] mixedfruit commented on FLINK-24025: bzip2 : CVE-2019-12900 and cve-2016-3189 from librocksdbjni-linux-ppc64le.so , librocksdbjni-linux64.so, librocksdbjni-osx.jnilib and librocksdbjni-linux32.so > The components on which Flink depends may contain vulnerabilities. If yes, > fix them. > > > Key: FLINK-24025 > URL: https://issues.apache.org/jira/browse/FLINK-24025 > Project: Flink > Issue Type: Improvement > Components: Build System >Affects Versions: 1.11.3 >Reporter: mixedfruit >Priority: Minor > > In Flink v1.11.3 contains netty(version:3.10.6) > commons-compress(version:1.20) slf4j(version:1.7.15) > cxf-rt-rs-json-basic(version:3.4.0) and bzip2(version:1.0.6). There are many > vulnerabilities, like > CVE-2020-13954,CVE-2021-22696,CVE-2021-30468,CVE-2018-8088, > CVE-2021-21409,CVE-2021-35517 etc. please confirm these version and fix. thx -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
flinkbot edited a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-893364145 ## CI report: * 9740204dd6b8740b09c3c0e5ff612c546522c267 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22985) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22968) * c1fceb63708e32d8927bc9350656754e4fb7f012 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22986) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hailuand commented on pull request #13396: [FLINK-19248] add missing lastJobExecutionResult assignment after job finished in ContexEnvironment
hailuand commented on pull request #13396: URL: https://github.com/apache/flink/pull/13396#issuecomment-907540388 Hi @aljoscha, @jsycdut, We ran into this problem today, and bringing over this patch locally solved our issue 邏 Is there any plans to move this forward? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
flinkbot edited a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-893364145 ## CI report: * 9740204dd6b8740b09c3c0e5ff612c546522c267 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22985) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22968) * c1fceb63708e32d8927bc9350656754e4fb7f012 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal removed a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
RocMarshal removed a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-907527273 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17019: [FLINK-23854][connectors/Kafka] Ensure lingering Kafka transactions are aborted reliably.
flinkbot edited a comment on pull request #17019: URL: https://github.com/apache/flink/pull/17019#issuecomment-907120059 ## CI report: * fa25b27d642ab7e6af356ca1e9738a006d48 UNKNOWN * 88bc7df3d9b9cbe9bc615d55964cfd23de0dc4b5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22984) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
flinkbot edited a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-893364145 ## CI report: * 9740204dd6b8740b09c3c0e5ff612c546522c267 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22968) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22985) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15352) develop MySQLCatalog to connect Flink with MySQL tables and ecosystem
[ https://issues.apache.org/jira/browse/FLINK-15352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406071#comment-17406071 ] Roc Marshal commented on FLINK-15352: - Could someone help me to advance it ? Thank you.:) > develop MySQLCatalog to connect Flink with MySQL tables and ecosystem > -- > > Key: FLINK-15352 > URL: https://issues.apache.org/jira/browse/FLINK-15352 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: Bowen Li >Assignee: Roc Marshal >Priority: Minor > Labels: pull-request-available > Attachments: research-results.tar.gz > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] RocMarshal commented on pull request #17023: [Hotfix][runtime] Reuse the code of 'check savepoint preconditions'.
RocMarshal commented on pull request #17023: URL: https://github.com/apache/flink/pull/17023#issuecomment-907528527 @dianfu Could you help me to review this PR ? Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
RocMarshal commented on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-907527273 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23976) Add additional availability timing metrics to Job lifecycle events
[ https://issues.apache.org/jira/browse/FLINK-23976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405557#comment-17405557 ] Shen Zhu edited comment on FLINK-23976 at 8/27/21, 11:39 PM: - Hey Seth([~sjwiesman] ), I have some ideas for this ticket. *startingTime* (1) If job is in terminal state, return -1 (2) If job doesn't have RUNNING state, return 0 (3) If job is not in terminal state and has RUNNING state, return the diff between RUNNING and CREATED *cancelingTime* (1) If job doesn't have CANCELING state, return 0 (2) If job has CANCELING state, then check if job has CANCELD state, if yes then return the diff between CANCELD and CALCELING, if no then return System.currentTimeMillis() - CALCELING timestamp. What do you think about it, if it looks good do you, could you please assign this ticket to me? Thanks for your help! was (Author: shenzhu0127): Hey Seth([~sjwiesman] ), I have some ideas for this ticket. *startingTime* (1) If job is in terminal state, return -1 (2) If job doesn't have RUNNING state, return 0 (3) If job is not in terminal state and has RUNNING state, return the diff between RUNNING and CREATED *cancelingTime* (1) If job is in terminal state, return -1 (2) If job doesn't have CANCELING state, return 0 (3) If job is not in terminal state and has CANCELING state, then check if job has CANCELD state, if yes then return the diff between CANCELD and CALCELING, if no then return System.currentTimeMillis() - CALCELING timestamp. What do you think about it, if it looks good do you, could you please assign this ticket to me? Thanks for your help! > Add additional availability timing metrics to Job lifecycle events > -- > > Key: FLINK-23976 > URL: https://issues.apache.org/jira/browse/FLINK-23976 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: Seth Wiesman >Priority: Major > > Flink currently contains a number of availability lifecycle metrics[1] > showing how long it takes to move through different job status'. We propose > adding two additional metrics; startingTime, and cancellingTime (open to > bikeshedding on the metric names). > > * startingTime is the time it takes a job to get to running. > * cancellingTime is the time spent in status CANCELLING > > > [1]https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/metrics/#availability -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23158) Source transformation is not added to the StreamExecutionEnvironment explicitly
[
https://issues.apache.org/jira/browse/FLINK-23158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-23158:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Source transformation is not added to the StreamExecutionEnvironment
> explicitly
> ---
>
> Key: FLINK-23158
> URL: https://issues.apache.org/jira/browse/FLINK-23158
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Priority: Major
> Labels: pull-request-available, stale-major
>
> Currently for the implementation of `StreamExecutionEnvironment#fromSource()`
> and `StreamExecutionEnvironment#addSource()`, the SourceTransformation is not
> added to the transformation list explicitly, this make the job with a single
> source could not run directly. For example,
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.addSource(new SourceFunction() {
> @Override
> public void run(SourceContext sourceContext)
> throws Exception {
> }
> @Override
> public void cancel() {
> }
> });
> env.execute();
> {code}
> would throws the exception:
> {code:java}
> Exception in thread "main" java.lang.IllegalStateException: No operators
> defined in streaming topology. Cannot execute.
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraphGenerator(StreamExecutionEnvironment.java:2019)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2010)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:1995)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1834)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1817)
> at test.SingleSourceTest.main(SingleSourceTest.java:41)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21302) Fix NPE when use row_number() in over agg
[
https://issues.apache.org/jira/browse/FLINK-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-21302:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Fix NPE when use row_number() in over agg
> -
>
> Key: FLINK-21302
> URL: https://issues.apache.org/jira/browse/FLINK-21302
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: JING ZHANG
>Assignee: JING ZHANG
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.13.0, 1.14.0
>
>
> `NullPointException` would be thrown out if SQL contains row_number() in Over
> Aggregate.
> {code:scala}
> @Test
> def testRowNumberOnOver(): Unit = {
> val t = failingDataSource(TestData.tupleData5)
> .toTable(tEnv, 'a, 'b, 'c, 'd, 'e, 'proctime.proctime)
> tEnv.registerTable("MyTable", t)
> val sqlQuery = "SELECT a, ROW_NUMBER() OVER (PARTITION BY a ORDER BY
> proctime()) FROM MyTable"
> val sink = new TestingAppendSink
> tEnv.sqlQuery(sqlQuery).toAppendStream[Row].addSink(sink)
> env.execute()
> }{code}
>
> The following exception would be thrown out.
> {code:java}
> java.lang.NullPointerExceptionjava.lang.NullPointerException at
> scala.collection.mutable.ArrayOps$ofInt$.length$extension(ArrayOps.scala:240)
> at scala.collection.mutable.ArrayOps$ofInt.length(ArrayOps.scala:240) at
> scala.collection.SeqLike$class.size(SeqLike.scala:106) at
> scala.collection.mutable.ArrayOps$ofInt.size(ArrayOps.scala:234) at
> scala.collection.mutable.Builder$class.sizeHint(Builder.scala:69) at
> scala.collection.mutable.ArrayBuilder.sizeHint(ArrayBuilder.scala:22) at
> scala.collection.TraversableLike$class.builder$1(TraversableLike.scala:230)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:233) at
> scala.collection.mutable.ArrayOps$ofInt.map(ArrayOps.scala:234) at
> org.apache.flink.table.planner.codegen.agg.DeclarativeAggCodeGen.(DeclarativeAggCodeGen.scala:82)
> at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator$$anonfun$3.apply(AggsHandlerCodeGenerator.scala:222)
> at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator$$anonfun$3.apply(AggsHandlerCodeGenerator.scala:214)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator.initialAggregateInformation(AggsHandlerCodeGenerator.scala:214)
> at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator.generateAggsHandler(AggsHandlerCodeGenerator.scala:325)
> at
> org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecOverAggregate.createUnboundedOverProcessFunction(StreamExecOverAggregate.java:262)
> at
> org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecOverAggregate.translateToPlanInternal(StreamExecOverAggregate.java:154)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:128)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:247)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:65)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:128)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:247)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecLegacySink.translateToTransformation(CommonExecLegacySink.java:167)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecLegacySink.translateToPlanInternal(CommonExecLegacySink.java:136)
> at
>
[jira] [Updated] (FLINK-23167) Port Kinesis Table API e2e tests to release-1.12 branch
[ https://issues.apache.org/jira/browse/FLINK-23167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-23167: --- Labels: kinesis stale-major (was: kinesis) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Port Kinesis Table API e2e tests to release-1.12 branch > --- > > Key: FLINK-23167 > URL: https://issues.apache.org/jira/browse/FLINK-23167 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Affects Versions: 1.12.6 >Reporter: Emre Kartoglu >Priority: Major > Labels: kinesis, stale-major > Original Estimate: 24h > Remaining Estimate: 24h > > https://issues.apache.org/jira/browse/FLINK-20042 added e2e tests for the > Kinesis Table API. This was only done for versions >=1.13 however. > We need to port these tests to the release-1.12 branch as version 1.12 > supports the same functionality that needs the same (or a similar) test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-10705) Rework Flink Web Dashboard
[ https://issues.apache.org/jira/browse/FLINK-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-10705: --- Labels: auto-deprioritized-major auto-unassigned pull-request-available (was: auto-unassigned pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Rework Flink Web Dashboard > -- > > Key: FLINK-10705 > URL: https://issues.apache.org/jira/browse/FLINK-10705 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.6.2, 1.8.0 >Reporter: Fabian Wollert >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Attachments: 3rdpartylicenses.txt, image-2018-10-29-09-17-24-115.png, > snapshot.jpeg > > Time Spent: 20m > Remaining Estimate: 0h > > The Flink Dashboard is very simple currently and should get updated. This is > the umbrella ticket for other tickets regarding this. Please check the > sub-tickets for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17019: [FLINK-23854][connectors/Kafka] Ensure lingering Kafka transactions are aborted reliably.
flinkbot edited a comment on pull request #17019: URL: https://github.com/apache/flink/pull/17019#issuecomment-907120059 ## CI report: * fa25b27d642ab7e6af356ca1e9738a006d48 UNKNOWN * 58821b8a12ba72f74ce4ad6dfddfa91f6ccfac42 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22959) * 88bc7df3d9b9cbe9bc615d55964cfd23de0dc4b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22984) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17027: [FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot edited a comment on pull request #17027: URL: https://github.com/apache/flink/pull/17027#issuecomment-907420052 ## CI report: * 9c9d08b774aaae21355baed5a79379ccb4ff922c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22978) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17019: [FLINK-23854][connectors/Kafka] Ensure lingering Kafka transactions are aborted reliably.
flinkbot edited a comment on pull request #17019: URL: https://github.com/apache/flink/pull/17019#issuecomment-907120059 ## CI report: * fa25b27d642ab7e6af356ca1e9738a006d48 UNKNOWN * 58821b8a12ba72f74ce4ad6dfddfa91f6ccfac42 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22959) * 88bc7df3d9b9cbe9bc615d55964cfd23de0dc4b5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24007) Support Avro timestamp conversion with precision greater than three
[
https://issues.apache.org/jira/browse/FLINK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406049#comment-17406049
]
Xingcan Cui commented on FLINK-24007:
-
Will close this since it's duplicated with
https://issues.apache.org/jira/browse/FLINK-23589
> Support Avro timestamp conversion with precision greater than three
> ---
>
> Key: FLINK-24007
> URL: https://issues.apache.org/jira/browse/FLINK-24007
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.13.2
>Reporter: Xingcan Cui
>Priority: Major
>
> {{AvroSchemaConverter.convertToSchema()}} doesn't support timestamp with
> precision > 3 now. This seems to be a bug and should be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-24007) Support Avro timestamp conversion with precision greater than three
[
https://issues.apache.org/jira/browse/FLINK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui closed FLINK-24007.
---
Resolution: Duplicate
> Support Avro timestamp conversion with precision greater than three
> ---
>
> Key: FLINK-24007
> URL: https://issues.apache.org/jira/browse/FLINK-24007
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.13.2
>Reporter: Xingcan Cui
>Priority: Major
>
> {{AvroSchemaConverter.convertToSchema()}} doesn't support timestamp with
> precision > 3 now. This seems to be a bug and should be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24030) PulsarSourceITCase>SourceTestSuiteBase.testMultipleSplits failed
[
https://issues.apache.org/jira/browse/FLINK-24030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406039#comment-17406039
]
Yufan Sheng commented on FLINK-24030:
-
[~pnowojski] All the build failure are listed in the comment. I have seen some
build failure is the same with you.
> PulsarSourceITCase>SourceTestSuiteBase.testMultipleSplits failed
>
>
> Key: FLINK-24030
> URL: https://issues.apache.org/jira/browse/FLINK-24030
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Piotr Nowojski
>Priority: Major
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22936=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461
> root cause:
> {noformat}
> Aug 27 09:41:42 Caused by:
> org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException:
> Consumer not found
> Aug 27 09:41:42 at
> org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987)
> Aug 27 09:41:42 at
> org.apache.pulsar.client.impl.PulsarClientImpl.close(PulsarClientImpl.java:658)
> Aug 27 09:41:42 at
> org.apache.flink.connector.pulsar.source.reader.source.PulsarSourceReaderBase.close(PulsarSourceReaderBase.java:83)
> Aug 27 09:41:42 at
> org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader.close(PulsarOrderedSourceReader.java:170)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.api.operators.SourceOperator.close(SourceOperator.java:308)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:141)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:127)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1015)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:859)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:747)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> {noformat}
> Top level error:
> {noformat}
> WARNING: The following warnings have been detected: WARNING: Return type,
> java.util.Map org.apache.pulsar.common.policies.data.NamespaceIsolationData>, of method,
> public java.util.Map org.apache.pulsar.common.policies.data.NamespaceIsolationData>
> org.apache.pulsar.broker.admin.impl.ClustersBase.getNamespaceIsolationPolicies(java.lang.String)
> throws java.lang.Exception, is not resolvable to a concrete type.
> Aug 27 09:41:42 [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 357.849 s <<< FAILURE! - in
> org.apache.flink.connector.pulsar.source.PulsarSourceITCase
> Aug 27 09:41:42 [ERROR] testMultipleSplits{TestEnvironment,
> ExternalContext}[1] Time elapsed: 5.391 s <<< ERROR!
> Aug 27 09:41:42 java.lang.RuntimeException: Failed to fetch next result
> Aug 27 09:41:42 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> Aug 27 09:41:42 at
> org.apache.flink.connectors.test.common.utils.TestDataMatchers$MultipleSplitDataMatcher.matchesSafely(TestDataMatchers.java:151)
> Aug 27 09:41:42 at
> org.apache.flink.connectors.test.common.utils.TestDataMatchers$MultipleSplitDataMatcher.matchesSafely(TestDataMatchers.java:133)
> Aug 27 09:41:42 at
> org.hamcrest.TypeSafeDiagnosingMatcher.matches(TypeSafeDiagnosingMatcher.java:55)
> Aug 27 09:41:42 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:12)
> Aug 27 09:41:42 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> Aug 27 09:41:42 at
> org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase.testMultipleSplits(SourceTestSuiteBase.java:156)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17029: [BP-1.12][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot edited a comment on pull request #17029: URL: https://github.com/apache/flink/pull/17029#issuecomment-907420779 ## CI report: * da36e822462676e9668ef583d17cc65cc9916bdc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22980) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
flinkbot edited a comment on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-904042149 ## CI report: * 04c9fc636bd6f07bf47654b5fc6d80c939169bc4 UNKNOWN * 7ef526d579a122a4545d3f872ec97e99a820e07d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24005) Resource requirements declaration may be incorrect if JobMaster disconnects with a TaskManager with available slots in the SlotPool
[
https://issues.apache.org/jira/browse/FLINK-24005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406028#comment-17406028
]
Till Rohrmann commented on FLINK-24005:
---
I think the problem is the following: When we call
{{DeclarativeSlotPoolService.releaseTaskManager}}, then we release the slots in
the {{DecalarativeSlotPool}}. This will remove all bookkeeping information from
the pool and fail the payloads of the slots. Failing the payloads will
eventually call {{DeclarativeSlotPoolBridge.releaseSlot(SlotRequestId,
Throwable)}}. In this method we call call
{{DeclarativeSlotPool.freeReservedSlot}} which returns the {{ResourceCounter}}
if there was a reserved slot that we could free. However, in this case, we have
already removed the slot and, thus, the {{DeclarativeSlotPool}} no longer knows
about the released slot and their resource profile. I think that is why we
reduce the resource requirements by all released slots when
{{DeclarativeSlotPoolService.releaseTaskManager}} is called.
As shown by Zhu Zhu, this is not correct and we would actually only want to
reduce the resource requirements by the number of assigned slots. Conceptually,
we would like to let {{DeclarativeSlotPoolBridge.releaseSlot}} do the correct
accounting if we knew the {{ResourceProfile}} of the previously released slots.
> Resource requirements declaration may be incorrect if JobMaster disconnects
> with a TaskManager with available slots in the SlotPool
> ---
>
> Key: FLINK-24005
> URL: https://issues.apache.org/jira/browse/FLINK-24005
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.12.5, 1.13.2
>Reporter: Zhu Zhu
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.14.0, 1.12.6, 1.13.3
>
> Attachments: decrease_resource_requirements.log
>
>
> When a TaskManager disconnects with JobMaster, it will trigger the
> `DeclarativeSlotPoolService#decreaseResourceRequirementsBy()` for all the
> slots that are registered to the JobMaster from the TaskManager. If the slots
> are still available, i.e. not assigned to any task, the
> `decreaseResourceRequirementsBy` may lead to incorrect resource requirements
> declaration.
> For example, there is one job with 3 source tasks only. It requires 3 slots
> and declares for 3 slots. Initially all the tasks are running. Suddenly one
> task failed and waits for some delay before restarting. The previous slot is
> returned to the SlotPool. Now the job requires 2 slots and declares for 2
> slots. At this moment, the TaskManager of that returned slot get lost. After
> the triggered `decreaseResourceRequirementsBy`, the job only declares for 1
> slot. Finally, when the failed task starts to re-schedule, the job will
> declare for 2 slots while it actually needs 3 slots.
> The attached log of a real job and logs of the added test in
> https://github.com/zhuzhurk/flink/commit/59ca0ac5fa9c77b97c6e8a43dcc53ca8a0ad6c37
> can demonstrate this case.
> Note that the real job is configured with a large
> "restart-strategy.fixed-delay.delay" and and large "slot.idle.timeout". So
> possibly in production it is a rare case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] AHeise commented on a change in pull request #17019: [FLINK-23854][connectors/Kafka] Ensure lingering Kafka transactions are aborted reliably.
AHeise commented on a change in pull request #17019:
URL: https://github.com/apache/flink/pull/17019#discussion_r697712620
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##
@@ -254,11 +254,9 @@ private void assignNonFinishedStateToTask(
operatorID.getGeneratedOperatorID(), operatorSubtaskState);
}
-if (!statelessTask) {
-JobManagerTaskRestore taskRestore =
-new JobManagerTaskRestore(restoreCheckpointId, taskState);
-currentExecutionAttempt.setInitialState(taskRestore);
-}
+JobManagerTaskRestore taskRestore =
+new JobManagerTaskRestore(restoreCheckpointId, taskState);
+currentExecutionAttempt.setInitialState(taskRestore);
Review comment:
I adjusted `ExecutionVertex#getPreferredLocationBasedOnState` to also
check if there is state. That should restore the old behavior. (`empty` on
fresh starts and stateless tasks) So good catch!
For `TaskStateManagerImpl`, the `SequentialChannelStateReaderImpl` already
creates empty state when initialized, so that is effectively unchanged. For
`getInputRescalingDescriptor`, `getOutputRescalingDescriptor`, and
`isFinishedOnRestore` the return value is also the same (NO_RESCALE / false).
`prioritizedOperatorState` returns a different
`PrioritizedOperatorSubtaskState` where all fields are the same as before
except the newly added `restoreCheckpointId`. I have not seen anything that
directly checks for the
`PrioritizedOperatorSubtaskState.EMPTY_NON_RESTORED_INSTANCE` on equality.
TL;DR there is no change in `TaskStateManagerImpl`.
Apart from that almost all `@Nullable` remain valid as it is still null if
the job is started anew. My change is only for stateless tasks on recovery. So
`Execution#setInitialState` will now be called for every task but only on
recovery. The field must stay `@Nullable` and with it all depending paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24005) Resource requirements declaration may be incorrect if JobMaster disconnects with a TaskManager with available slots in the SlotPool
[
https://issues.apache.org/jira/browse/FLINK-24005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406024#comment-17406024
]
Till Rohrmann commented on FLINK-24005:
---
This is a very good finding that we need to fix. I am also not sure why we
can't simply rely on the {{DeclarativeSlotPoolBridge.releaseSlot(SlotRequestId
slotRequestId, Throwable cause)}}. There must be something that prevents this
method from being called if the slot is failed via
{{DeclarativeSlotPoolService.releaseTaskManager}}.
> Resource requirements declaration may be incorrect if JobMaster disconnects
> with a TaskManager with available slots in the SlotPool
> ---
>
> Key: FLINK-24005
> URL: https://issues.apache.org/jira/browse/FLINK-24005
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.12.5, 1.13.2
>Reporter: Zhu Zhu
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.14.0, 1.12.6, 1.13.3
>
> Attachments: decrease_resource_requirements.log
>
>
> When a TaskManager disconnects with JobMaster, it will trigger the
> `DeclarativeSlotPoolService#decreaseResourceRequirementsBy()` for all the
> slots that are registered to the JobMaster from the TaskManager. If the slots
> are still available, i.e. not assigned to any task, the
> `decreaseResourceRequirementsBy` may lead to incorrect resource requirements
> declaration.
> For example, there is one job with 3 source tasks only. It requires 3 slots
> and declares for 3 slots. Initially all the tasks are running. Suddenly one
> task failed and waits for some delay before restarting. The previous slot is
> returned to the SlotPool. Now the job requires 2 slots and declares for 2
> slots. At this moment, the TaskManager of that returned slot get lost. After
> the triggered `decreaseResourceRequirementsBy`, the job only declares for 1
> slot. Finally, when the failed task starts to re-schedule, the job will
> declare for 2 slots while it actually needs 3 slots.
> The attached log of a real job and logs of the added test in
> https://github.com/zhuzhurk/flink/commit/59ca0ac5fa9c77b97c6e8a43dcc53ca8a0ad6c37
> can demonstrate this case.
> Note that the real job is configured with a large
> "restart-strategy.fixed-delay.delay" and and large "slot.idle.timeout". So
> possibly in production it is a rare case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16987: [FLINK-23968][tests] Remove unused code from TestBaseUtils
flinkbot edited a comment on pull request #16987: URL: https://github.com/apache/flink/pull/16987#issuecomment-905577505 ## CI report: * b873c48ed77b2bcdcb950861fa35e6299b6d9e81 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22977) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
flinkbot edited a comment on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-904042149 ## CI report: * 04c9fc636bd6f07bf47654b5fc6d80c939169bc4 UNKNOWN * 7ef526d579a122a4545d3f872ec97e99a820e07d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17028: [BP-1.13][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot edited a comment on pull request #17028: URL: https://github.com/apache/flink/pull/17028#issuecomment-907420483 ## CI report: * 9c0ba6c38c664e0c2f87daa3ae086a4d47003d7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22979) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17029: [BP-1.12][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot edited a comment on pull request #17029: URL: https://github.com/apache/flink/pull/17029#issuecomment-907420779 ## CI report: * da36e822462676e9668ef583d17cc65cc9916bdc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22980) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17027: [FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot edited a comment on pull request #17027: URL: https://github.com/apache/flink/pull/17027#issuecomment-907420052 ## CI report: * 9c9d08b774aaae21355baed5a79379ccb4ff922c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22978) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16915: [FLINK-9925][tests] Harden ClientTest by making handler shareable
flinkbot edited a comment on pull request #16915: URL: https://github.com/apache/flink/pull/16915#issuecomment-902862695 ## CI report: * 54cff8c8965e8277bc66ce800a4b2f134b03fccd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22974) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16575: [FLINK-23139][state] Add TaskStateRegistry
flinkbot edited a comment on pull request #16575: URL: https://github.com/apache/flink/pull/16575#issuecomment-885295771 ## CI report: * eed9672d2ac736e5261d4e3511ccacab1b77d683 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22973) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-24027) FileSystems list excessive dependencies in NOTICE
[
https://issues.apache.org/jira/browse/FLINK-24027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ingo Bürk reassigned FLINK-24027:
-
Assignee: Ingo Bürk
> FileSystems list excessive dependencies in NOTICE
> -
>
> Key: FLINK-24027
> URL: https://issues.apache.org/jira/browse/FLINK-24027
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.14.0
>Reporter: Chesnay Schepler
>Assignee: Ingo Bürk
>Priority: Blocker
> Fix For: 1.14.0
>
>
> The LicenseChecker finds several dependencies that are listed in the NOTICE
> but do not show up in the shade-plugin output. It could be that after the
> recent AWS/Hadoop bumps these are no longer being bundled (needs
> confirmation!).
> {code}
> 17:05:14,651 WARN NoticeFileChecker [] - Dependency
> com.fasterxml.jackson.core:jackson-annotations:2.12.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-azure-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,651 WARN NoticeFileChecker [] - Dependency
> com.fasterxml.jackson.core:jackson-databind:2.12.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-azure-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 is mentioned in
> NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> org.wildfly.openssl:wildfly-openssl:1.0.7.Final is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> commons-lang:commons-lang:2.6 is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,741 WARN NoticeFileChecker [] - Dependency
> commons-lang:commons-lang:2.6 is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 is mentioned in
> NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> org.wildfly.openssl:wildfly-openssl:1.0.7.Final is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> commons-lang:commons-lang:2.6 is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24027) FileSystems list excessive dependencies in NOTICE
[
https://issues.apache.org/jira/browse/FLINK-24027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405999#comment-17405999
]
Ingo Bürk commented on FLINK-24027:
---
Thanks, [~chesnay]. I'll look into this.
> FileSystems list excessive dependencies in NOTICE
> -
>
> Key: FLINK-24027
> URL: https://issues.apache.org/jira/browse/FLINK-24027
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.14.0
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.14.0
>
>
> The LicenseChecker finds several dependencies that are listed in the NOTICE
> but do not show up in the shade-plugin output. It could be that after the
> recent AWS/Hadoop bumps these are no longer being bundled (needs
> confirmation!).
> {code}
> 17:05:14,651 WARN NoticeFileChecker [] - Dependency
> com.fasterxml.jackson.core:jackson-annotations:2.12.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-azure-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,651 WARN NoticeFileChecker [] - Dependency
> com.fasterxml.jackson.core:jackson-databind:2.12.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-azure-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 is mentioned in
> NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> org.wildfly.openssl:wildfly-openssl:1.0.7.Final is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,652 WARN NoticeFileChecker [] - Dependency
> commons-lang:commons-lang:2.6 is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,741 WARN NoticeFileChecker [] - Dependency
> commons-lang:commons-lang:2.6 is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 is mentioned in
> NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 is mentioned in NOTICE
> file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> org.wildfly.openssl:wildfly-openssl:1.0.7.Final is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> 17:05:14,743 WARN NoticeFileChecker [] - Dependency
> commons-lang:commons-lang:2.6 is mentioned in NOTICE file
> /__w/1/s/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/NOTICE,
> but is not expected there
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17029: [BP-1.12][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot commented on pull request #17029: URL: https://github.com/apache/flink/pull/17029#issuecomment-907420779 ## CI report: * da36e822462676e9668ef583d17cc65cc9916bdc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17028: [BP-1.13][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot commented on pull request #17028: URL: https://github.com/apache/flink/pull/17028#issuecomment-907420483 ## CI report: * 9c0ba6c38c664e0c2f87daa3ae086a4d47003d7e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17027: [FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot commented on pull request #17027: URL: https://github.com/apache/flink/pull/17027#issuecomment-907420052 ## CI report: * 9c9d08b774aaae21355baed5a79379ccb4ff922c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
flinkbot edited a comment on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-904042149 ## CI report: * 04c9fc636bd6f07bf47654b5fc6d80c939169bc4 UNKNOWN * 454b7277df57678f94dc74901a54b3b22e241276 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22689) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22975) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22761) * 7ef526d579a122a4545d3f872ec97e99a820e07d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24005) Resource requirements declaration may be incorrect if JobMaster disconnects with a TaskManager with available slots in the SlotPool
[ https://issues.apache.org/jira/browse/FLINK-24005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-24005: -- Fix Version/s: 1.13.3 1.12.6 1.14.0 > Resource requirements declaration may be incorrect if JobMaster disconnects > with a TaskManager with available slots in the SlotPool > --- > > Key: FLINK-24005 > URL: https://issues.apache.org/jira/browse/FLINK-24005 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.14.0, 1.12.5, 1.13.2 >Reporter: Zhu Zhu >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.14.0, 1.12.6, 1.13.3 > > Attachments: decrease_resource_requirements.log > > > When a TaskManager disconnects with JobMaster, it will trigger the > `DeclarativeSlotPoolService#decreaseResourceRequirementsBy()` for all the > slots that are registered to the JobMaster from the TaskManager. If the slots > are still available, i.e. not assigned to any task, the > `decreaseResourceRequirementsBy` may lead to incorrect resource requirements > declaration. > For example, there is one job with 3 source tasks only. It requires 3 slots > and declares for 3 slots. Initially all the tasks are running. Suddenly one > task failed and waits for some delay before restarting. The previous slot is > returned to the SlotPool. Now the job requires 2 slots and declares for 2 > slots. At this moment, the TaskManager of that returned slot get lost. After > the triggered `decreaseResourceRequirementsBy`, the job only declares for 1 > slot. Finally, when the failed task starts to re-schedule, the job will > declare for 2 slots while it actually needs 3 slots. > The attached log of a real job and logs of the added test in > https://github.com/zhuzhurk/flink/commit/59ca0ac5fa9c77b97c6e8a43dcc53ca8a0ad6c37 > can demonstrate this case. > Note that the real job is configured with a large > "restart-strategy.fixed-delay.delay" and and large "slot.idle.timeout". So > possibly in production it is a rare case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17029: [BP-1.12][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot commented on pull request #17029: URL: https://github.com/apache/flink/pull/17029#issuecomment-907397324 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit da36e822462676e9668ef583d17cc65cc9916bdc (Fri Aug 27 18:37:24 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17028: [BP-1.13][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot commented on pull request #17028: URL: https://github.com/apache/flink/pull/17028#issuecomment-907396112 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9c0ba6c38c664e0c2f87daa3ae086a4d47003d7e (Fri Aug 27 18:35:05 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17027: [FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
flinkbot commented on pull request #17027: URL: https://github.com/apache/flink/pull/17027#issuecomment-907396134 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9c9d08b774aaae21355baed5a79379ccb4ff922c (Fri Aug 27 18:35:08 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann opened a new pull request #17029: [BP-1.12][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
tillrohrmann opened a new pull request #17029: URL: https://github.com/apache/flink/pull/17029 Backport of #17027 to `release-1.12`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann opened a new pull request #17028: [BP-1.13][FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
tillrohrmann opened a new pull request #17028: URL: https://github.com/apache/flink/pull/17028 Backport of #17027 to `release-1.13`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann opened a new pull request #17027: [FLINK-23954][tests] Fix e2e test test_queryable_state_restart_tm.sh
tillrohrmann opened a new pull request #17027: URL: https://github.com/apache/flink/pull/17027 The problem was that we output the number of state entries at the time when the checkpoint is confirmed. However, at this point in time we might have already added more elements to the map. Hence, we risk reporting more elements than are actually contained in the checkpoint. The solution is to remember what the element count was when creating the checkpoint. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-23954) Queryable state (rocksdb) with TM restart end-to-end test fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-23954:
-
Assignee: Till Rohrmann
> Queryable state (rocksdb) with TM restart end-to-end test fails on azure
>
>
> Key: FLINK-23954
> URL: https://issues.apache.org/jira/browse/FLINK-23954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.12.6, 1.13.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22714=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=11090
> {code}
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 Running 'Queryable state (rocksdb) with TM restart end-to-end
> test'
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-17362481511
> Aug 24 09:54:17 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.14-SNAPSHOT-bin/flink-1.14-SNAPSHOT
> Aug 24 09:54:17 Adding flink-queryable-state-runtime to lib/
> Aug 24 09:54:17 Starting cluster.
> Aug 24 09:54:18 Starting standalonesession daemon on host fv-az123-794.
> Aug 24 09:54:19 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:19 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:20 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:21 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:22 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:23 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:24 Dispatcher REST endpoint is up.
> Aug 24 09:54:31 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:31 Starting to wait for completion of 10 checkpoints
> Aug 24 09:54:31 2/10 completed checkpoints
> Aug 24 09:54:33 6/10 completed checkpoints
> Aug 24 09:54:35 6/10 completed checkpoints
> Aug 24 09:54:37 SERVER: 127.0.0.1
> Aug 24 09:54:37 PORT: 9069
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:38 MapState has 17 entries
> Aug 24 09:54:38 TaskManager 412715 killed.
> Aug 24 09:54:38 Number of running task managers 1 is not yet 0.
> Aug 24 09:54:42 Number of running task managers has reached 0.
> Aug 24 09:54:42 Latest snapshot count was 22
> Aug 24 09:54:43 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:43 Number of running task managers 0 is not yet 1.
> Aug 24 09:54:47 Number of running task managers has reached 1.
> Aug 24 09:54:49 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:49 Starting to wait for completion of 16 checkpoints
> Aug 24 09:54:49 11/16 completed checkpoints
> Aug 24 09:54:51 11/16 completed checkpoints
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:54 after: 21
> Aug 24 09:54:54 An error occurred
> Aug 24 09:54:54 [FAIL] Test script contains errors.
> Aug 24 09:54:54 Checking of logs skipped.
> Aug 24 09:54:54
> Aug 24 09:54:54 [FAIL] 'Queryable state (rocksdb) with TM restart end-to-end
> test' failed after 0 minutes and 37 seconds! Test exited with exit code 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24030) PulsarSourceITCase>SourceTestSuiteBase.testMultipleSplits failed
[
https://issues.apache.org/jira/browse/FLINK-24030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405962#comment-17405962
]
Piotr Nowojski commented on FLINK-24030:
Are you sure? The exception stack trace is different.
> PulsarSourceITCase>SourceTestSuiteBase.testMultipleSplits failed
>
>
> Key: FLINK-24030
> URL: https://issues.apache.org/jira/browse/FLINK-24030
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Piotr Nowojski
>Priority: Major
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22936=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461
> root cause:
> {noformat}
> Aug 27 09:41:42 Caused by:
> org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException:
> Consumer not found
> Aug 27 09:41:42 at
> org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987)
> Aug 27 09:41:42 at
> org.apache.pulsar.client.impl.PulsarClientImpl.close(PulsarClientImpl.java:658)
> Aug 27 09:41:42 at
> org.apache.flink.connector.pulsar.source.reader.source.PulsarSourceReaderBase.close(PulsarSourceReaderBase.java:83)
> Aug 27 09:41:42 at
> org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader.close(PulsarOrderedSourceReader.java:170)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.api.operators.SourceOperator.close(SourceOperator.java:308)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:141)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:127)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1015)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:859)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:747)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> Aug 27 09:41:42 at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> {noformat}
> Top level error:
> {noformat}
> WARNING: The following warnings have been detected: WARNING: Return type,
> java.util.Map org.apache.pulsar.common.policies.data.NamespaceIsolationData>, of method,
> public java.util.Map org.apache.pulsar.common.policies.data.NamespaceIsolationData>
> org.apache.pulsar.broker.admin.impl.ClustersBase.getNamespaceIsolationPolicies(java.lang.String)
> throws java.lang.Exception, is not resolvable to a concrete type.
> Aug 27 09:41:42 [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 357.849 s <<< FAILURE! - in
> org.apache.flink.connector.pulsar.source.PulsarSourceITCase
> Aug 27 09:41:42 [ERROR] testMultipleSplits{TestEnvironment,
> ExternalContext}[1] Time elapsed: 5.391 s <<< ERROR!
> Aug 27 09:41:42 java.lang.RuntimeException: Failed to fetch next result
> Aug 27 09:41:42 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> Aug 27 09:41:42 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> Aug 27 09:41:42 at
> org.apache.flink.connectors.test.common.utils.TestDataMatchers$MultipleSplitDataMatcher.matchesSafely(TestDataMatchers.java:151)
> Aug 27 09:41:42 at
> org.apache.flink.connectors.test.common.utils.TestDataMatchers$MultipleSplitDataMatcher.matchesSafely(TestDataMatchers.java:133)
> Aug 27 09:41:42 at
> org.hamcrest.TypeSafeDiagnosingMatcher.matches(TypeSafeDiagnosingMatcher.java:55)
> Aug 27 09:41:42 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:12)
> Aug 27 09:41:42 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> Aug 27 09:41:42 at
> org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase.testMultipleSplits(SourceTestSuiteBase.java:156)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23776) Performance regression on 14.08.2021 in FLIP-27
[
https://issues.apache.org/jira/browse/FLINK-23776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405965#comment-17405965
]
Arvid Heise commented on FLINK-23776:
-
Merged an improvement as fd429d084264357d3cfbfd8b2b62cf8327a8fd79.
> Performance regression on 14.08.2021 in FLIP-27
> ---
>
> Key: FLINK-23776
> URL: https://issues.apache.org/jira/browse/FLINK-23776
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Benchmarks
>Affects Versions: 1.14.0
>Reporter: Piotr Nowojski
>Assignee: Arvid Heise
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> http://codespeed.dak8s.net:8000/timeline/?ben=mapSink.F27_UNBOUNDED=2
> http://codespeed.dak8s.net:8000/timeline/?ben=mapRebalanceMapSink.F27_UNBOUNDED=2
> {noformat}
> git ls 7b60a964b1..7f3636f6b4
> 7f3636f6b4f [2 days ago] [FLINK-23652][connectors] Adding common source
> metrics. [Arvid Heise]
> 97c8f72b813 [3 months ago] [FLINK-23652][connectors] Adding common sink
> metrics. [Arvid Heise]
> 48da20e8f88 [3 months ago] [FLINK-23652][test] Adding InMemoryMetricReporter
> and using it by default in MiniClusterResource. [Arvid Heise]
> 63ee60859ca [3 months ago] [FLINK-23652][core/metrics] Extract
> Operator(IO)MetricGroup interfaces and expose them in RuntimeContext [Arvid
> Heise]
> 5d5e39b614b [2 days ago] [refactor][connectors] Only use
> MockSplitReader.Builder for instantiation. [Arvid Heise]
> b927035610c [3 months ago] [refactor][core] Extract common context creation
> in CollectionExecutor [Arvid Heise]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #16990: [FLINK-23776][datastream] Optimize source metric calculation
AHeise merged pull request #16990: URL: https://github.com/apache/flink/pull/16990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
flinkbot edited a comment on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-906989888 ## CI report: * 2eccdcb7ea2438682caad05ffa50be41af48af5b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22972) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16987: [FLINK-23968][tests] Remove unused code from TestBaseUtils
flinkbot edited a comment on pull request #16987: URL: https://github.com/apache/flink/pull/16987#issuecomment-905577505 ## CI report: * 480855e82a32496f64bcbf05dd4ea7ce205186db Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22840) * b873c48ed77b2bcdcb950861fa35e6299b6d9e81 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22977) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
flinkbot edited a comment on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-904042149 ## CI report: * 04c9fc636bd6f07bf47654b5fc6d80c939169bc4 UNKNOWN * 454b7277df57678f94dc74901a54b3b22e241276 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22689) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22975) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22761) * 7ef526d579a122a4545d3f872ec97e99a820e07d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16915: [FLINK-9925][tests] Harden ClientTest by making handler shareable
flinkbot edited a comment on pull request #16915: URL: https://github.com/apache/flink/pull/16915#issuecomment-902862695 ## CI report: * 8d94bd1869bebcf45df6488b45effb5b419ae52d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22573) * 54cff8c8965e8277bc66ce800a4b2f134b03fccd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22974) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16773: [FLINK-21090][tests] Add IT case for stop-with-savepoint and final checkpoint
flinkbot edited a comment on pull request #16773: URL: https://github.com/apache/flink/pull/16773#issuecomment-896196766 ## CI report: * f11f93a01d1d0e769d04eb0b66a34914526096cc UNKNOWN * 61cc9c310c02dea756bc02436d7fab4654caaa1a UNKNOWN * 043df6eaf3e6e69b3871edea72291a9a2266bf3a UNKNOWN * 71daf051661e450cb649f4dba52d3a70e569e085 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16575: [FLINK-23139][state] Add TaskStateRegistry
flinkbot edited a comment on pull request #16575: URL: https://github.com/apache/flink/pull/16575#issuecomment-885295771 ## CI report: * e17fff1f519cb0b7e8f20dd97ff8d78a0f1391b5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21734) * eed9672d2ac736e5261d4e3511ccacab1b77d683 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22973) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23954) Queryable state (rocksdb) with TM restart end-to-end test fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-23954:
--
Fix Version/s: 1.13.3
1.12.6
> Queryable state (rocksdb) with TM restart end-to-end test fails on azure
>
>
> Key: FLINK-23954
> URL: https://issues.apache.org/jira/browse/FLINK-23954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.12.6, 1.13.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22714=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=11090
> {code}
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 Running 'Queryable state (rocksdb) with TM restart end-to-end
> test'
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-17362481511
> Aug 24 09:54:17 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.14-SNAPSHOT-bin/flink-1.14-SNAPSHOT
> Aug 24 09:54:17 Adding flink-queryable-state-runtime to lib/
> Aug 24 09:54:17 Starting cluster.
> Aug 24 09:54:18 Starting standalonesession daemon on host fv-az123-794.
> Aug 24 09:54:19 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:19 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:20 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:21 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:22 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:23 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:24 Dispatcher REST endpoint is up.
> Aug 24 09:54:31 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:31 Starting to wait for completion of 10 checkpoints
> Aug 24 09:54:31 2/10 completed checkpoints
> Aug 24 09:54:33 6/10 completed checkpoints
> Aug 24 09:54:35 6/10 completed checkpoints
> Aug 24 09:54:37 SERVER: 127.0.0.1
> Aug 24 09:54:37 PORT: 9069
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:38 MapState has 17 entries
> Aug 24 09:54:38 TaskManager 412715 killed.
> Aug 24 09:54:38 Number of running task managers 1 is not yet 0.
> Aug 24 09:54:42 Number of running task managers has reached 0.
> Aug 24 09:54:42 Latest snapshot count was 22
> Aug 24 09:54:43 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:43 Number of running task managers 0 is not yet 1.
> Aug 24 09:54:47 Number of running task managers has reached 1.
> Aug 24 09:54:49 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:49 Starting to wait for completion of 16 checkpoints
> Aug 24 09:54:49 11/16 completed checkpoints
> Aug 24 09:54:51 11/16 completed checkpoints
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:54 after: 21
> Aug 24 09:54:54 An error occurred
> Aug 24 09:54:54 [FAIL] Test script contains errors.
> Aug 24 09:54:54 Checking of logs skipped.
> Aug 24 09:54:54
> Aug 24 09:54:54 [FAIL] 'Queryable state (rocksdb) with TM restart end-to-end
> test' failed after 0 minutes and 37 seconds! Test exited with exit code 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23954) Queryable state (rocksdb) with TM restart end-to-end test fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405957#comment-17405957
]
Till Rohrmann commented on FLINK-23954:
---
Stupid me: I think I know what the problem is. With FLINK-23097 I changed that
we only output the number of elements once the checkpoint is completed.
However, I did not output the number of entries at the point in time when the
checkpoint was taken but when the {{notifyCheckpointComplete}} message arrived.
This means that the number of entries could have grown and, thus, we are
reporting wrong values.
> Queryable state (rocksdb) with TM restart end-to-end test fails on azure
>
>
> Key: FLINK-23954
> URL: https://issues.apache.org/jira/browse/FLINK-23954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22714=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=11090
> {code}
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 Running 'Queryable state (rocksdb) with TM restart end-to-end
> test'
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-17362481511
> Aug 24 09:54:17 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.14-SNAPSHOT-bin/flink-1.14-SNAPSHOT
> Aug 24 09:54:17 Adding flink-queryable-state-runtime to lib/
> Aug 24 09:54:17 Starting cluster.
> Aug 24 09:54:18 Starting standalonesession daemon on host fv-az123-794.
> Aug 24 09:54:19 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:19 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:20 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:21 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:22 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:23 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:24 Dispatcher REST endpoint is up.
> Aug 24 09:54:31 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:31 Starting to wait for completion of 10 checkpoints
> Aug 24 09:54:31 2/10 completed checkpoints
> Aug 24 09:54:33 6/10 completed checkpoints
> Aug 24 09:54:35 6/10 completed checkpoints
> Aug 24 09:54:37 SERVER: 127.0.0.1
> Aug 24 09:54:37 PORT: 9069
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:38 MapState has 17 entries
> Aug 24 09:54:38 TaskManager 412715 killed.
> Aug 24 09:54:38 Number of running task managers 1 is not yet 0.
> Aug 24 09:54:42 Number of running task managers has reached 0.
> Aug 24 09:54:42 Latest snapshot count was 22
> Aug 24 09:54:43 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:43 Number of running task managers 0 is not yet 1.
> Aug 24 09:54:47 Number of running task managers has reached 1.
> Aug 24 09:54:49 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:49 Starting to wait for completion of 16 checkpoints
> Aug 24 09:54:49 11/16 completed checkpoints
> Aug 24 09:54:51 11/16 completed checkpoints
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:54 after: 21
> Aug 24 09:54:54 An error occurred
> Aug 24 09:54:54 [FAIL] Test script contains errors.
> Aug 24 09:54:54 Checking of logs skipped.
> Aug 24 09:54:54
> Aug 24 09:54:54 [FAIL] 'Queryable state (rocksdb) with TM restart end-to-end
> test' failed after 0 minutes and 37 seconds! Test exited with exit code 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16987: [FLINK-23968][tests] Remove unused code from TestBaseUtils
flinkbot edited a comment on pull request #16987: URL: https://github.com/apache/flink/pull/16987#issuecomment-905577505 ## CI report: * 480855e82a32496f64bcbf05dd4ea7ce205186db Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22840) * b873c48ed77b2bcdcb950861fa35e6299b6d9e81 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16915: [FLINK-9925][tests] Harden ClientTest by making handler shareable
flinkbot edited a comment on pull request #16915: URL: https://github.com/apache/flink/pull/16915#issuecomment-902862695 ## CI report: * 8d94bd1869bebcf45df6488b45effb5b419ae52d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22573) * 54cff8c8965e8277bc66ce800a4b2f134b03fccd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23954) Queryable state (rocksdb) with TM restart end-to-end test fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405955#comment-17405955
]
Till Rohrmann commented on FLINK-23954:
---
On the other hand, when accessing the MapState in the {{open}} method, then we
should set an empty {{prefixBytes}} which should give us all RocksDB entries.
Hence, we should actually get the same count on open as what we had when
creating the latest checkpoint.
One more addition: We seem to use unaligned checkpoints. But I don't think that
this has something to do with the test failure.
> Queryable state (rocksdb) with TM restart end-to-end test fails on azure
>
>
> Key: FLINK-23954
> URL: https://issues.apache.org/jira/browse/FLINK-23954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22714=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=11090
> {code}
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 Running 'Queryable state (rocksdb) with TM restart end-to-end
> test'
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-17362481511
> Aug 24 09:54:17 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.14-SNAPSHOT-bin/flink-1.14-SNAPSHOT
> Aug 24 09:54:17 Adding flink-queryable-state-runtime to lib/
> Aug 24 09:54:17 Starting cluster.
> Aug 24 09:54:18 Starting standalonesession daemon on host fv-az123-794.
> Aug 24 09:54:19 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:19 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:20 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:21 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:22 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:23 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:24 Dispatcher REST endpoint is up.
> Aug 24 09:54:31 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:31 Starting to wait for completion of 10 checkpoints
> Aug 24 09:54:31 2/10 completed checkpoints
> Aug 24 09:54:33 6/10 completed checkpoints
> Aug 24 09:54:35 6/10 completed checkpoints
> Aug 24 09:54:37 SERVER: 127.0.0.1
> Aug 24 09:54:37 PORT: 9069
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:38 MapState has 17 entries
> Aug 24 09:54:38 TaskManager 412715 killed.
> Aug 24 09:54:38 Number of running task managers 1 is not yet 0.
> Aug 24 09:54:42 Number of running task managers has reached 0.
> Aug 24 09:54:42 Latest snapshot count was 22
> Aug 24 09:54:43 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:43 Number of running task managers 0 is not yet 1.
> Aug 24 09:54:47 Number of running task managers has reached 1.
> Aug 24 09:54:49 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:49 Starting to wait for completion of 16 checkpoints
> Aug 24 09:54:49 11/16 completed checkpoints
> Aug 24 09:54:51 11/16 completed checkpoints
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:54 after: 21
> Aug 24 09:54:54 An error occurred
> Aug 24 09:54:54 [FAIL] Test script contains errors.
> Aug 24 09:54:54 Checking of logs skipped.
> Aug 24 09:54:54
> Aug 24 09:54:54 [FAIL] 'Queryable state (rocksdb) with TM restart end-to-end
> test' failed after 0 minutes and 37 seconds! Test exited with exit code 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
flinkbot edited a comment on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-904042149 ## CI report: * 04c9fc636bd6f07bf47654b5fc6d80c939169bc4 UNKNOWN * 454b7277df57678f94dc74901a54b3b22e241276 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22689) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22761) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22975) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17010: [FLINK-24003][python] Fix lookback mode doesn't work when mixing use of Python Table API and Python DataStream API
flinkbot edited a comment on pull request #17010: URL: https://github.com/apache/flink/pull/17010#issuecomment-906989888 ## CI report: * 5cb7b56308564c40a068088912b1a5c18e42611f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22970) * 2eccdcb7ea2438682caad05ffa50be41af48af5b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22972) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17009: [FLINK-23466][network] Fix the bug that buffer listeners may not be notified when recycling buffers
flinkbot edited a comment on pull request #17009: URL: https://github.com/apache/flink/pull/17009#issuecomment-906989802 ## CI report: * 1cda0c085f25dc1b8e9e36d23ac8bf30d85f58e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22969) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23954) Queryable state (rocksdb) with TM restart end-to-end test fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405946#comment-17405946
]
Till Rohrmann edited comment on FLINK-23954 at 8/27/21, 6:01 PM:
-
The fewer entries could be caused that we count the events in the {{open}}
method where we haven't set a proper key yet?
was (Author: till.rohrmann):
The fewer entries could be caused that we count the events in the {{open}}
method where we haven't set a proper key yet.
> Queryable state (rocksdb) with TM restart end-to-end test fails on azure
>
>
> Key: FLINK-23954
> URL: https://issues.apache.org/jira/browse/FLINK-23954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22714=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=11090
> {code}
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 Running 'Queryable state (rocksdb) with TM restart end-to-end
> test'
> Aug 24 09:54:17
> ==
> Aug 24 09:54:17 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-17362481511
> Aug 24 09:54:17 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.14-SNAPSHOT-bin/flink-1.14-SNAPSHOT
> Aug 24 09:54:17 Adding flink-queryable-state-runtime to lib/
> Aug 24 09:54:17 Starting cluster.
> Aug 24 09:54:18 Starting standalonesession daemon on host fv-az123-794.
> Aug 24 09:54:19 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:19 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:20 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:21 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:22 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:23 Waiting for Dispatcher REST endpoint to come up...
> Aug 24 09:54:24 Dispatcher REST endpoint is up.
> Aug 24 09:54:31 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:31 Starting to wait for completion of 10 checkpoints
> Aug 24 09:54:31 2/10 completed checkpoints
> Aug 24 09:54:33 6/10 completed checkpoints
> Aug 24 09:54:35 6/10 completed checkpoints
> Aug 24 09:54:37 SERVER: 127.0.0.1
> Aug 24 09:54:37 PORT: 9069
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:38 MapState has 17 entries
> Aug 24 09:54:38 TaskManager 412715 killed.
> Aug 24 09:54:38 Number of running task managers 1 is not yet 0.
> Aug 24 09:54:42 Number of running task managers has reached 0.
> Aug 24 09:54:42 Latest snapshot count was 22
> Aug 24 09:54:43 Starting taskexecutor daemon on host fv-az123-794.
> Aug 24 09:54:43 Number of running task managers 0 is not yet 1.
> Aug 24 09:54:47 Number of running task managers has reached 1.
> Aug 24 09:54:49 Job (f230364470a082b78dc34147f8da38f1) is running.
> Aug 24 09:54:49 Starting to wait for completion of 16 checkpoints
> Aug 24 09:54:49 11/16 completed checkpoints
> Aug 24 09:54:51 11/16 completed checkpoints
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Aug 24 09:54:54 after: 21
> Aug 24 09:54:54 An error occurred
> Aug 24 09:54:54 [FAIL] Test script contains errors.
> Aug 24 09:54:54 Checking of logs skipped.
> Aug 24 09:54:54
> Aug 24 09:54:54 [FAIL] 'Queryable state (rocksdb) with TM restart end-to-end
> test' failed after 0 minutes and 37 seconds! Test exited with exit code 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-training] alpinegizmo commented on a change in pull request #31: [FLINK-23653] exercise and test rework
alpinegizmo commented on a change in pull request #31:
URL: https://github.com/apache/flink-training/pull/31#discussion_r697621198
##
File path:
rides-and-fares/src/main/scala/org/apache/flink/training/exercises/ridesandfares/scala/RidesAndFaresExercise.scala
##
@@ -18,55 +18,73 @@
package org.apache.flink.training.exercises.ridesandfares.scala
+import org.apache.flink.api.common.JobExecutionResult
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.configuration.Configuration
import org.apache.flink.streaming.api.functions.co.RichCoFlatMapFunction
+import org.apache.flink.streaming.api.functions.sink.{PrintSinkFunction,
SinkFunction}
+import org.apache.flink.streaming.api.functions.source.SourceFunction
import org.apache.flink.streaming.api.scala.{StreamExecutionEnvironment, _}
-import org.apache.flink.training.exercises.common.datatypes.{TaxiFare,
TaxiRide}
+import org.apache.flink.training.exercises.common.datatypes.{RideAndFare,
TaxiFare, TaxiRide}
import org.apache.flink.training.exercises.common.sources.{TaxiFareGenerator,
TaxiRideGenerator}
-import org.apache.flink.training.exercises.common.utils.ExerciseBase._
-import org.apache.flink.training.exercises.common.utils.{ExerciseBase,
MissingSolutionException}
+import
org.apache.flink.training.exercises.common.utils.MissingSolutionException
import org.apache.flink.util.Collector
/**
- * The "Stateful Enrichment" exercise of the Flink training in the docs.
+ * The Stateful Enrichment exercise from the Flink training.
*
* The goal for this exercise is to enrich TaxiRides with fare information.
- *
*/
object RidesAndFaresExercise {
- def main(args: Array[String]) {
+ class RidesAndFaresJob(rideSource: SourceFunction[TaxiRide],
+ fareSource: SourceFunction[TaxiFare],
+ sink: SinkFunction[RideAndFare]) {
-// set up streaming execution environment
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-env.setParallelism(ExerciseBase.parallelism)
+def execute(): JobExecutionResult = {
+ val env = StreamExecutionEnvironment.getExecutionEnvironment
Review comment:
True. I guess I don't see much value in adding it to the scala solution.
The reason it's there in the java solution is so that people can run that
solution and then examine the checkpoints left behind by running the app
provided in the state processor exercise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] nicolasraga1 commented on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
nicolasraga1 commented on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-907369512 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nicolasraga1 removed a comment on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
nicolasraga1 removed a comment on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-907368898 Publish an amended commit with the requested changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nicolasraga1 closed pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
nicolasraga1 closed pull request #16946: URL: https://github.com/apache/flink/pull/16946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nicolasraga1 commented on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
nicolasraga1 commented on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-907368898 Publish an amended commit with the requested changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nicolasraga1 removed a comment on pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
nicolasraga1 removed a comment on pull request #16946: URL: https://github.com/apache/flink/pull/16946#issuecomment-907368750 Publish an amended commit with the requested changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
