[GitHub] [flink] flinkbot edited a comment on pull request #13588: [hotfix][docs] fix typo in event_timestamps_watermarks.md
flinkbot edited a comment on pull request #13588: URL: https://github.com/apache/flink/pull/13588#issuecomment-706842690 ## CI report: * 50a542163c4bf31883ae2c1e7211f0e38692ead5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7403) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #13479: [FLINK-19414][parquet] Introduce ParquetColumnarRowInputFormat
JingsongLi commented on a change in pull request #13479:
URL: https://github.com/apache/flink/pull/13479#discussion_r503048887
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetVectorizedInputFormat.java
##
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.parquet;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.base.source.reader.SourceReaderOptions;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.connector.file.src.util.CheckpointedPosition;
+import org.apache.flink.connector.file.src.util.Pool;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.parquet.utils.SerializableConfiguration;
+import org.apache.flink.formats.parquet.vector.ColumnBatchFactory;
+import org.apache.flink.formats.parquet.vector.ParquetDecimalVector;
+import org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
+import org.apache.flink.table.data.vector.writable.WritableColumnVector;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import static
org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createColumnReader;
+import static
org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector;
+import static org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups;
+import static
org.apache.parquet.format.converter.ParquetMetadataConverter.range;
+import static org.apache.parquet.hadoop.ParquetFileReader.readFooter;
+import static org.apache.parquet.hadoop.ParquetInputFormat.getFilter;
+
+/**
+ * Parquet {@link BulkFormat} that reads data from the file to {@link
VectorizedColumnBatch} in
+ * vectorized mode.
+ */
+public abstract class ParquetVectorizedInputFormat implements BulkFormat
{
+
+ private static final long serialVersionUID = 1L;
+
+ private final SerializableConfiguration hadoopConfig;
+ private final String[] projectedFields;
+ private final LogicalType[] projectedTypes;
+ private final ColumnBatchFactory batchFactory;
+ private final int batchSize;
+ private final boolean isUtcTimestamp;
+ private final boolean isCaseSensitive;
+
+ public ParquetVectorizedInputFormat(
+ SerializableConfiguration hadoopConfig,
+ String[] projectedFields,
+ LogicalType[] projectedTypes,
+ ColumnBatchFactory batchFactory,
+ int batchSize,
+ boolean isUtcTimestamp,
+ boolean isCaseSensitive) {
+ Preconditions.checkArgument(
+ projectedFields.length == projectedTypes.length,
+ "The length(%s) of projectedFields should equal
to the length(%s) projectedTypes",
+ projectedFields.length,
+ projectedTypes.length);
+
+ this.hadoopConfig = hadoopConfig;
+ this.projectedFields = projectedFields;
[GitHub] [flink] flinkbot edited a comment on pull request #13296: [FLINK-18774][format][debezium] Support debezium-avro format
flinkbot edited a comment on pull request #13296: URL: https://github.com/apache/flink/pull/13296#issuecomment-684611193 ## CI report: * c266341c63cc4cdc28a6585a82294710fd8ebbc5 UNKNOWN * bfa5c99bf226d5ad09d4aa83a61d588c262d5044 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7406) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13574: [FLINK-18323] Add a Kafka source implementation based on FLIP-27.
flinkbot edited a comment on pull request #13574: URL: https://github.com/apache/flink/pull/13574#issuecomment-706309837 ## CI report: * b149514ece5711884868a8b80ff3b43afa00f011 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7348) * f8925ca0a84f19fc3c6e37ee1f428c329bd50a34 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7405) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
flinkbot edited a comment on pull request #13519: URL: https://github.com/apache/flink/pull/13519#issuecomment-701236311 ## CI report: * fa89bab5fc48538a35ccc69508da0d89527f98ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7395) * 2b1c9dcf9d5e3f27ad6635b58ccca2a6737b8b03 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7404) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #13479: [FLINK-19414][parquet] Introduce ParquetColumnarRowInputFormat
JingsongLi commented on a change in pull request #13479:
URL: https://github.com/apache/flink/pull/13479#discussion_r503034628
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetVectorizedInputFormat.java
##
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.parquet;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.base.source.reader.SourceReaderOptions;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.connector.file.src.util.CheckpointedPosition;
+import org.apache.flink.connector.file.src.util.Pool;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.parquet.utils.SerializableConfiguration;
+import org.apache.flink.formats.parquet.vector.ColumnBatchFactory;
+import org.apache.flink.formats.parquet.vector.ParquetDecimalVector;
+import org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
+import org.apache.flink.table.data.vector.writable.WritableColumnVector;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import static
org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createColumnReader;
+import static
org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector;
+import static org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups;
+import static
org.apache.parquet.format.converter.ParquetMetadataConverter.range;
+import static org.apache.parquet.hadoop.ParquetFileReader.readFooter;
+import static org.apache.parquet.hadoop.ParquetInputFormat.getFilter;
+
+/**
+ * Parquet {@link BulkFormat} that reads data from the file to {@link
VectorizedColumnBatch} in
+ * vectorized mode.
+ */
+public abstract class ParquetVectorizedInputFormat implements BulkFormat
{
+
+ private static final long serialVersionUID = 1L;
+
+ private final SerializableConfiguration hadoopConfig;
+ private final String[] projectedFields;
+ private final LogicalType[] projectedTypes;
+ private final ColumnBatchFactory batchFactory;
+ private final int batchSize;
+ private final boolean isUtcTimestamp;
+ private final boolean isCaseSensitive;
+
+ public ParquetVectorizedInputFormat(
+ SerializableConfiguration hadoopConfig,
+ String[] projectedFields,
+ LogicalType[] projectedTypes,
+ ColumnBatchFactory batchFactory,
+ int batchSize,
+ boolean isUtcTimestamp,
+ boolean isCaseSensitive) {
+ Preconditions.checkArgument(
+ projectedFields.length == projectedTypes.length,
+ "The length(%s) of projectedFields should equal
to the length(%s) projectedTypes",
+ projectedFields.length,
+ projectedTypes.length);
+
+ this.hadoopConfig = hadoopConfig;
+ this.projectedFields = projectedFields;
[GitHub] [flink] flinkbot edited a comment on pull request #13296: [FLINK-18774][format][debezium] Support debezium-avro format
flinkbot edited a comment on pull request #13296: URL: https://github.com/apache/flink/pull/13296#issuecomment-684611193 ## CI report: * c266341c63cc4cdc28a6585a82294710fd8ebbc5 UNKNOWN * c6301b9c4f43e851a0d47972bd52880b69b720c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7186) * bfa5c99bf226d5ad09d4aa83a61d588c262d5044 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on pull request #13574: [FLINK-18323] Add a Kafka source implementation based on FLIP-27.
stevenzwu commented on pull request #13574: URL: https://github.com/apache/flink/pull/13574#issuecomment-706852026 thx. LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a change in pull request #13574: [FLINK-18323] Add a Kafka source implementation based on FLIP-27.
stevenzwu commented on a change in pull request #13574:
URL: https://github.com/apache/flink/pull/13574#discussion_r503030859
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java
##
@@ -0,0 +1,398 @@
+/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.reader;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.connector.base.source.reader.RecordsWithSplitIds;
+import org.apache.flink.connector.base.source.reader.splitreader.SplitReader;
+import
org.apache.flink.connector.base.source.reader.splitreader.SplitsAddition;
+import org.apache.flink.connector.base.source.reader.splitreader.SplitsChange;
+import org.apache.flink.connector.kafka.source.KafkaSourceOptions;
+import
org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializer;
+import org.apache.flink.connector.kafka.source.split.KafkaPartitionSplit;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+import java.util.Set;
+import java.util.StringJoiner;
+
+/**
+ * A {@link SplitReader} implementation that reads records from Kafka
partitions.
+ *
+ * The returned type are in the format of {@code tuple3(record, offset and
timestamp}.
+ *
+ * @param the type of the record to be emitted from the Source.
+ */
+public class KafkaPartitionSplitReader implements SplitReader, KafkaPartitionSplit> {
+ private static final Logger LOG =
LoggerFactory.getLogger(KafkaPartitionSplitReader.class);
+ private static final long POLL_TIMEOUT = 1L;
+
+ private final KafkaConsumer consumer;
+ private final KafkaRecordDeserializer deserializationSchema;
+ private final Map stoppingOffsets;
+ private final SimpleCollector collector;
+ private final String groupId;
+
+ public KafkaPartitionSplitReader(
+ Properties props,
+ KafkaRecordDeserializer deserializationSchema) {
+ Properties consumerProps = new Properties();
+ consumerProps.putAll(props);
+ consumerProps.setProperty(ConsumerConfig.CLIENT_ID_CONFIG,
createConsumerClientId(props));
+ this.consumer = new KafkaConsumer<>(consumerProps);
+ this.stoppingOffsets = new HashMap<>();
+ this.deserializationSchema = deserializationSchema;
+ this.collector = new SimpleCollector<>();
+ this.groupId =
consumerProps.getProperty(ConsumerConfig.GROUP_ID_CONFIG);
+ }
+
+ @Override
+ public RecordsWithSplitIds> fetch() throws
IOException {
+ KafkaPartitionSplitRecords>
recordsBySplits = new KafkaPartitionSplitRecords<>();
+ ConsumerRecords consumerRecords;
+ try {
+ consumerRecords =
consumer.poll(Duration.ofMillis(POLL_TIMEOUT));
+ } catch (WakeupException we) {
+ return recordsBySplits;
+ }
+
+ List finishedPartitions = new ArrayList<>();
+ for (TopicPartition tp : consumerRecords.partitions()) {
+ long stoppingOffset = getStoppingOffset(tp);
+ String splitId = tp.toString();
+ Collection> recordsForSplit =
recordsBySplits.recordsForSplit(splitId);
+
[GitHub] [flink] flinkbot edited a comment on pull request #13574: [FLINK-18323] Add a Kafka source implementation based on FLIP-27.
flinkbot edited a comment on pull request #13574: URL: https://github.com/apache/flink/pull/13574#issuecomment-706309837 ## CI report: * b149514ece5711884868a8b80ff3b43afa00f011 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7348) * f8925ca0a84f19fc3c6e37ee1f428c329bd50a34 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13588: [hotfix][docs] fix typo in event_timestamps_watermarks.md
flinkbot edited a comment on pull request #13588: URL: https://github.com/apache/flink/pull/13588#issuecomment-706842690 ## CI report: * 50a542163c4bf31883ae2c1e7211f0e38692ead5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7403) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
flinkbot edited a comment on pull request #13519: URL: https://github.com/apache/flink/pull/13519#issuecomment-701236311 ## CI report: * fa89bab5fc48538a35ccc69508da0d89527f98ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7395) * 2b1c9dcf9d5e3f27ad6635b58ccca2a6737b8b03 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on a change in pull request #13574: [FLINK-18323] Add a Kafka source implementation based on FLIP-27.
becketqin commented on a change in pull request #13574:
URL: https://github.com/apache/flink/pull/13574#discussion_r502855189
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java
##
@@ -0,0 +1,370 @@
+/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.enumerator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.api.connector.source.SplitsAssignment;
+import org.apache.flink.connector.base.source.event.NoMoreSplitsEvent;
+import org.apache.flink.connector.kafka.source.KafkaSourceOptions;
+import
org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
+import
org.apache.flink.connector.kafka.source.enumerator.subscriber.KafkaSubscriber;
+import org.apache.flink.connector.kafka.source.split.KafkaPartitionSplit;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.function.ThrowingRunnable;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.ListConsumerGroupOffsetsOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.consumer.OffsetAndTimestamp;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.flink.util.ComponentClosingUtils.closeWithTimeout;
+
+/**
+ * The enumerator class for Kafka source.
+ */
+@Internal
+public class KafkaSourceEnumerator implements
SplitEnumerator {
+ private static final Logger LOG =
LoggerFactory.getLogger(KafkaSourceEnumerator.class);
+ private final KafkaSubscriber subscriber;
+ private final OffsetsInitializer startingOffsetInitializer;
+ private final OffsetsInitializer stoppingOffsetInitializer;
+ private final Properties properties;
+ private final long partitionDiscoveryIntervalMs;
+ private final SplitEnumeratorContext context;
+
+ // The internal states of the enumerator.
+ /** This set is only accessed by the partition discovery callable in
the callAsync() method, i.e worker thread. */
+ private final Set discoveredPartitions;
+ /** The current assignment by reader id. Only accessed by the
coordinator thread. */
+ private final Map>
readerIdToSplitAssignments;
+ /** The discovered and initialized partition splits that are waiting
for owner reader to be ready. */
+ private final Map>
pendingPartitionSplitAssignment;
+ /** The consumer group id used for this KafkaSource. */
+ private final String consumerGroupId;
+
+ // Lazily instantiated or mutable fields.
+ private KafkaConsumer consumer;
+ private AdminClient adminClient;
+ private boolean noMoreNewPartitionSplits = false;
+
+ public KafkaSourceEnumerator(
+ KafkaSubscriber subscriber,
+ OffsetsInitializer startingOffsetInitializer,
+ OffsetsInitializer stoppingOffsetInitializer,
+ Properties properties,
+ SplitEnumeratorContext context) {
+ this(subscriber, startingOffsetInitializer,
stoppingOffsetInitializer, properties, context, new HashMap<>());
+ }
+
+ public KafkaSourceEnumerator(
+ KafkaSubscriber subscriber,
+
[GitHub] [flink] zhoulii opened a new pull request #13588: [hotfix][docs] fix typo in event_timestamps_watermarks.md
zhoulii opened a new pull request #13588: URL: https://github.com/apache/flink/pull/13588 ## What is the purpose of the change fix typo in `event_timestamps_watermarks.md` and `event_timestamps_watermarks.zh.md` ## Brief change log - change `AssignerWithPeriodicWatermarks` to `AssignerWithPunctuatedWatermarks` in `event_timestamps_watermarks.md` - change `AssignerWithPeriodicWatermarks` to `AssignerWithPunctuatedWatermarks` in `event_timestamps_watermarks.zh.md` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13588: [hotfix][docs] fix typo in event_timestamps_watermarks.md
flinkbot commented on pull request #13588: URL: https://github.com/apache/flink/pull/13588#issuecomment-706842690 ## CI report: * 50a542163c4bf31883ae2c1e7211f0e38692ead5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
zhuxiaoshang commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r503024880
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ private transient Counter numLateRecordsDropped;
+
+ @VisibleForTesting
+ public Counter getCounter() {
Review comment:
got it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13588: [hotfix][docs] fix typo in event_timestamps_watermarks.md
flinkbot commented on pull request #13588: URL: https://github.com/apache/flink/pull/13588#issuecomment-706839600 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 50a542163c4bf31883ae2c1e7211f0e38692ead5 (Mon Oct 12 03:41:35 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19101) The SelectivityEstimator throw an NullPointerException when convertValueInterval with string type
[
https://issues.apache.org/jira/browse/FLINK-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fa zheng closed FLINK-19101.
Resolution: Cannot Reproduce
> The SelectivityEstimator throw an NullPointerException when
> convertValueInterval with string type
> -
>
> Key: FLINK-19101
> URL: https://issues.apache.org/jira/browse/FLINK-19101
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.2, 1.11.1
>Reporter: fa zheng
>Priority: Critical
> Fix For: 1.12.0
>
>
> It will throw java.lang.NullPointerException because SelectivityEstimator
> convert character value interval to string without considering null
> situation.
> {code:scala}
> def convertValueInterval(
> interval: ValueInterval,
> typeFamily: RelDataTypeFamily): ValueInterval = {
> require(interval != null && typeFamily != null)
> interval match {
> case ValueInterval.empty | ValueInterval.infinite => interval
> case _ =>
> val (lower, includeLower) = interval match {
> case li: WithLower => (li.lower, li.includeLower)
> case _ => (null, false)
> }
> val (upper, includeUpper) = interval match {
> case ui: WithUpper => (ui.upper, ui.includeUpper)
> case _ => (null, false)
> }
> typeFamily match {
> case SqlTypeFamily.NUMERIC | SqlTypeFamily.BOOLEAN |
> SqlTypeFamily.DATE |
>SqlTypeFamily.TIME | SqlTypeFamily.TIMESTAMP =>
> ValueInterval(
> comparableToDouble(lower),
> comparableToDouble(upper),
> includeLower,
> includeUpper)
> case SqlTypeFamily.CHARACTER =>
> ValueInterval(
> lower.toString, //It can't call toString directly
> upper.toString //It can't call toString directly
> includeLower,
> includeUpper)
> case _ => throw new UnsupportedOperationException(s"Unsupported
> typeFamily: $typeFamily")
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19567) JoinITCase.testInnerJoinOutputWithPk is unstable for POJO type inference
[
https://issues.apache.org/jira/browse/FLINK-19567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212104#comment-17212104
]
Danny Chen commented on FLINK-19567:
I encountered this problem when upgrade the planner Calcite version to 1.26.0,
but the error should not be caused by the upgrade.
> JoinITCase.testInnerJoinOutputWithPk is unstable for POJO type inference
>
>
> Key: FLINK-19567
> URL: https://issues.apache.org/jira/browse/FLINK-19567
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.11.2
>Reporter: Danny Chen
>Priority: Major
>
> The
> {{org.apache.flink.table.runtime.stream.table.JoinITCase.testInnerJoinOutputWithPk}}
> is unstable, it fails randomly in the mvn test but success in the IDEA local
> runner.
> Here is the stacktrace:
> {code:xml}
> [ERROR]
> testInnerJoinOutputWithPk(org.apache.flink.table.runtime.stream.table.JoinITCase)
> Time elapsed: 0.044 s <<< ERROR!
> org.apache.flink.table.codegen.CodeGenException: Incompatible types of
> expression and result type.
> Expression[GeneratedExpression(result$19166,isNull$19167,,GenericType,false)]
> type is [GenericType], result type is
> [GenericType]
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$generateResultExpression$2.apply(CodeGenerator.scala:379)
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$generateResultExpression$2.apply(CodeGenerator.scala:377)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.flink.table.codegen.CodeGenerator.generateResultExpression(CodeGenerator.scala:377)
> at
> org.apache.flink.table.codegen.CodeGenerator.generateConverterResultExpression(CodeGenerator.scala:295)
> at
> org.apache.flink.table.plan.nodes.datastream.StreamScan$class.generateConversionProcessFunction(StreamScan.scala:115)
> at
> org.apache.flink.table.plan.nodes.datastream.StreamScan$class.convertToInternalRow(StreamScan.scala:74)
> at
> org.apache.flink.table.plan.nodes.datastream.DataStreamScan.convertToInternalRow(DataStreamScan.scala:46)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] libenchao commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
libenchao commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r503017321
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ private transient Counter numLateRecordsDropped;
+
+ @VisibleForTesting
+ public Counter getCounter() {
Review comment:
I mean `protected getCounter()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-19567) JoinITCase.testInnerJoinOutputWithPk is unstable for POJO type inference
Danny Chen created FLINK-19567:
--
Summary: JoinITCase.testInnerJoinOutputWithPk is unstable for POJO
type inference
Key: FLINK-19567
URL: https://issues.apache.org/jira/browse/FLINK-19567
Project: Flink
Issue Type: Bug
Components: Table SQL / Legacy Planner
Affects Versions: 1.11.2
Reporter: Danny Chen
The
{{org.apache.flink.table.runtime.stream.table.JoinITCase.testInnerJoinOutputWithPk}}
is unstable, it fails randomly in the mvn test but success in the IDEA local
runner.
Here is the stacktrace:
{code:xml}
[ERROR]
testInnerJoinOutputWithPk(org.apache.flink.table.runtime.stream.table.JoinITCase)
Time elapsed: 0.044 s <<< ERROR!
org.apache.flink.table.codegen.CodeGenException: Incompatible types of
expression and result type.
Expression[GeneratedExpression(result$19166,isNull$19167,,GenericType,false)]
type is [GenericType], result type is
[GenericType]
at
org.apache.flink.table.codegen.CodeGenerator$$anonfun$generateResultExpression$2.apply(CodeGenerator.scala:379)
at
org.apache.flink.table.codegen.CodeGenerator$$anonfun$generateResultExpression$2.apply(CodeGenerator.scala:377)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.flink.table.codegen.CodeGenerator.generateResultExpression(CodeGenerator.scala:377)
at
org.apache.flink.table.codegen.CodeGenerator.generateConverterResultExpression(CodeGenerator.scala:295)
at
org.apache.flink.table.plan.nodes.datastream.StreamScan$class.generateConversionProcessFunction(StreamScan.scala:115)
at
org.apache.flink.table.plan.nodes.datastream.StreamScan$class.convertToInternalRow(StreamScan.scala:74)
at
org.apache.flink.table.plan.nodes.datastream.DataStreamScan.convertToInternalRow(DataStreamScan.scala:46)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuxiaoshang commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
zhuxiaoshang commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r503015445
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ private transient Counter numLateRecordsDropped;
+
+ @VisibleForTesting
+ public Counter getCounter() {
Review comment:
we can access it by `getCounter()`,do you mean that change Counter to
`protected` and remove `getCounter() `?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
zhuxiaoshang commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r503015445
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ private transient Counter numLateRecordsDropped;
+
+ @VisibleForTesting
+ public Counter getCounter() {
Review comment:
we can access it by `getCounter()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] libenchao commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
libenchao commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r503014325
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ private transient Counter numLateRecordsDropped;
+
+ @VisibleForTesting
+ public Counter getCounter() {
Review comment:
we could not access it if it's `private`, `protected` methods can be
accessed in tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-19565) java.util.Collections$UnmodifiableList to field org.apache.flink.runtime.jobgraph.JobVertex.operatorIDs of type java.util.ArrayList in instance of org.apache.flink.runti
[
https://issues.apache.org/jira/browse/FLINK-19565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yardstrong updated FLINK-19565:
---
Environment:
I add {color:#4c9aff}flink-csv-1.10.2.jar{color} &
[flink-sql-connector-kafka_2.12-1.10.2.jar|https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/1.10.2/flink-sql-connector-kafka_2.12-1.10.2.jar]{color:#4c9aff}
{color}
{color:#172b4d}to the lib dir. I run the sql below with sql-client:{color}
_CREATE TABLE kafkaSource (_
_a VARCHAR,_
{color:#505f79} _b VARCHAR_{color}
{color:#505f79}_) WITH (_{color}
{color:#505f79} _'connector' = 'kafka',_{color}
{color:#505f79} _'topic' = 'source001',_{color}
{color:#505f79} _'properties.bootstrap.servers' = 'localhost:9092',_{color}
{color:#505f79} _'properties.group.id' = 'consumer001',_{color}
{color:#505f79} _'format' = 'csv',_{color}
{color:#505f79} _'scan.startup.mode' = 'latest-offset'_{color}
{color:#505f79}_);_{color}
{color:#505f79}_CREATE TABLE kafkaSink (_{color}
{color:#505f79} _a VARCHAR,_{color}
{color:#505f79} _b VARCHAR_{color}
{color:#505f79}_) WITH (_{color}
{color:#505f79} _'connector' = 'kafka',_{color}
{color:#505f79} _'topic' = 'sink001',_{color}
{color:#505f79} _'properties.bootstrap.servers' = 'localhost:9092',_{color}
{color:#505f79} _'format' = 'csv'_{color}
{color:#505f79}_);_{color}
{color:#505f79}_INSERT INTO kafkaSink_{color}
{color:#505f79}_SELECT *_{color}
{color:#505f79}_FROM kafkaSource;_{color}
{color:#172b4d}Then the exception {color}appeared.
was:
I add {color:#4c9aff}flink-csv-1.10.2.jar{color} &
{color:#4c9aff}flink-sql-connector-kafka_2.11-1.11.0.jar {color}
{color:#4c9aff}to the lib dir. I run the sql below with sql-client:{color}
_CREATE TABLE kafkaSource (_
_a VARCHAR,_
{color:#505f79} _b VARCHAR_{color}
{color:#505f79}_) WITH (_{color}
{color:#505f79} _'connector' = 'kafka',_{color}
{color:#505f79} _'topic' = 'source001',_{color}
{color:#505f79} _'properties.bootstrap.servers' = 'localhost:9092',_{color}
{color:#505f79} _'properties.group.id' = 'consumer001',_{color}
{color:#505f79} _'format' = 'csv',_{color}
{color:#505f79} _'scan.startup.mode' = 'latest-offset'_{color}
{color:#505f79}_);_{color}
{color:#505f79}_CREATE TABLE kafkaSink (_{color}
{color:#505f79} _a VARCHAR,_{color}
{color:#505f79} _b VARCHAR_{color}
{color:#505f79}_) WITH (_{color}
{color:#505f79} _'connector' = 'kafka',_{color}
{color:#505f79} _'topic' = 'sink001',_{color}
{color:#505f79} _'properties.bootstrap.servers' = 'localhost:9092',_{color}
{color:#505f79} _'format' = 'csv'_{color}
{color:#505f79}_);_{color}
{color:#505f79}_INSERT INTO kafkaSink_{color}
{color:#505f79}_SELECT *_{color}
{color:#505f79}_FROM kafkaSource;_{color}
{color:#172b4d}Then the exception {color}appeared.
> java.util.Collections$UnmodifiableList to field
> org.apache.flink.runtime.jobgraph.JobVertex.operatorIDs of type
> java.util.ArrayList in instance of org.apache.flink.runtime.jobgraph.JobVertex
> --
>
> Key: FLINK-19565
> URL: https://issues.apache.org/jira/browse/FLINK-19565
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.2
> Environment: I add {color:#4c9aff}flink-csv-1.10.2.jar{color} &
> [flink-sql-connector-kafka_2.12-1.10.2.jar|https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/1.10.2/flink-sql-connector-kafka_2.12-1.10.2.jar]{color:#4c9aff}
> {color}
> {color:#172b4d}to the lib dir. I run the sql below with sql-client:{color}
>
> _CREATE TABLE kafkaSource (_
> _a VARCHAR,_
> {color:#505f79} _b VARCHAR_{color}
> {color:#505f79}_) WITH (_{color}
> {color:#505f79} _'connector' = 'kafka',_{color}
> {color:#505f79} _'topic' = 'source001',_{color}
> {color:#505f79} _'properties.bootstrap.servers' = 'localhost:9092',_{color}
> {color:#505f79} _'properties.group.id' = 'consumer001',_{color}
> {color:#505f79} _'format' = 'csv',_{color}
> {color:#505f79} _'scan.startup.mode' = 'latest-offset'_{color}
> {color:#505f79}_);_{color}
> {color:#505f79}_CREATE TABLE kafkaSink (_{color}
> {color:#505f79} _a VARCHAR,_{color}
> {color:#505f79} _b VARCHAR_{color}
> {color:#505f79}_) WITH (_{color}
> {color:#505f79} _'connector' = 'kafka',_{color}
> {color:#505f79} _'topic' = 'sink001',_{color}
> {color:#505f79} _'properties.bootstrap.servers' = 'localhost:9092',_{color}
> {color:#505f79} _'format' = 'csv'_{color}
> {color:#505f79}_);_{color}
> {color:#505f79}_INSERT INTO kafkaSink_{color}
> {color:#505f79}_SELECT *_{color}
> {color:#505f79}_FROM kafkaSource;_{color}
>
> {color:#172b4d}Then the exception {color}appeared.
>
[jira] [Assigned] (FLINK-19534) build_wheels: linux failed with "ValueError: Incompatible component merge"
[
https://issues.apache.org/jira/browse/FLINK-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu reassigned FLINK-19534:
---
Assignee: Huang Xingbo
> build_wheels: linux failed with "ValueError: Incompatible component merge"
> --
>
> Key: FLINK-19534
> URL: https://issues.apache.org/jira/browse/FLINK-19534
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.11.2
>Reporter: Robert Metzger
>Assignee: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7276=logs=fe7ebddc-3e2f-5c50-79ee-226c8653f218=b2830442-93c7-50ff-36f4-5b3e2dca8c83
> {code}
> 2020-10-07T21:26:02.2736541Z + export PATH
> 2020-10-07T21:26:02.2737242Z +
> /home/vsts/work/1/s/flink-python/dev/.conda/bin/conda install -c conda-forge
> patchelf=0.11 -y
> 2020-10-07T21:26:08.7666087Z Collecting package metadata
> (current_repodata.json): ...working... failed
> 2020-10-07T21:26:08.7767051Z
> 2020-10-07T21:26:08.7767792Z # >> ERROR REPORT
> <<
> 2020-10-07T21:26:08.7768252Z
> 2020-10-07T21:26:08.7768724Z Traceback (most recent call last):
> 2020-10-07T21:26:08.7770155Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/exceptions.py",
> line 1062, in __call__
> 2020-10-07T21:26:08.7770935Z return func(*args, **kwargs)
> 2020-10-07T21:26:08.7771943Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/main.py",
> line 84, in _main
> 2020-10-07T21:26:08.7772700Z exit_code = do_call(args, p)
> 2020-10-07T21:26:08.7773701Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/conda_argparse.py",
> line 82, in do_call
> 2020-10-07T21:26:08.7774481Z exit_code = getattr(module,
> func_name)(args, parser)
> 2020-10-07T21:26:08.7775977Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/main_install.py",
> line 20, in execute
> 2020-10-07T21:26:08.067Z install(args, parser, 'install')
> 2020-10-07T21:26:08.7778141Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/install.py",
> line 256, in install
> 2020-10-07T21:26:08.7778910Z force_reinstall=context.force_reinstall
> or context.force,
> 2020-10-07T21:26:08.7780010Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 112, in solve_for_transaction
> 2020-10-07T21:26:08.7780853Z force_remove, force_reinstall)
> 2020-10-07T21:26:08.7781835Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 150, in solve_for_diff
> 2020-10-07T21:26:08.7782503Z force_remove)
> 2020-10-07T21:26:08.7783604Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 249, in solve_final_state
> 2020-10-07T21:26:08.7784374Z ssc = self._collect_all_metadata(ssc)
> 2020-10-07T21:26:08.7785369Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/common/io.py",
> line 88, in decorated
> 2020-10-07T21:26:08.7798322Z return f(*args, **kwds)
> 2020-10-07T21:26:08.7799259Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 389, in _collect_all_metadata
> 2020-10-07T21:26:08.7799750Z index, r = self._prepare(prepared_specs)
> 2020-10-07T21:26:08.7800610Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 974, in _prepare
> 2020-10-07T21:26:08.7801063Z self.subdirs, prepared_specs,
> self._repodata_fn)
> 2020-10-07T21:26:08.7801942Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/index.py",
> line 216, in get_reduced_index
> 2020-10-07T21:26:08.7802388Z push_record(record)
> 2020-10-07T21:26:08.7803182Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/index.py",
> line 190, in push_record
> 2020-10-07T21:26:08.7803624Z combined_depends =
> record.combined_depends
> 2020-10-07T21:26:08.7804456Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/models/records.py",
> line 326, in combined_depends
> 2020-10-07T21:26:08.7804901Z MatchSpec(spec, optional=True) for spec
> in self.constrains or ()
> 2020-10-07T21:26:08.7805615Z File
>
[jira] [Commented] (FLINK-19534) build_wheels: linux failed with "ValueError: Incompatible component merge"
[
https://issues.apache.org/jira/browse/FLINK-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212097#comment-17212097
]
Dian Fu commented on FLINK-19534:
-
[~hxbks2ks] Thanks a lot!
> build_wheels: linux failed with "ValueError: Incompatible component merge"
> --
>
> Key: FLINK-19534
> URL: https://issues.apache.org/jira/browse/FLINK-19534
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.11.2
>Reporter: Robert Metzger
>Assignee: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7276=logs=fe7ebddc-3e2f-5c50-79ee-226c8653f218=b2830442-93c7-50ff-36f4-5b3e2dca8c83
> {code}
> 2020-10-07T21:26:02.2736541Z + export PATH
> 2020-10-07T21:26:02.2737242Z +
> /home/vsts/work/1/s/flink-python/dev/.conda/bin/conda install -c conda-forge
> patchelf=0.11 -y
> 2020-10-07T21:26:08.7666087Z Collecting package metadata
> (current_repodata.json): ...working... failed
> 2020-10-07T21:26:08.7767051Z
> 2020-10-07T21:26:08.7767792Z # >> ERROR REPORT
> <<
> 2020-10-07T21:26:08.7768252Z
> 2020-10-07T21:26:08.7768724Z Traceback (most recent call last):
> 2020-10-07T21:26:08.7770155Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/exceptions.py",
> line 1062, in __call__
> 2020-10-07T21:26:08.7770935Z return func(*args, **kwargs)
> 2020-10-07T21:26:08.7771943Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/main.py",
> line 84, in _main
> 2020-10-07T21:26:08.7772700Z exit_code = do_call(args, p)
> 2020-10-07T21:26:08.7773701Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/conda_argparse.py",
> line 82, in do_call
> 2020-10-07T21:26:08.7774481Z exit_code = getattr(module,
> func_name)(args, parser)
> 2020-10-07T21:26:08.7775977Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/main_install.py",
> line 20, in execute
> 2020-10-07T21:26:08.067Z install(args, parser, 'install')
> 2020-10-07T21:26:08.7778141Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/install.py",
> line 256, in install
> 2020-10-07T21:26:08.7778910Z force_reinstall=context.force_reinstall
> or context.force,
> 2020-10-07T21:26:08.7780010Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 112, in solve_for_transaction
> 2020-10-07T21:26:08.7780853Z force_remove, force_reinstall)
> 2020-10-07T21:26:08.7781835Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 150, in solve_for_diff
> 2020-10-07T21:26:08.7782503Z force_remove)
> 2020-10-07T21:26:08.7783604Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 249, in solve_final_state
> 2020-10-07T21:26:08.7784374Z ssc = self._collect_all_metadata(ssc)
> 2020-10-07T21:26:08.7785369Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/common/io.py",
> line 88, in decorated
> 2020-10-07T21:26:08.7798322Z return f(*args, **kwds)
> 2020-10-07T21:26:08.7799259Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 389, in _collect_all_metadata
> 2020-10-07T21:26:08.7799750Z index, r = self._prepare(prepared_specs)
> 2020-10-07T21:26:08.7800610Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 974, in _prepare
> 2020-10-07T21:26:08.7801063Z self.subdirs, prepared_specs,
> self._repodata_fn)
> 2020-10-07T21:26:08.7801942Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/index.py",
> line 216, in get_reduced_index
> 2020-10-07T21:26:08.7802388Z push_record(record)
> 2020-10-07T21:26:08.7803182Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/index.py",
> line 190, in push_record
> 2020-10-07T21:26:08.7803624Z combined_depends =
> record.combined_depends
> 2020-10-07T21:26:08.7804456Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/models/records.py",
> line 326, in combined_depends
> 2020-10-07T21:26:08.7804901Z MatchSpec(spec, optional=True) for spec
> in self.constrains or ()
> 2020-10-07T21:26:08.7805615Z File
>
[jira] [Commented] (FLINK-19534) build_wheels: linux failed with "ValueError: Incompatible component merge"
[
https://issues.apache.org/jira/browse/FLINK-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212096#comment-17212096
]
Huang Xingbo commented on FLINK-19534:
--
Thanks [~dian.fu] and [~rmetzger] for reporting this. I will take a look.
> build_wheels: linux failed with "ValueError: Incompatible component merge"
> --
>
> Key: FLINK-19534
> URL: https://issues.apache.org/jira/browse/FLINK-19534
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.11.2
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7276=logs=fe7ebddc-3e2f-5c50-79ee-226c8653f218=b2830442-93c7-50ff-36f4-5b3e2dca8c83
> {code}
> 2020-10-07T21:26:02.2736541Z + export PATH
> 2020-10-07T21:26:02.2737242Z +
> /home/vsts/work/1/s/flink-python/dev/.conda/bin/conda install -c conda-forge
> patchelf=0.11 -y
> 2020-10-07T21:26:08.7666087Z Collecting package metadata
> (current_repodata.json): ...working... failed
> 2020-10-07T21:26:08.7767051Z
> 2020-10-07T21:26:08.7767792Z # >> ERROR REPORT
> <<
> 2020-10-07T21:26:08.7768252Z
> 2020-10-07T21:26:08.7768724Z Traceback (most recent call last):
> 2020-10-07T21:26:08.7770155Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/exceptions.py",
> line 1062, in __call__
> 2020-10-07T21:26:08.7770935Z return func(*args, **kwargs)
> 2020-10-07T21:26:08.7771943Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/main.py",
> line 84, in _main
> 2020-10-07T21:26:08.7772700Z exit_code = do_call(args, p)
> 2020-10-07T21:26:08.7773701Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/conda_argparse.py",
> line 82, in do_call
> 2020-10-07T21:26:08.7774481Z exit_code = getattr(module,
> func_name)(args, parser)
> 2020-10-07T21:26:08.7775977Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/main_install.py",
> line 20, in execute
> 2020-10-07T21:26:08.067Z install(args, parser, 'install')
> 2020-10-07T21:26:08.7778141Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/cli/install.py",
> line 256, in install
> 2020-10-07T21:26:08.7778910Z force_reinstall=context.force_reinstall
> or context.force,
> 2020-10-07T21:26:08.7780010Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 112, in solve_for_transaction
> 2020-10-07T21:26:08.7780853Z force_remove, force_reinstall)
> 2020-10-07T21:26:08.7781835Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 150, in solve_for_diff
> 2020-10-07T21:26:08.7782503Z force_remove)
> 2020-10-07T21:26:08.7783604Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 249, in solve_final_state
> 2020-10-07T21:26:08.7784374Z ssc = self._collect_all_metadata(ssc)
> 2020-10-07T21:26:08.7785369Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/common/io.py",
> line 88, in decorated
> 2020-10-07T21:26:08.7798322Z return f(*args, **kwds)
> 2020-10-07T21:26:08.7799259Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 389, in _collect_all_metadata
> 2020-10-07T21:26:08.7799750Z index, r = self._prepare(prepared_specs)
> 2020-10-07T21:26:08.7800610Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/solve.py",
> line 974, in _prepare
> 2020-10-07T21:26:08.7801063Z self.subdirs, prepared_specs,
> self._repodata_fn)
> 2020-10-07T21:26:08.7801942Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/index.py",
> line 216, in get_reduced_index
> 2020-10-07T21:26:08.7802388Z push_record(record)
> 2020-10-07T21:26:08.7803182Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/core/index.py",
> line 190, in push_record
> 2020-10-07T21:26:08.7803624Z combined_depends =
> record.combined_depends
> 2020-10-07T21:26:08.7804456Z File
> "/home/vsts/work/1/s/flink-python/dev/.conda/lib/python3.7/site-packages/conda/models/records.py",
> line 326, in combined_depends
> 2020-10-07T21:26:08.7804901Z MatchSpec(spec, optional=True) for spec
> in self.constrains or ()
> 2020-10-07T21:26:08.7805615Z File
[GitHub] [flink] zhuxiaoshang commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
zhuxiaoshang commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r503012630
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ private transient Counter numLateRecordsDropped;
+
+ @VisibleForTesting
+ public Counter getCounter() {
Review comment:
Reason for consideration of future sub-class extension?In that way,i'll
change the constant to `private` in `RowTimeRangeBoundedPrecedingFunction` and
`RowTimeRowsBoundedPrecedingFunction`.
WDYT
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on pull request #13576: [FLINK-19512] Introduce the new sink API
stevenzwu commented on pull request #13576: URL: https://github.com/apache/flink/pull/13576#issuecomment-706824133 thanks a lot for explaining. LGTM! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19489) SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets stuck
[
https://issues.apache.org/jira/browse/FLINK-19489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212093#comment-17212093
]
Dian Fu commented on FLINK-19489:
-
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7400=logs=298e20ef-7951-5965-0e79-ea664ddc435e=b4cd3436-dbe8-556d-3bca-42f92c3cbf2f
> SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets stuck
> ---
>
> Key: FLINK-19489
> URL: https://issues.apache.org/jira/browse/FLINK-19489
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7158=logs=298e20ef-7951-5965-0e79-ea664ddc435e=b4cd3436-dbe8-556d-3bca-42f92c3cbf2f
> {code}
> 020-10-01T21:55:34.9982203Z "main" #1 prio=5 os_prio=0 cpu=1048.80ms
> elapsed=921.99s tid=0x7f8c00015800 nid=0xf6e in Object.wait()
> [0x7f8c06648000]
> 2020-10-01T21:55:34.9982807Zjava.lang.Thread.State: WAITING (on object
> monitor)
> 2020-10-01T21:55:34.9983177Z at
> java.lang.Object.wait(java.base@11.0.7/Native Method)
> 2020-10-01T21:55:34.9983871Z - waiting on <0x8e0be190> (a
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
> 2020-10-01T21:55:34.9984581Z at
> java.lang.Thread.join(java.base@11.0.7/Thread.java:1305)
> 2020-10-01T21:55:34.9985433Z - waiting to re-lock in wait()
> <0x8e0be190> (a
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
> 2020-10-01T21:55:34.9985998Z at
> org.apache.flink.core.testutils.CheckedThread.trySync(CheckedThread.java:112)
> 2020-10-01T21:55:34.9986511Z at
> org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:100)
> 2020-10-01T21:55:34.9987004Z at
> org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:89)
> 2020-10-01T21:55:34.9987707Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread.shutdown(SplitFetcherTest.java:301)
> 2020-10-01T21:55:34.9988427Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent(SplitFetcherTest.java:131)
> 2020-10-01T21:55:34.9989025Z at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.7/Native
> Method)
> 2020-10-01T21:55:34.9989531Z at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.7/NativeMethodAccessorImpl.java:62)
> 2020-10-01T21:55:34.9990117Z at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.7/DelegatingMethodAccessorImpl.java:43)
> 2020-10-01T21:55:34.9990626Z at
> java.lang.reflect.Method.invoke(java.base@11.0.7/Method.java:566)
> 2020-10-01T21:55:34.9991078Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-01T21:55:34.9991602Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-01T21:55:34.9992119Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-01T21:55:34.9992749Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-01T21:55:34.9993229Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-01T21:55:34.9993700Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-01T21:55:34.9994202Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-01T21:55:34.9994670Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-01T21:55:34.9995098Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-01T21:55:34.9995524Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-01T21:55:34.9995965Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-01T21:55:34.9996403Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-01T21:55:34.9996816Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-01T21:55:34.9997268Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-10-01T21:55:34.9997695Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-10-01T21:55:34.9998077Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-01T21:55:34.9998510Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-01T21:55:34.9998941Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-01T21:55:34.380Z at
>
[GitHub] [flink] flinkbot edited a comment on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot edited a comment on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7391) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7402) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7393) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19539) JMXReporterFactoryTest fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-19539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212086#comment-17212086
]
Dian Fu commented on FLINK-19539:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7400=logs=e1276d0f-df12-55ec-86b5-c0ad597d83c9=906e9244-f3be-5604-1979-e767c8a6f6d9
> JMXReporterFactoryTest fails on Azure
> -
>
> Key: FLINK-19539
> URL: https://issues.apache.org/jira/browse/FLINK-19539
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
>Affects Versions: 1.12.0
>Reporter: Roman Khachatryan
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.12.0
>
>
> The following build failed
> [https://dev.azure.com/khachatryanroman/810e80cc-0656-4d3c-9d8c-186764456a01/_apis/build/builds/289/logs/106]
> {code}
> [ERROR]
> testPortRangeArgument(org.apache.flink.metrics.jmx.JMXReporterFactoryTest)
> Time elapsed: 0.02 s <<< FAILURE!
> java.lang.AssertionError:
>
> Expected: (a value equal to or greater than <9000> and a value less than or
> equal to <9010>)
> but: a value less than or equal to <9010> <9040> was greater than <9010>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.metrics.jmx.JMXReporterFactoryTest.testPortRangeArgument(JMXReporterFactoryTest.java:46)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
>
> I see the following problems in the code:
> - tests in JMXReporterFactoryTest assumes JMXService.jmxServer wasn't started
> or was stopped
> - JMXService.jmxServer is not volatile
> cc: [~chesnay], [~rongr]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17424) SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) failed due to download error
[
https://issues.apache.org/jira/browse/FLINK-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212084#comment-17212084
]
Dian Fu commented on FLINK-17424:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7400=logs=739e6eac-8312-5d31-d437-294c4d26fced=a68b8d89-50e9-5977-4500-f4fde4f57f9b
> SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) failed due to
> download error
>
>
> Key: FLINK-17424
> URL: https://issues.apache.org/jira/browse/FLINK-17424
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yu Li
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> `SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1)` failed in
> release-1.10 crone job with below error:
> {noformat}
> Preparing Elasticsearch(version=7)...
> Downloading Elasticsearch from
> https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.1-linux-x86_64.tar.gz
> ...
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 4 276M4 13.3M0 0 28.8M 0 0:00:09 --:--:-- 0:00:09 28.8M
> 42 276M 42 117M0 0 80.7M 0 0:00:03 0:00:01 0:00:02 80.7M
> 70 276M 70 196M0 0 79.9M 0 0:00:03 0:00:02 0:00:01 79.9M
> 89 276M 89 248M0 0 82.3M 0 0:00:03 0:00:03 --:--:-- 82.4M
> curl: (56) GnuTLS recv error (-54): Error in the pull function.
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
> 0curl: (7) Failed to connect to localhost port 9200: Connection refused
> [FAIL] Test script contains errors.
> {noformat}
> https://api.travis-ci.org/v3/job/680222168/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19423) Primary key position cause JDBC SQL upsert sink ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212083#comment-17212083
]
Jark Wu commented on FLINK-19423:
-
The following SQL will produce DELETE/UPDATE_BEFORE to the sink.
{code:sql}
select *
from (
select key, count(*) as cnt
from T
group by key
) where cnt < 10;
{code}
> Primary key position cause JDBC SQL upsert sink ArrayIndexOutOfBoundsException
> --
>
> Key: FLINK-19423
> URL: https://issues.apache.org/jira/browse/FLINK-19423
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC, Table SQL / Ecosystem
>Affects Versions: 1.11.1
>Reporter: limbo
>Priority: Major
>
> We found that the primary key position can cause
> ArrayIndexOutOfBoundsException
> the sink like that( the primary key select the position of 1, 3):
> {code:java}
> CREATE TABLE `test`(
> col1 STRING,
> col2 STRING,
> col3 STRING,
> PRIMARY KEY (col1, col3) NOT ENFORCED ) WITH (
> 'connector' = 'jdbc',
> ...
> ){code}
> when the DELETE (cdc message) come , it will raise
> ArrayIndexOutOfBoundsException:
> {code:java}
> Caused by: java.lang.RuntimeException: Writing records to JDBC failed....
> 10 moreCaused by: java.lang.ArrayIndexOutOfBoundsException: 2at
> org.apache.flink.table.data.GenericRowData.getString(GenericRowData.java:169)
> at
> org.apache.flink.table.data.RowData.lambda$createFieldGetter$245ca7d1$1(RowData.java:310)
> at
> org.apache.flink.connector.jdbc.table.JdbcDynamicOutputFormatBuilder.getPrimaryKey(JdbcDynamicOutputFormatBuilder.java:216)
> at
> org.apache.flink.connector.jdbc.table.JdbcDynamicOutputFormatBuilder.lambda$createRowKeyExtractor$7(JdbcDynamicOutputFormatBuilder.java:193)
> at
> org.apache.flink.connector.jdbc.table.JdbcDynamicOutputFormatBuilder.lambda$createKeyedRowExecutor$3fd497bb$1(JdbcDynamicOutputFormatBuilder.java:128)
> at
> org.apache.flink.connector.jdbc.internal.executor.KeyedBatchStatementExecutor.executeBatch(KeyedBatchStatementExecutor.java:71)
> at
> org.apache.flink.connector.jdbc.internal.executor.BufferReduceStatementExecutor.executeBatch(BufferReduceStatementExecutor.java:99)
> at
> org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.attemptFlush(JdbcBatchingOutputFormat.java:200)
> at
> org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.flush(JdbcBatchingOutputFormat.java:171)
> ... 8 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] libenchao commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
libenchao commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r503005784
##
File path:
flink-table/flink-table-runtime-blink/src/test/java/org/apache/flink/table/runtime/operators/over/RowTimeOverWindowTestBase.java
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.over;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.generated.AggsHandleFunction;
+import org.apache.flink.table.runtime.generated.GeneratedAggsHandleFunction;
+import org.apache.flink.table.runtime.util.BinaryRowDataKeySelector;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarCharType;
+
+/**
+ * Base class for row-time over window test.
+ **/
+public class RowTimeOverWindowTestBase {
+ protected static GeneratedAggsHandleFunction aggsHandleFunction =
+ new GeneratedAggsHandleFunction("Function", "", new Object[0]) {
+ @Override
+ public AggsHandleFunction newInstance(ClassLoader
classLoader) {
+ return new SumAggsHandleFunction(1);
+ }
+ };
+
+ protected LogicalType[] inputFieldTypes = new LogicalType[]{
+ new VarCharType(VarCharType.MAX_LENGTH),
+ new BigIntType(),
+ new BigIntType()
+ };
+ protected LogicalType[] accTypes = new LogicalType[]{new BigIntType()};
+
+ protected BinaryRowDataKeySelector keySelector = new
BinaryRowDataKeySelector(new int[]{0}, inputFieldTypes);
+ protected TypeInformation keyType =
keySelector.getProducedType();
+
+ protected OneInputStreamOperatorTestHarness
createTestHarness(
+ KeyedProcessOperator operator)
throws Exception {
Review comment:
```suggestion
KeyedProcessOperator
operator) throws Exception {
```
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
Review comment:
could be `private`?
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractRowTimeUnboundedPrecedingOver.java
##
@@ -68,6 +70,17 @@
protected transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ private transient Counter numLateRecordsDropped;
+
+ @VisibleForTesting
+ public Counter getCounter() {
Review comment:
could be `protected`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot edited a comment on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7393) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7391) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7402) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Shawn-Hx commented on pull request #13410: [FLINK-19247][docs-zh] Update Chinese documentation after removal of Kafka 0.10 and 0.11
Shawn-Hx commented on pull request #13410: URL: https://github.com/apache/flink/pull/13410#issuecomment-706817037 Hi, @klion26 . I have made changes according to your advice. Please take a look. Thanks~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong commented on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
wangxlong commented on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706815374 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19455) Module 'flink-sql-connector-hive-2.3.6', 'flink-docs' and 'flnk-dist' build fail by maven-enforcer-plugin
[
https://issues.apache.org/jira/browse/FLINK-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212072#comment-17212072
]
Yangze Guo commented on FLINK-19455:
[~chesnay] Could you take a look? I think it might be related to FLINK-19243.
> Module 'flink-sql-connector-hive-2.3.6', 'flink-docs' and 'flnk-dist' build
> fail by maven-enforcer-plugin
> -
>
> Key: FLINK-19455
> URL: https://issues.apache.org/jira/browse/FLINK-19455
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.12.0
>Reporter: hailong wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> I run command 'mvn clean package' in flink-sql-connector-hive-2.3.6, and it
> failed,
> {code:java}
> Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with
> message:
> Found Banned Dependency: org.apache.kafka:kafka_2.10:jar:0.10.2.0
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> For root pom has a rule as follow:
> {code:java}
> bannedDependencies>
>
> *:*_2.12
> *:*_2.10
>
>
> {code}
> As for kafka_2.10 dependency is useless, So we can exclude it as follow:
> {code:java}
>
>org.apache.hive
>hive-exec
>2.3.6
>
>
> log4j
> log4j
>
>
> org.slf4j
> slf4j-log4j12
>
>
> org.pentaho
> pentaho-aggdesigner-algorithm
>
>
> org.apache.kafka
> kafka_2.10
>
>
> {code}
> It also failed in flink-docs and flink-disk module:
> {code:java}
> Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with
> message:
> Found Banned Dependency: org.yaml:snakeyaml:jar:1.24
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] xiaolong-sn commented on a change in pull request #13015: [FLINK-18536][kinesis] Adding enhanced fan-out related configurations.
xiaolong-sn commented on a change in pull request #13015:
URL: https://github.com/apache/flink/pull/13015#discussion_r503000401
##
File path:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/publisher/fanout/FanOutRecordPublisherConfiguration.java
##
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout;
+
+import
org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants;
+import
org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.EFORegistrationType;
+import
org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.RecordPublisherType;
+import org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * This is a configuration class for enhanced fan-out components.
+ */
+public class FanOutRecordPublisherConfiguration {
+
+ /**
+* The efo registration type for de-/registration of streams.
+*/
+ private final EFORegistrationType efoRegistrationType;
+
+ /**
+* The efo stream consumer name. Should not be Null if the
efoRegistrationType is either LAZY or EAGER.
+*/
+ @Nullable
+ private String consumerName;
+
+ /**
+* The manual set efo consumer arns for each stream. Should not be Null
if the efoRegistrationType is NONE
+*/
+ @Nullable
+ private Map streamConsumerArns;
+
+ /**
+* Base backoff millis for the deregister stream operation.
+*/
+ private final int subscribeToShardMaxRetries;
+
+ /**
+* Maximum backoff millis for the subscribe to shard operation.
+*/
+ private final long subscribeToShardMaxBackoffMillis;
+
+ /**
+* Base backoff millis for the subscribe to shard operation.
+*/
+ private final long subscribeToShardBaseBackoffMillis;
+
+ /**
+* Exponential backoff power constant for the subscribe to shard
operation.
+*/
+ private final double subscribeToShardExpConstant;
+
+ /**
+* Base backoff millis for the register stream operation.
+*/
+ private final long registerStreamBaseBackoffMillis;
+
+ /**
+* Maximum backoff millis for the register stream operation.
+*/
+ private final long registerStreamMaxBackoffMillis;
+
+ /**
+* Exponential backoff power constant for the register stream operation.
+*/
+ private final double registerStreamExpConstant;
+
+ /**
+* Maximum retry attempts for the register stream operation.
+*/
+ private final int registerStreamMaxRetries;
+
+ /**
+* Base backoff millis for the deregister stream operation.
+*/
+ private final long deregisterStreamBaseBackoffMillis;
+
+ /**
+* Maximum backoff millis for the deregister stream operation.
+*/
+ private final long deregisterStreamMaxBackoffMillis;
+
+ /**
+* Exponential backoff power constant for the deregister stream
operation.
+*/
+ private final double deregisterStreamExpConstant;
+
+ /**
+* Maximum retry attempts for the deregister stream operation.
+*/
+ private final int deregisterStreamMaxRetries;
+
+ /**
+* Max retries for the describe stream operation.
+*/
+ private final int describeStreamMaxRetries;
+
+ /**
+* Backoff millis for the describe stream operation.
+*/
+ private final long describeStreamBaseBackoffMillis;
+
+ /**
+* Maximum backoff millis for the describe stream operation.
+*/
+ private final long describeStreamMaxBackoffMillis;
+
+ /**
+* Exponential backoff power constant for the describe stream operation.
+*/
+ private final double
[GitHub] [flink] xiaolong-sn commented on a change in pull request #13015: [FLINK-18536][kinesis] Adding enhanced fan-out related configurations.
xiaolong-sn commented on a change in pull request #13015:
URL: https://github.com/apache/flink/pull/13015#discussion_r503000484
##
File path:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisConfigUtil.java
##
@@ -44,13 +51,18 @@
@Internal
public class KinesisConfigUtil {
- /** Maximum number of items to pack into an PutRecords request. **/
+ /**
+* Maximum number of items to pack into an PutRecords request.
+**/
protected static final String COLLECTION_MAX_COUNT =
"CollectionMaxCount";
- /** Maximum number of items to pack into an aggregated record. **/
+ /**
+* Maximum number of items to pack into an aggregated record.
+**/
Review comment:
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] guoweiM commented on a change in pull request #13576: [FLINK-19512] Introduce the new sink API
guoweiM commented on a change in pull request #13576:
URL: https://github.com/apache/flink/pull/13576#discussion_r502990728
##
File path:
flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.MetricGroup;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * This interface lets the sink developer build a simple sink topology, which
could guarantee the exactly once
+ * semantics in both batch and stream execution mode if there is a {@link
Committer} or {@link GlobalCommitter}.
+ * 1. The {@link Writer} is responsible for producing the committable.
+ * 2. The {@link Committer} is responsible for committing a single committable.
+ * 3. The {@link GlobalCommitter} is responsible for committing an aggregated
committable, which we called the global
+ *committable. There is only one instance of the {@link GlobalCommitter}.
+ * Note: Developers need to ensure the idempotence of {@link Committer} and
{@link GlobalCommitter}.
+ *
+ * @param The type of the sink's input
+ * @param The type of the committable data
+ * @paramThe type of the aggregated committable
+ * @param The type of the writer's state
+ */
+@Experimental
+public interface Sink {
Review comment:
Yes steven. At first I have same feeling as you. But I found that there
may be four combinations:
1. `Writer`
2. `Writer` + `Committer`
3. `Writer` + `GlobalCommitter`
4. `Writer` + `Committer` + `GlobalCommitter`
We have to provide the four "different" `XXXSink` classes if we want provide
all these pattern to the developer.
For me I think it might too many on term of the number of a sink api.
Actually I would like to say that this sink api is more like a "building
topology" pattern. (Of course it is not so flexible currently).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] guoweiM commented on a change in pull request #13576: [FLINK-19512] Introduce the new sink API
guoweiM commented on a change in pull request #13576:
URL: https://github.com/apache/flink/pull/13576#discussion_r502990728
##
File path:
flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.MetricGroup;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * This interface lets the sink developer build a simple sink topology, which
could guarantee the exactly once
+ * semantics in both batch and stream execution mode if there is a {@link
Committer} or {@link GlobalCommitter}.
+ * 1. The {@link Writer} is responsible for producing the committable.
+ * 2. The {@link Committer} is responsible for committing a single committable.
+ * 3. The {@link GlobalCommitter} is responsible for committing an aggregated
committable, which we called the global
+ *committable. There is only one instance of the {@link GlobalCommitter}.
+ * Note: Developers need to ensure the idempotence of {@link Committer} and
{@link GlobalCommitter}.
+ *
+ * @param The type of the sink's input
+ * @param The type of the committable data
+ * @paramThe type of the aggregated committable
+ * @param The type of the writer's state
+ */
+@Experimental
+public interface Sink {
Review comment:
Yes steven. At first I have same feeling as you. But I found that there
may be four combinations:
1. `Writer`
2. `Writer` + `Committer`
3. `Writer` + `GlobalCommitter`
4. `Writer` + `Committer` + `GlobalCommitter`
So we have to provide the four "different" `XXXSink` classes if we want
provide all these pattern to the developer.
For me I think it might too many on term of the number of a sink api.
Actually I would like to say that this sink api is more like a "building
topology" pattern. (Of course it is not so flexible currently).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] guoweiM commented on a change in pull request #13576: [FLINK-19512] Introduce the new sink API
guoweiM commented on a change in pull request #13576:
URL: https://github.com/apache/flink/pull/13576#discussion_r502987944
##
File path:
flink-core/src/main/java/org/apache/flink/api/connector/sink/Writer.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.annotation.Experimental;
+
+import java.util.List;
+
+/**
+ * The interface is responsible for writing data and handling any potential
tmp area used to write yet un-staged data, e.g. in-progress files.
+ * The data (or metadata pointing to where the actual data is staged) ready to
commit is returned to the system by the {@link #prepareCommit(boolean)}.
+ *
+ * @param The type of the writer's input
+ * @param The type of the committable data
+ * @paramThe type of the writer's state
+ */
+@Experimental
+public interface Writer extends AutoCloseable {
+
+ /**
+* Add an element to the writer.
+* @param element The input record
+* @param context The additional information about the input record
+*/
+ void write(InputT element, Context context);
+
+ /**
+* Prepare for a commit.
+* @param flush whether flushing the un-staged data or not
+* @return The data is ready to commit.
+*/
+ List prepareCommit(boolean flush);
+
+ /**
+* @return the writer's state.
+*/
+ List snapshotState();
Review comment:
Currently the benefit of `List` is that this could
rebalance the workload when changing the parallelism. For example the
`StreamingFileSink` redistributed the `Bucket` when the parallelism scales up
or down.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] guoweiM commented on a change in pull request #13576: [FLINK-19512] Introduce the new sink API
guoweiM commented on a change in pull request #13576:
URL: https://github.com/apache/flink/pull/13576#discussion_r502987944
##
File path:
flink-core/src/main/java/org/apache/flink/api/connector/sink/Writer.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.annotation.Experimental;
+
+import java.util.List;
+
+/**
+ * The interface is responsible for writing data and handling any potential
tmp area used to write yet un-staged data, e.g. in-progress files.
+ * The data (or metadata pointing to where the actual data is staged) ready to
commit is returned to the system by the {@link #prepareCommit(boolean)}.
+ *
+ * @param The type of the writer's input
+ * @param The type of the committable data
+ * @paramThe type of the writer's state
+ */
+@Experimental
+public interface Writer extends AutoCloseable {
+
+ /**
+* Add an element to the writer.
+* @param element The input record
+* @param context The additional information about the input record
+*/
+ void write(InputT element, Context context);
+
+ /**
+* Prepare for a commit.
+* @param flush whether flushing the un-staged data or not
+* @return The data is ready to commit.
+*/
+ List prepareCommit(boolean flush);
+
+ /**
+* @return the writer's state.
+*/
+ List snapshotState();
Review comment:
Currently the benefit of `List` is that this could
rebalance the workload when changing the parallelism. For example currently the
`StreamingFileSink` redistributed the `Bucket` when the parallelism scales up
or down.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13550: [FLINK-19474] Implement a state backend that holds a single key at a time
flinkbot edited a comment on pull request #13550: URL: https://github.com/apache/flink/pull/13550#issuecomment-704399024 ## CI report: * eaa536db01b506ec59d94a3120fb93ba81cbe69e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7399) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13550: [FLINK-19474] Implement a state backend that holds a single key at a time
flinkbot edited a comment on pull request #13550: URL: https://github.com/apache/flink/pull/13550#issuecomment-704399024 ## CI report: * 804b8f73417efce2dceae397211094b8991e8590 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7398) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7336) * eaa536db01b506ec59d94a3120fb93ba81cbe69e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7399) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
flinkbot edited a comment on pull request #13519: URL: https://github.com/apache/flink/pull/13519#issuecomment-701236311 ## CI report: * fa89bab5fc48538a35ccc69508da0d89527f98ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7395) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a change in pull request #13576: [FLINK-19512] Introduce the new sink API
stevenzwu commented on a change in pull request #13576:
URL: https://github.com/apache/flink/pull/13576#discussion_r502739686
##
File path:
flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.MetricGroup;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * This interface lets the sink developer build a simple sink topology, which
could guarantee the exactly once
+ * semantics in both batch and stream execution mode if there is a {@link
Committer} or {@link GlobalCommitter}.
+ * 1. The {@link Writer} is responsible for producing the committable.
+ * 2. The {@link Committer} is responsible for committing a single committable.
+ * 3. The {@link GlobalCommitter} is responsible for committing an aggregated
committable, which we called the global
+ *committable. There is only one instance of the {@link GlobalCommitter}.
+ * Note: Developers need to ensure the idempotence of {@link Committer} and
{@link GlobalCommitter}.
+ *
+ * @param The type of the sink's input
+ * @param The type of the committable data
+ * @paramThe type of the aggregated committable
+ * @param The type of the writer's state
+ */
+@Experimental
+public interface Sink {
Review comment:
looking at the Optional return value for createCommitter and
createGlobalCommitter, I am wondering if we should have separate
`TranscationalSink` and `GlobalTransactionalSink`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a change in pull request #13576: [FLINK-19512] Introduce the new sink API
stevenzwu commented on a change in pull request #13576:
URL: https://github.com/apache/flink/pull/13576#discussion_r502739686
##
File path:
flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.MetricGroup;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * This interface lets the sink developer build a simple sink topology, which
could guarantee the exactly once
+ * semantics in both batch and stream execution mode if there is a {@link
Committer} or {@link GlobalCommitter}.
+ * 1. The {@link Writer} is responsible for producing the committable.
+ * 2. The {@link Committer} is responsible for committing a single committable.
+ * 3. The {@link GlobalCommitter} is responsible for committing an aggregated
committable, which we called the global
+ *committable. There is only one instance of the {@link GlobalCommitter}.
+ * Note: Developers need to ensure the idempotence of {@link Committer} and
{@link GlobalCommitter}.
+ *
+ * @param The type of the sink's input
+ * @param The type of the committable data
+ * @paramThe type of the aggregated committable
+ * @param The type of the writer's state
+ */
+@Experimental
+public interface Sink {
Review comment:
looking at the Optional return value for createCommitter and
createGlobalCommitter, I am wondering if we should have a
`GlobalTransactionalSink` that has `createGlobalCommitter`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13587: [FLINK-18004][docs] Update checkpoint UI related pictures in documentation
flinkbot edited a comment on pull request #13587: URL: https://github.com/apache/flink/pull/13587#issuecomment-706735017 ## CI report: * 0039abf44ea2a2aec6942c4c572656a5bf1a5f90 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7397) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot edited a comment on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7393) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7391) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13550: [FLINK-19474] Implement a state backend that holds a single key at a time
flinkbot edited a comment on pull request #13550: URL: https://github.com/apache/flink/pull/13550#issuecomment-704399024 ## CI report: * 804b8f73417efce2dceae397211094b8991e8590 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7398) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7336) * eaa536db01b506ec59d94a3120fb93ba81cbe69e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7399) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13550: [FLINK-19474] Implement a state backend that holds a single key at a time
flinkbot edited a comment on pull request #13550: URL: https://github.com/apache/flink/pull/13550#issuecomment-704399024 ## CI report: * 804b8f73417efce2dceae397211094b8991e8590 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7398) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7336) * eaa536db01b506ec59d94a3120fb93ba81cbe69e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13587: [FLINK-18004][docs] Update checkpoint UI related pictures in documentation
flinkbot edited a comment on pull request #13587: URL: https://github.com/apache/flink/pull/13587#issuecomment-706735017 ## CI report: * 0039abf44ea2a2aec6942c4c572656a5bf1a5f90 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7397) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13584: [hotfix][typo] Fix typo in MiniCluster
flinkbot edited a comment on pull request #13584: URL: https://github.com/apache/flink/pull/13584#issuecomment-706656585 ## CI report: * ab9911fee445c24dba308c6bae19ef51db8abec8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7392) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7390) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13550: [FLINK-19474] Implement a state backend that holds a single key at a time
flinkbot edited a comment on pull request #13550: URL: https://github.com/apache/flink/pull/13550#issuecomment-704399024 ## CI report: * 804b8f73417efce2dceae397211094b8991e8590 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7336) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7398) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #13550: [FLINK-19474] Implement a state backend that holds a single key at a time
dawidwys commented on pull request #13550: URL: https://github.com/apache/flink/pull/13550#issuecomment-706737508 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13587: [FLINK-18004][docs] Update checkpoint UI related pictures in documentation
flinkbot commented on pull request #13587: URL: https://github.com/apache/flink/pull/13587#issuecomment-706735017 ## CI report: * 0039abf44ea2a2aec6942c4c572656a5bf1a5f90 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
flinkbot edited a comment on pull request #13519: URL: https://github.com/apache/flink/pull/13519#issuecomment-701236311 ## CI report: * c3ce856573d7f6b1240e97113eab4e35c705e638 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7182) * fa89bab5fc48538a35ccc69508da0d89527f98ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7395) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13587: [FLINK-18004][docs] Update checkpoint UI related pictures in documentation
flinkbot commented on pull request #13587: URL: https://github.com/apache/flink/pull/13587#issuecomment-706733342 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0039abf44ea2a2aec6942c4c572656a5bf1a5f90 (Sun Oct 11 16:47:48 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18004).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18004) Update checkpoint UI related pictures in documentation
[ https://issues.apache.org/jira/browse/FLINK-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18004: --- Labels: pull-request-available (was: ) > Update checkpoint UI related pictures in documentation > -- > > Key: FLINK-18004 > URL: https://issues.apache.org/jira/browse/FLINK-18004 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.11.0 >Reporter: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > After FLINK-13390 which clarifies what the "state size" means on incremental > checkpoint, the checkpoint UI has already changed, and we should also update > related documentation to not mislead users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Myasuka opened a new pull request #13587: [FLINK-18004][docs] Update checkpoint UI related pictures in documentation
Myasuka opened a new pull request #13587: URL: https://github.com/apache/flink/pull/13587 ## What is the purpose of the change After FLINK-13390 which clarifies what the "state size" means on incremental checkpoint, the checkpoint UI has already changed, and we should also update related documentation to not mislead users. ## Brief change log - Update checkpoint related web UI. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? docs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19555) Remove runAsyncWithoutFencing from MesosResourceManager
[
https://issues.apache.org/jira/browse/FLINK-19555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-19555.
-
Resolution: Fixed
Fixed via bfd85fdb76b672fc519cfe635e3334316be9d42d
> Remove runAsyncWithoutFencing from MesosResourceManager
> ---
>
> Key: FLINK-19555
> URL: https://issues.apache.org/jira/browse/FLINK-19555
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Mesos, Runtime / Coordination
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> With FLINK-9936 we introduced that the {{MesosResourceManager}} can process
> {{Disconnected}} messages also if the {{MesosResourceManager}} is not the
> leader. Since we are re-creating the monitors and coordinators in case of a
> new leader session. I believe that this call is no longer necessary. Hence, I
> propose to remove it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann closed pull request #13573: [FLINK-19555] Remove unfenced execution of Disconnected message from MesosResourceManager
tillrohrmann closed pull request #13573: URL: https://github.com/apache/flink/pull/13573 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #13573: [FLINK-19555] Remove unfenced execution of Disconnected message from MesosResourceManager
tillrohrmann commented on pull request #13573: URL: https://github.com/apache/flink/pull/13573#issuecomment-706732164 Thanks for the review @xintongsong. Jepsen-ci has passed: https://travis-ci.com/github/dataArtisans/flink-jepsen-ci/builds/189052564. Merging this PR now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
flinkbot edited a comment on pull request #13519: URL: https://github.com/apache/flink/pull/13519#issuecomment-701236311 ## CI report: * c3ce856573d7f6b1240e97113eab4e35c705e638 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7182) * fa89bab5fc48538a35ccc69508da0d89527f98ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
zhuxiaoshang commented on a change in pull request #13519:
URL: https://github.com/apache/flink/pull/13519#discussion_r502937004
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/RowTimeRangeBoundedPrecedingFunction.java
##
@@ -86,12 +90,25 @@
private transient AggsHandleFunction function;
+ //
+ // Metrics
+ //
+ protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME =
"numLateRecordsDropped";
+ protected static final String LATE_ELEMENTS_DROPPED_RATE_METRIC_NAME =
"lateRecordsDroppedRate";
+ private transient Counter numLateRecordsDropped;
+ private transient Meter lateRecordsDroppedRate;
+
+ @VisibleForTesting
+ public Counter getCounter() {
+ return numLateRecordsDropped;
+ }
+
public RowTimeRangeBoundedPrecedingFunction(
- GeneratedAggsHandleFunction genAggsHandler,
- LogicalType[] accTypes,
- LogicalType[] inputFieldTypes,
- long precedingOffset,
- int rowTimeIdx) {
+ GeneratedAggsHandleFunction genAggsHandler,
+ LogicalType[] accTypes,
+ LogicalType[] inputFieldTypes,
+ long precedingOffset,
+ int rowTimeIdx) {
Review comment:
Sorry for my IDEA code format changed this,i'll revert it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang commented on a change in pull request #13519: [FLINK-18850][Table SQL / Runtime]Add late records dropped metric for row time over windows
zhuxiaoshang commented on a change in pull request #13519: URL: https://github.com/apache/flink/pull/13519#discussion_r502935531 ## File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/over/RowTimeRangeBoundedPrecedingFunction.java ## @@ -86,12 +90,25 @@ private transient AggsHandleFunction function; + // + // Metrics + // + protected static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped"; + protected static final String LATE_ELEMENTS_DROPPED_RATE_METRIC_NAME = "lateRecordsDroppedRate"; Review comment: I have saw the discussions in FLINK-15431,i'll follow it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13586: Single task result partial record header
flinkbot edited a comment on pull request #13586: URL: https://github.com/apache/flink/pull/13586#issuecomment-706724200 ## CI report: * e3cef29a579eff88416b9e34bfe51f022d38a0d0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7394) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13586: Single task result partial record header
flinkbot commented on pull request #13586: URL: https://github.com/apache/flink/pull/13586#issuecomment-706724200 ## CI report: * e3cef29a579eff88416b9e34bfe51f022d38a0d0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15156) Warn user if System.exit() is called in user code
[ https://issues.apache.org/jira/browse/FLINK-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211949#comment-17211949 ] Hwanju Kim commented on FLINK-15156: FWIW, the following is what we have done: * Flink user security manager is added for general user sandbox checking, where currently only the exit is checked (others can be added later here). * The added one is forwarding all the checks but its overridden ones to previous security manager, if any (like decorator). * The security manager is set when JM and TM start (if configured, as described in the last bullet point). * Exit check has enabling/disabling point via a method only to affect user code, as Flink runtime needs to exit for some cases (e.g., fatal error). ** Once enabled, any thread spawned from the main thread inherits the enable flag. * What's enclosed by this enabled exit check is currently best-effort, not covering all the places where user code is involved. Main places are: ** main() in JM (currently for invokeInteractiveModeForExecution) ** StreamTask.invoke, triggerCheckpoint, cancel. * New exception, UserSystemExitException, is defined to be thrown when user code attempts to exit JVM. This has default message to warn the user. ** In main(), it's wrapped into ProgramInvocationException. ** In UDF, it fails the exiting task, thereby shipping the exception to JM triggering fail-over. * This security manager is only added if configuration (under security section) in flink-conf.yaml is enabled (disabled by default). The configuration is per check case (but currently only disallow-system-exit is available). Please let me know if anyone wants to review the patch, or just discussion if anything does not make sense. > Warn user if System.exit() is called in user code > - > > Key: FLINK-15156 > URL: https://issues.apache.org/jira/browse/FLINK-15156 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > It would make debugging Flink errors easier if we would intercept and log > calls to System.exit() through the SecurityManager. > A user recently had an error where the JobManager was shutting down because > of a System.exit() in the user code: > https://lists.apache.org/thread.html/b28dabcf3068d489f38399c456c80d48569fcdf74b15f8bb95d532d0%40%3Cuser.flink.apache.org%3E > If I remember correctly, we had such issues before. > I put this ticket into the "Runtime / Coordination" component, as it is > mostly about improving the usability / debuggability in that area. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot edited a comment on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7393) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7391) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13586: Single task result partial record header
flinkbot commented on pull request #13586: URL: https://github.com/apache/flink/pull/13586#issuecomment-706721089 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e3cef29a579eff88416b9e34bfe51f022d38a0d0 (Sun Oct 11 15:23:15 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur opened a new pull request #13586: Single task result partial record header
curcur opened a new pull request #13586: URL: https://github.com/apache/flink/pull/13586 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot edited a comment on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7391) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7393) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong commented on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
wangxlong commented on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706717574 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13584: [hotfix][typo] Fix typo in MiniCluster
flinkbot edited a comment on pull request #13584: URL: https://github.com/apache/flink/pull/13584#issuecomment-706656585 ## CI report: * ab9911fee445c24dba308c6bae19ef51db8abec8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7392) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7390) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13584: [hotfix][typo] Fix typo in MiniCluster
flinkbot edited a comment on pull request #13584: URL: https://github.com/apache/flink/pull/13584#issuecomment-706656585 ## CI report: * ab9911fee445c24dba308c6bae19ef51db8abec8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7390) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7392) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Shawn-Hx commented on pull request #13584: [hotfix][typo] Fix typo in MiniCluster
Shawn-Hx commented on pull request #13584: URL: https://github.com/apache/flink/pull/13584#issuecomment-706700317 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot edited a comment on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7391) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #13299: [FLINK-19072][table-planner] Import Temporal Table join rule for Stream
leonardBang commented on a change in pull request #13299:
URL: https://github.com/apache/flink/pull/13299#discussion_r502901175
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdUniqueKeys.scala
##
@@ -244,7 +243,25 @@ class FlinkRelMdUniqueKeys private extends
MetadataHandler[BuiltInMetadata.Uniqu
ignoreNulls: Boolean): JSet[ImmutableBitSet] = {
val inputUniqueKeys = mq.getUniqueKeys(rel.getInput, ignoreNulls)
val rankFunColumnIndex =
RankUtil.getRankNumberColumnIndex(rel).getOrElse(-1)
-if (rankFunColumnIndex < 0) {
+//TODO current deduplicate on row time is still a Rank,
+// remove this after support deduplicate on row time
+val canConvertToDeduplicate: Boolean = {
Review comment:
yes, this part will be rework in the commit of supporting deduplicate on
row time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-18383) Translate "JDBC SQL Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211867#comment-17211867 ] limbo commented on FLINK-18383: --- [~zhoupeijie] still working on it? > Translate "JDBC SQL Connector" page into Chinese > > > Key: FLINK-18383 > URL: https://issues.apache.org/jira/browse/FLINK-18383 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: zhoupeijie >Priority: Major > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/jdbc.html > The markdown file is located in flink/docs/dev/table/connectors/jdbc.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] leonardBang commented on a change in pull request #13299: [FLINK-19072][table-planner] Import Temporal Table join rule for Stream
leonardBang commented on a change in pull request #13299:
URL: https://github.com/apache/flink/pull/13299#discussion_r502891596
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamExecTemporalJoin.scala
##
@@ -59,21 +64,21 @@ class StreamExecTemporalJoin(
with StreamPhysicalRel
with StreamExecNode[RowData] {
- override def requireWatermark: Boolean = {
-val nonEquiJoinRex = getJoinInfo.getRemaining(cluster.getRexBuilder)
-
-var rowtimeJoin: Boolean = false
-val visitor = new RexVisitorImpl[Unit](true) {
- override def visitCall(call: RexCall): Unit = {
-if (call.getOperator == TEMPORAL_JOIN_CONDITION) {
- rowtimeJoin = TemporalJoinUtil.isRowtimeCall(call)
-} else {
- call.getOperands.foreach(node => node.accept(this))
-}
+ def rightInputUniqueKeyContainsJoinKey(): Boolean = {
+val right = getInput(1)
+val rightUniqueKeys = getCluster.getMetadataQuery.getUniqueKeys(right)
+if (rightUniqueKeys != null) {
+ val joinKeys = keyPairs.map(_.target).toArray
+ rightUniqueKeys.exists {
+uniqueKey => joinKeys.forall(uniqueKey.toArray.contains(_))
Review comment:
good catch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13584: [hotfix][typo] Fix typo in MiniCluster
flinkbot edited a comment on pull request #13584: URL: https://github.com/apache/flink/pull/13584#issuecomment-706656585 ## CI report: * ab9911fee445c24dba308c6bae19ef51db8abec8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7390) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot edited a comment on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7391) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #13299: [FLINK-19072][table-planner] Import Temporal Table join rule for Stream
leonardBang commented on a change in pull request #13299:
URL: https://github.com/apache/flink/pull/13299#discussion_r502886842
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableRule.scala
##
@@ -74,23 +178,150 @@ abstract class
LogicalCorrelateToJoinFromTemporalTableRule(
val rel = builder.build()
call.transformTo(rel)
}
+}
+
+
+/**
+ * General temporal table join rule to rewrite the original Correlate into a
Join.
+ */
+abstract class LogicalCorrelateToJoinFromGeneralTemporalTableRule(
+operand: RelOptRuleOperand,
+description: String)
+ extends LogicalCorrelateToJoinFromTemporalTableRule(operand, description) {
+
+ protected def extractRightTimeInputRef(
+ leftInput: RelNode,
+ snapshot: LogicalSnapshot): Option[RexNode] = {
+val rightFields = snapshot.getRowType.getFieldList.asScala
+val timeAttributeFields = rightFields.filter(
+ f => f.getType.isInstanceOf[TimeIndicatorRelDataType])
+val rexBuilder = snapshot.getCluster.getRexBuilder
+
+if (timeAttributeFields != null && timeAttributeFields.length == 1) {
+ val leftFieldCnt = leftInput.getRowType.getFieldCount
+ val timeColIndex = leftFieldCnt +
rightFields.indexOf(timeAttributeFields.get(0))
+ val timeColDataType = timeAttributeFields.get(0).getType
+ Some(rexBuilder.makeInputRef(timeColDataType, timeColIndex))
+} else {
+ None
+}
+ }
+
+ protected def extractSnapshotTimeInputRef(
+ leftInput: RelNode,
+ snapshot: LogicalSnapshot): Option[RexInputRef] = {
+val leftRowType = leftInput.getRowType
+val leftFields = leftRowType.getFieldList
+val periodField = snapshot.getPeriod.asInstanceOf[RexFieldAccess].getField
+if (leftFields.contains(periodField)) {
+ val index = leftRowType.getFieldList.indexOf(periodField)
+ Some(RexInputRef.of(index, leftRowType))
+} else {
+ None
+}
+ }
+
+ override def onMatch(call: RelOptRuleCall): Unit = {
+val correlate: LogicalCorrelate = call.rel(0)
+val leftInput: RelNode = call.rel(1)
+val filterCondition = getFilterCondition(call)
+val snapshot = getLogicalSnapshot(call)
+
+val leftRowType = leftInput.getRowType
+val joinCondition = filterCondition.accept(new RexShuttle() {
+ // change correlate variable expression to normal RexInputRef (which is
from left side)
+ override def visitFieldAccess(fieldAccess: RexFieldAccess): RexNode = {
+fieldAccess.getReferenceExpr match {
+ case corVar: RexCorrelVariable =>
+require(correlate.getCorrelationId.equals(corVar.id))
+val index = leftRowType.getFieldList.indexOf(fieldAccess.getField)
+RexInputRef.of(index, leftRowType)
+ case _ => super.visitFieldAccess(fieldAccess)
+}
+ }
+ // update the field index from right side
+ override def visitInputRef(inputRef: RexInputRef): RexNode = {
+val rightIndex = leftRowType.getFieldCount + inputRef.getIndex
+new RexInputRef(rightIndex, inputRef.getType)
+ }
+})
+
+validateSnapshotInCorrelate(snapshot, correlate)
+
+val (leftJoinKey, rightJoinKey) = {
+ val rexBuilder = correlate.getCluster.getRexBuilder
+ val relBuilder = call.builder()
+ relBuilder.push(leftInput)
+ relBuilder.push(snapshot)
+ val rewriteJoin = relBuilder.join(correlate.getJoinType,
joinCondition).build()
+ val joinInfo = rewriteJoin.asInstanceOf[LogicalJoin].analyzeCondition()
+ val leftJoinKey = joinInfo.leftKeys.map(i =>
rexBuilder.makeInputRef(leftInput, i))
+ val rightJoinKey = joinInfo.rightKeys.map(i => {
+val leftFieldCnt = leftInput.getRowType.getFieldCount
+val leftKeyType = snapshot.getRowType.getFieldList.get(i).getType
+rexBuilder.makeInputRef(leftKeyType, leftFieldCnt + i)
+ })
+ (leftJoinKey, rightJoinKey)
+}
+
+val snapshotTimeInputRef = extractSnapshotTimeInputRef(leftInput, snapshot)
+ .getOrElse(throw new ValidationException("Temporal Table Join requires
time attribute in the " +
+s"left table, but no row time attribute found."))
+
+val rexBuilder = correlate.getCluster.getRexBuilder
+val temporalCondition = if(isRowTimeTemporalTableJoin(snapshot)) {
+ val rightTimeInputRef = extractRightTimeInputRef(leftInput, snapshot)
+ if (rightTimeInputRef.isEmpty ||
!isRowtimeIndicatorType(rightTimeInputRef.get.getType)) {
+ throw new ValidationException("Event-Time Temporal Table Join
requires both" +
+s" primary key and row time attribute in versioned table," +
Review comment:
The `primary key` will checked in
`TemporalJoinRewriteWithUniqueKeyRule.`
This is an automated message from the Apache Git Service.
To respond to the
[GitHub] [flink] flinkbot commented on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot commented on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706672042 ## CI report: * 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19455) Module 'flink-sql-connector-hive-2.3.6', 'flink-docs' and 'flnk-dist' build fail by maven-enforcer-plugin
[
https://issues.apache.org/jira/browse/FLINK-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hailong wang updated FLINK-19455:
-
Summary: Module 'flink-sql-connector-hive-2.3.6', 'flink-docs' and
'flnk-dist' build fail by maven-enforcer-plugin (was: Module
'flink-sql-connector-hive-2.3.6', 'flink-docs' and 'flnk-dist'build fail by
maven-enforcer-plugin)
> Module 'flink-sql-connector-hive-2.3.6', 'flink-docs' and 'flnk-dist' build
> fail by maven-enforcer-plugin
> -
>
> Key: FLINK-19455
> URL: https://issues.apache.org/jira/browse/FLINK-19455
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.12.0
>Reporter: hailong wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> I run command 'mvn clean package' in flink-sql-connector-hive-2.3.6, and it
> failed,
> {code:java}
> Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with
> message:
> Found Banned Dependency: org.apache.kafka:kafka_2.10:jar:0.10.2.0
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> For root pom has a rule as follow:
> {code:java}
> bannedDependencies>
>
> *:*_2.12
> *:*_2.10
>
>
> {code}
> As for kafka_2.10 dependency is useless, So we can exclude it as follow:
> {code:java}
>
>org.apache.hive
>hive-exec
>2.3.6
>
>
> log4j
> log4j
>
>
> org.slf4j
> slf4j-log4j12
>
>
> org.pentaho
> pentaho-aggdesigner-algorithm
>
>
> org.apache.kafka
> kafka_2.10
>
>
> {code}
> It also failed in flink-docs and flink-disk module:
> {code:java}
> Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with
> message:
> Found Banned Dependency: org.yaml:snakeyaml:jar:1.24
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
flinkbot commented on pull request #13585: URL: https://github.com/apache/flink/pull/13585#issuecomment-706670347 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8914045aa237dd5cd2f9ccbf6382bb3a1e3ba279 (Sun Oct 11 08:33:03 UTC 2020) **Warnings:** * **2 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-19455).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19455) Module 'flink-sql-connector-hive-2.3.6', 'flink-docs' and 'flnk-dist'build fail by maven-enforcer-plugin
[
https://issues.apache.org/jira/browse/FLINK-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19455:
---
Labels: pull-request-available (was: )
> Module 'flink-sql-connector-hive-2.3.6', 'flink-docs' and 'flnk-dist'build
> fail by maven-enforcer-plugin
>
>
> Key: FLINK-19455
> URL: https://issues.apache.org/jira/browse/FLINK-19455
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.12.0
>Reporter: hailong wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> I run command 'mvn clean package' in flink-sql-connector-hive-2.3.6, and it
> failed,
> {code:java}
> Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with
> message:
> Found Banned Dependency: org.apache.kafka:kafka_2.10:jar:0.10.2.0
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> For root pom has a rule as follow:
> {code:java}
> bannedDependencies>
>
> *:*_2.12
> *:*_2.10
>
>
> {code}
> As for kafka_2.10 dependency is useless, So we can exclude it as follow:
> {code:java}
>
>org.apache.hive
>hive-exec
>2.3.6
>
>
> log4j
> log4j
>
>
> org.slf4j
> slf4j-log4j12
>
>
> org.pentaho
> pentaho-aggdesigner-algorithm
>
>
> org.apache.kafka
> kafka_2.10
>
>
> {code}
> It also failed in flink-docs and flink-disk module:
> {code:java}
> Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with
> message:
> Found Banned Dependency: org.yaml:snakeyaml:jar:1.24
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangxlong opened a new pull request #13585: [FLINK-19455][build] Fix banned dependencies failed when building locally
wangxlong opened a new pull request #13585: URL: https://github.com/apache/flink/pull/13585 ## What is the purpose of the change Fix banned dependencies failed when building locally [FLINK-19266](https://issues.apache.org/jira/projects/FLINK/issues/FLINK-19266) ## Brief change log 1. For the banned dependencies 'org.apache.kafka:kafka_2.10:jar:0.10.2.0', just excluding from 'flink-sql-connector-hive-2.3.6' module. 2. For the banned dependencies 'org.yaml:snakeyaml:jar:1.24'. This occur when other modules depend on 'flink-kubernetes' module. For 'maven-enforcer-plugin' is unable to recognize the discrepancy since the transitive dependency's version was altered by 'dependencyManagement'. So just excluding 'org.yaml:snakeyaml:jar:1.24' from 'io.fabric8:kubernetes-client' and depend on the version of 1.27. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no / don't know) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong commented on pull request #13528: [FLINK-19477][docs-zh][python] Translate page "python_table_api_connectors" into Chinese
wangxlong commented on pull request #13528: URL: https://github.com/apache/flink/pull/13528#issuecomment-706665847 Hi @dianfu CC in your free time, Thank you~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong commented on pull request #13537: [FLINK-19478][docs-zh][python] Translate page "intro_to_table_api" into Chinese
wangxlong commented on pull request #13537: URL: https://github.com/apache/flink/pull/13537#issuecomment-706665786 Hi @dianfu Could you help to have a review in your free time, Thank you~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19423) Primary key position cause JDBC SQL upsert sink ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211841#comment-17211841
]
limbo commented on FLINK-19423:
---
I'm trying to fix the code, and do some test, but find that the table API not
support DELETE , is there any function to test this case
> Primary key position cause JDBC SQL upsert sink ArrayIndexOutOfBoundsException
> --
>
> Key: FLINK-19423
> URL: https://issues.apache.org/jira/browse/FLINK-19423
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC, Table SQL / Ecosystem
>Affects Versions: 1.11.1
>Reporter: limbo
>Priority: Major
>
> We found that the primary key position can cause
> ArrayIndexOutOfBoundsException
> the sink like that( the primary key select the position of 1, 3):
> {code:java}
> CREATE TABLE `test`(
> col1 STRING,
> col2 STRING,
> col3 STRING,
> PRIMARY KEY (col1, col3) NOT ENFORCED ) WITH (
> 'connector' = 'jdbc',
> ...
> ){code}
> when the DELETE (cdc message) come , it will raise
> ArrayIndexOutOfBoundsException:
> {code:java}
> Caused by: java.lang.RuntimeException: Writing records to JDBC failed....
> 10 moreCaused by: java.lang.ArrayIndexOutOfBoundsException: 2at
> org.apache.flink.table.data.GenericRowData.getString(GenericRowData.java:169)
> at
> org.apache.flink.table.data.RowData.lambda$createFieldGetter$245ca7d1$1(RowData.java:310)
> at
> org.apache.flink.connector.jdbc.table.JdbcDynamicOutputFormatBuilder.getPrimaryKey(JdbcDynamicOutputFormatBuilder.java:216)
> at
> org.apache.flink.connector.jdbc.table.JdbcDynamicOutputFormatBuilder.lambda$createRowKeyExtractor$7(JdbcDynamicOutputFormatBuilder.java:193)
> at
> org.apache.flink.connector.jdbc.table.JdbcDynamicOutputFormatBuilder.lambda$createKeyedRowExecutor$3fd497bb$1(JdbcDynamicOutputFormatBuilder.java:128)
> at
> org.apache.flink.connector.jdbc.internal.executor.KeyedBatchStatementExecutor.executeBatch(KeyedBatchStatementExecutor.java:71)
> at
> org.apache.flink.connector.jdbc.internal.executor.BufferReduceStatementExecutor.executeBatch(BufferReduceStatementExecutor.java:99)
> at
> org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.attemptFlush(JdbcBatchingOutputFormat.java:200)
> at
> org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.flush(JdbcBatchingOutputFormat.java:171)
> ... 8 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13584: [hotfix][typo] Fix typo in MiniCluster
flinkbot edited a comment on pull request #13584: URL: https://github.com/apache/flink/pull/13584#issuecomment-706656585 ## CI report: * ab9911fee445c24dba308c6bae19ef51db8abec8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7390) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13584: [hotfix][typo] Fix typo in MiniCluster
flinkbot commented on pull request #13584: URL: https://github.com/apache/flink/pull/13584#issuecomment-706656585 ## CI report: * ab9911fee445c24dba308c6bae19ef51db8abec8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
