[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces URL: https://github.com/apache/flink/pull/8476#discussion_r288433577 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamInputProcessor.java ## @@ -93,12 +77,6 @@ /** Number of input channels the valve needs to handle. */ private final int numInputChannels; Review comment: `numInputChannels` can be a local variable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r287910131
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/Input.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.io.AsyncDataInput;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+
+import java.io.Closeable;
+
+/**
+ * Basic interface for inputs of stream operators.
+ */
+@Internal
+public interface Input extends AsyncDataInput, Closeable {
Review comment:
> @sunhaibotb, are you fine with renaming those renames? (and fyi, since
renaming will affect a wip `StreamSelectableTwoInputStreamProcessor` PR.
It's fine to me. @pnowojski @StefanRRichter
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r286742814
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamInputProcessor.java
##
@@ -156,9 +129,41 @@ public StreamInputProcessor(
}
public boolean processInput() throws Exception {
- if (isFinished) {
- return false;
+ initializeNumRecordsIn();
+
+ StreamElement recordOrMark = input.pollNext();
+ if (recordOrMark == null) {
+ input.isAvailable().get();
+ return input.isFinished();
}
+
+ processElement(recordOrMark);
+ return true;
+ }
+
+ private void processElement(StreamElement recordOrMark) throws
Exception {
+ if (recordOrMark.isWatermark()) {
+ // handle watermark
+
statusWatermarkValve.inputWatermark(recordOrMark.asWatermark(), currentChannel);
Review comment:
In fact, I care about that the input channel index is exposed to input
processors. As long as it's not exposed to the input processor (for example,
encapsulating a class as you said), it's fine to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r286741019
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/NetworkInput.java
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer.DeserializationResult;
+import
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent;
+import org.apache.flink.runtime.io.network.partition.consumer.InputGate;
+import org.apache.flink.runtime.plugable.DeserializationDelegate;
+import org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Implementation of taking {@link InputGate} as {@link Input}.
+ */
+@Internal
+public final class NetworkInput implements Input {
+
+ private final int numberOfInputChannels;
+
+ private final CheckpointBarrierHandler barrierHandler;
+
+ private final DeserializationDelegate
deserializationDelegate;
+
+ private final
RecordDeserializer>[]
recordDeserializers;
+
+ /**
+* The channel from which a buffer came, tracked so that we can
appropriately map
+* the watermarks and watermark statuses to the correct channel index
of the correct valve.
+*/
+ private int currentChannel = -1;
+
+ private RecordDeserializer>
currentRecordDeserializer = null;
+
+ private boolean isFinished = false;
+
+ @SuppressWarnings("unchecked")
+ public NetworkInput(
+ CheckpointBarrierHandler barrierHandler,
+ TypeSerializer inputSerializer,
+ IOManager ioManager) {
+ this.barrierHandler = barrierHandler;
+ this.numberOfInputChannels =
barrierHandler.getNumberOfInputChannels();
+ this.deserializationDelegate = new
NonReusingDeserializationDelegate<>(
+ new StreamElementSerializer<>(inputSerializer));
+
+ // Initialize one deserializer per input channel
+ this.recordDeserializers = new
SpillingAdaptiveSpanningRecordDeserializer[numberOfInputChannels];
+ for (int i = 0; i < recordDeserializers.length; i++) {
+ recordDeserializers[i] = new
SpillingAdaptiveSpanningRecordDeserializer<>(
+ ioManager.getSpillingDirectoriesPaths());
+ }
+ }
+
+ @Override
+ public StreamElement pollNext() throws Exception {
+
+ while (true) {
+ // get the stream element from the deserializer
+ if (currentRecordDeserializer != null) {
+ DeserializationResult result =
currentRecordDeserializer.getNextRecord(deserializationDelegate);
+ if (result.isBufferConsumed()) {
+
currentRecordDeserializer.getCurrentBuffer().recycleBuffer();
+ currentRecordDeserializer = null;
+ }
+
+ if (result.isFullRecord()) {
+ return
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r286740477
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/Input.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.io.AsyncDataInput;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+
+import java.io.Closeable;
+
+/**
+ * Basic interface for inputs of stream operators.
+ */
+@Internal
+public interface Input extends AsyncDataInput, Closeable {
Review comment:
> Generally speaking Closeable is better and preferred - it extends
Autocloseable and provide stronger/easier to use contract of close() being
idempotent.
You are right.
> Is there a reason why Input#close() can not be idempotent?
No.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r286740477
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/Input.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.io.AsyncDataInput;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+
+import java.io.Closeable;
+
+/**
+ * Basic interface for inputs of stream operators.
+ */
+@Internal
+public interface Input extends AsyncDataInput, Closeable {
Review comment:
No. You are right.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285543831
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamInputProcessor.java
##
@@ -156,9 +129,41 @@ public StreamInputProcessor(
}
public boolean processInput() throws Exception {
- if (isFinished) {
- return false;
+ initializeNumRecordsIn();
+
+ StreamElement recordOrMark = input.pollNext();
+ if (recordOrMark == null) {
+ input.isAvailable().get();
+ return input.isFinished();
Review comment:
Should here be`return !input.isFinished()` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285542065
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/Input.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.io.AsyncDataInput;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+
+import java.io.Closeable;
+
+/**
+ * Basic interface for inputs of stream operators.
+ */
+@Internal
+public interface Input extends AsyncDataInput, Closeable {
+}
Review comment:
Similar to `InputChannel#getChannelIndex()`, adding `int getInputIndex()` to
`Input` can simplify some processing in `StreamTwoInputSelectableProcessor`. So
I suggest to add it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285522876
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/NetworkInput.java
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer.DeserializationResult;
+import
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent;
+import org.apache.flink.runtime.io.network.partition.consumer.InputGate;
+import org.apache.flink.runtime.plugable.DeserializationDelegate;
+import org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Implementation of taking {@link InputGate} as {@link Input}.
+ */
+@Internal
+public final class NetworkInput implements Input {
+
+ private final int numberOfInputChannels;
+
+ private final CheckpointBarrierHandler barrierHandler;
+
+ private final DeserializationDelegate
deserializationDelegate;
+
+ private final
RecordDeserializer>[]
recordDeserializers;
+
+ /**
+* The channel from which a buffer came, tracked so that we can
appropriately map
+* the watermarks and watermark statuses to the correct channel index
of the correct valve.
+*/
+ private int currentChannel = -1;
+
+ private RecordDeserializer>
currentRecordDeserializer = null;
+
+ private boolean isFinished = false;
+
+ @SuppressWarnings("unchecked")
+ public NetworkInput(
+ CheckpointBarrierHandler barrierHandler,
+ TypeSerializer inputSerializer,
+ IOManager ioManager) {
+ this.barrierHandler = barrierHandler;
+ this.numberOfInputChannels =
barrierHandler.getNumberOfInputChannels();
+ this.deserializationDelegate = new
NonReusingDeserializationDelegate<>(
+ new StreamElementSerializer<>(inputSerializer));
+
+ // Initialize one deserializer per input channel
+ this.recordDeserializers = new
SpillingAdaptiveSpanningRecordDeserializer[numberOfInputChannels];
+ for (int i = 0; i < recordDeserializers.length; i++) {
+ recordDeserializers[i] = new
SpillingAdaptiveSpanningRecordDeserializer<>(
+ ioManager.getSpillingDirectoriesPaths());
+ }
+ }
+
+ @Override
+ public StreamElement pollNext() throws Exception {
+
+ while (true) {
+ // get the stream element from the deserializer
+ if (currentRecordDeserializer != null) {
+ DeserializationResult result =
currentRecordDeserializer.getNextRecord(deserializationDelegate);
+ if (result.isBufferConsumed()) {
+
currentRecordDeserializer.getCurrentBuffer().recycleBuffer();
+ currentRecordDeserializer = null;
+ }
+
+ if (result.isFullRecord()) {
+ return
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285508328
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamInputProcessor.java
##
@@ -156,9 +129,41 @@ public StreamInputProcessor(
}
public boolean processInput() throws Exception {
- if (isFinished) {
- return false;
+ initializeNumRecordsIn();
+
+ StreamElement recordOrMark = input.pollNext();
+ if (recordOrMark == null) {
+ input.isAvailable().get();
+ return input.isFinished();
}
+
+ processElement(recordOrMark);
+ return true;
+ }
+
+ private void processElement(StreamElement recordOrMark) throws
Exception {
+ if (recordOrMark.isWatermark()) {
+ // handle watermark
+
statusWatermarkValve.inputWatermark(recordOrMark.asWatermark(), currentChannel);
Review comment:
1. It seems missing some logic about assigning value to `currentChannel` ,
and currently it is always - 1.
2. I think that the input channel index should be hidden in `NetworkInput`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285522876
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/NetworkInput.java
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer.DeserializationResult;
+import
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent;
+import org.apache.flink.runtime.io.network.partition.consumer.InputGate;
+import org.apache.flink.runtime.plugable.DeserializationDelegate;
+import org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Implementation of taking {@link InputGate} as {@link Input}.
+ */
+@Internal
+public final class NetworkInput implements Input {
+
+ private final int numberOfInputChannels;
+
+ private final CheckpointBarrierHandler barrierHandler;
+
+ private final DeserializationDelegate
deserializationDelegate;
+
+ private final
RecordDeserializer>[]
recordDeserializers;
+
+ /**
+* The channel from which a buffer came, tracked so that we can
appropriately map
+* the watermarks and watermark statuses to the correct channel index
of the correct valve.
+*/
+ private int currentChannel = -1;
+
+ private RecordDeserializer>
currentRecordDeserializer = null;
+
+ private boolean isFinished = false;
+
+ @SuppressWarnings("unchecked")
+ public NetworkInput(
+ CheckpointBarrierHandler barrierHandler,
+ TypeSerializer inputSerializer,
+ IOManager ioManager) {
+ this.barrierHandler = barrierHandler;
+ this.numberOfInputChannels =
barrierHandler.getNumberOfInputChannels();
+ this.deserializationDelegate = new
NonReusingDeserializationDelegate<>(
+ new StreamElementSerializer<>(inputSerializer));
+
+ // Initialize one deserializer per input channel
+ this.recordDeserializers = new
SpillingAdaptiveSpanningRecordDeserializer[numberOfInputChannels];
+ for (int i = 0; i < recordDeserializers.length; i++) {
+ recordDeserializers[i] = new
SpillingAdaptiveSpanningRecordDeserializer<>(
+ ioManager.getSpillingDirectoriesPaths());
+ }
+ }
+
+ @Override
+ public StreamElement pollNext() throws Exception {
+
+ while (true) {
+ // get the stream element from the deserializer
+ if (currentRecordDeserializer != null) {
+ DeserializationResult result =
currentRecordDeserializer.getNextRecord(deserializationDelegate);
+ if (result.isBufferConsumed()) {
+
currentRecordDeserializer.getCurrentBuffer().recycleBuffer();
+ currentRecordDeserializer = null;
+ }
+
+ if (result.isFullRecord()) {
+ return
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285522876
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/NetworkInput.java
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer;
+import
org.apache.flink.runtime.io.network.api.serialization.RecordDeserializer.DeserializationResult;
+import
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent;
+import org.apache.flink.runtime.io.network.partition.consumer.InputGate;
+import org.apache.flink.runtime.plugable.DeserializationDelegate;
+import org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Implementation of taking {@link InputGate} as {@link Input}.
+ */
+@Internal
+public final class NetworkInput implements Input {
+
+ private final int numberOfInputChannels;
+
+ private final CheckpointBarrierHandler barrierHandler;
+
+ private final DeserializationDelegate
deserializationDelegate;
+
+ private final
RecordDeserializer>[]
recordDeserializers;
+
+ /**
+* The channel from which a buffer came, tracked so that we can
appropriately map
+* the watermarks and watermark statuses to the correct channel index
of the correct valve.
+*/
+ private int currentChannel = -1;
+
+ private RecordDeserializer>
currentRecordDeserializer = null;
+
+ private boolean isFinished = false;
+
+ @SuppressWarnings("unchecked")
+ public NetworkInput(
+ CheckpointBarrierHandler barrierHandler,
+ TypeSerializer inputSerializer,
+ IOManager ioManager) {
+ this.barrierHandler = barrierHandler;
+ this.numberOfInputChannels =
barrierHandler.getNumberOfInputChannels();
+ this.deserializationDelegate = new
NonReusingDeserializationDelegate<>(
+ new StreamElementSerializer<>(inputSerializer));
+
+ // Initialize one deserializer per input channel
+ this.recordDeserializers = new
SpillingAdaptiveSpanningRecordDeserializer[numberOfInputChannels];
+ for (int i = 0; i < recordDeserializers.length; i++) {
+ recordDeserializers[i] = new
SpillingAdaptiveSpanningRecordDeserializer<>(
+ ioManager.getSpillingDirectoriesPaths());
+ }
+ }
+
+ @Override
+ public StreamElement pollNext() throws Exception {
+
+ while (true) {
+ // get the stream element from the deserializer
+ if (currentRecordDeserializer != null) {
+ DeserializationResult result =
currentRecordDeserializer.getNextRecord(deserializationDelegate);
+ if (result.isBufferConsumed()) {
+
currentRecordDeserializer.getCurrentBuffer().recycleBuffer();
+ currentRecordDeserializer = null;
+ }
+
+ if (result.isFullRecord()) {
+ return
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285509141
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamInputProcessor.java
##
@@ -156,9 +129,41 @@ public StreamInputProcessor(
}
public boolean processInput() throws Exception {
- if (isFinished) {
- return false;
+ initializeNumRecordsIn();
+
+ StreamElement recordOrMark = input.pollNext();
+ if (recordOrMark == null) {
+ input.isAvailable().get();
+ return input.isFinished();
}
+
+ processElement(recordOrMark);
+ return true;
+ }
+
+ private void processElement(StreamElement recordOrMark) throws
Exception {
+ if (recordOrMark.isWatermark()) {
+ // handle watermark
+
statusWatermarkValve.inputWatermark(recordOrMark.asWatermark(), currentChannel);
+ } else if (recordOrMark.isStreamStatus()) {
+ // handle stream status
+
statusWatermarkValve.inputStreamStatus(recordOrMark.asStreamStatus(),
currentChannel);
Review comment:
Same as the previous comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285508328
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamInputProcessor.java
##
@@ -156,9 +129,41 @@ public StreamInputProcessor(
}
public boolean processInput() throws Exception {
- if (isFinished) {
- return false;
+ initializeNumRecordsIn();
+
+ StreamElement recordOrMark = input.pollNext();
+ if (recordOrMark == null) {
+ input.isAvailable().get();
+ return input.isFinished();
}
+
+ processElement(recordOrMark);
+ return true;
+ }
+
+ private void processElement(StreamElement recordOrMark) throws
Exception {
+ if (recordOrMark.isWatermark()) {
+ // handle watermark
+
statusWatermarkValve.inputWatermark(recordOrMark.asWatermark(), currentChannel);
Review comment:
1. It seems missing some logic about assigning value to `currentChannel` ,
and currently it is always - 1.
2. I think that the channel index should be hidden in `NetworkInput`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285497650
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/Input.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.io.AsyncDataInput;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+
+import java.io.Closeable;
+
+/**
+ * Basic interface for inputs of stream operators.
+ */
+@Internal
+public interface Input extends AsyncDataInput, Closeable {
Review comment:
Can we change it to implement `AutoCloseable` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285484419
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGate.java
##
@@ -68,7 +69,7 @@
* will have an input gate attached to it. This will provide its input, which
will consist of one
* subpartition from each partition of the intermediate result.
*/
-public abstract class InputGate implements AutoCloseable {
+public abstract class InputGate implements
AsyncDataInput>, AutoCloseable {
public static final CompletableFuture AVAILABLE =
CompletableFuture.completedFuture(null);
Review comment:
The `InputGate#AVAILABLE` should be removed because the new
`AsyncDataInput` interface already has `AVAILABLE`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network] Introduce Input and NetworkInput interfaces
sunhaibotb commented on a change in pull request #8476: [FLINK-12490][network]
Introduce Input and NetworkInput interfaces
URL: https://github.com/apache/flink/pull/8476#discussion_r285484419
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGate.java
##
@@ -68,7 +69,7 @@
* will have an input gate attached to it. This will provide its input, which
will consist of one
* subpartition from each partition of the intermediate result.
*/
-public abstract class InputGate implements AutoCloseable {
+public abstract class InputGate implements
AsyncDataInput>, AutoCloseable {
public static final CompletableFuture AVAILABLE =
CompletableFuture.completedFuture(null);
Review comment:
The `InputGate#AVAILABLE` should be removed because the new
`AsyncDataInput` interface already has `AVAILABLE`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
