godfreyhe commented on code in PR #20324:
URL: https://github.com/apache/flink/pull/20324#discussion_r939866279
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -541,16 +632,9 @@ private ProcessFunction<RowData, RowData>
createSyncLookupJoinFunction(
isObjectReuseEnabled);
Optional<RelDataType> temporalTableOutputType =
- projectionOnTemporalTable != null
- ? Optional.of(
- RexUtil.createStructType(
- unwrapTypeFactory(relBuilder),
projectionOnTemporalTable))
- : Optional.empty();
- RowType rightRowType =
- projectionOnTemporalTable != null
- ? (RowType)
toLogicalType(temporalTableOutputType.get())
- : tableSourceRowType;
- GeneratedCollector<TableFunctionCollector<RowData>> generatedCollector
=
+ getProjectionRowTypeOnTemporalTable(relBuilder);
Review Comment:
we can call the `getProjectionRowTypeOnTemporalTable` method in `if
(projectionOnTemporalTable != null) ` branch, and the implementation of
`getProjectionRowTypeOnTemporalTable` method can be simplified
There is some duplicated code in `createAsyncLookupJoin` method
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/join/LookupJoinHarnessTest.java:
##########
@@ -238,13 +239,16 @@ public void flatMap(RowData value, Collector<RowData>
out) throws Exception {
* The {@link TestingFetcherCollector} is a simple implementation of {@link
* TableFunctionCollector} which combines left and right into a
JoinedRowData.
*/
- public static final class TestingFetcherCollector extends
TableFunctionCollector {
+ public static final class TestingFetcherCollector extends
ListenableCollector<RowData> {
private static final long serialVersionUID = -312754413938303160L;
@Override
- public void collect(Object record) {
+ public void collect(RowData record) {
RowData left = (RowData) getInput();
- RowData right = (RowData) record;
+ RowData right = record;
+ getCollectListener()
+ .ifPresent(listener -> ((CollectListener)
listener).onCollect(record));
Review Comment:
cast is no needed here
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -143,6 +154,8 @@ public abstract class CommonExecLookupJoin extends
ExecNodeBase<RowData>
public static final String LOOKUP_JOIN_TRANSFORMATION = "lookup-join";
+ public static final String LOOKUP_JOIN_WITH_STATE_TRANSFORMATION =
"lookup-join-with-state";
Review Comment:
how about `lookup-join-with-materialize` ?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/collector/ListenableCollector.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.collector;
+
+import org.apache.flink.annotation.Internal;
+
+import javax.annotation.Nullable;
+
+import java.util.Optional;
+
+/**
+ * A listenable collector for lookup join that can be called when an original
record was collected.
+ */
+@Internal
+public abstract class ListenableCollector<T> extends TableFunctionCollector<T>
{
+ @Nullable private CollectListener<T> collectListener;
+
+ public void setCollectListener(CollectListener<T> collectListener) {
Review Comment:
nit: @Nullable CollectListener<T> collectListener
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -329,8 +342,66 @@ private Transformation<RowData>
createSyncLookupJoinWithState(
isLeftOuterJoin,
isObjectReuseEnabled);
- // TODO then wrapper it into a keyed lookup function with state
FLINK-28568
- throw new UnsupportedOperationException("to be supported");
+ RowType rightRowType =
+ getRightRowType(
+ getProjectionRowTypeOnTemporalTable(relBuilder),
tableSourceRowType);
+
+ KeyedLookupJoinWrapper keyedLookupJoinWrapper =
+ new KeyedLookupJoinWrapper(
+ (LookupJoinRunner) processFunction,
+ StateConfigUtil.createTtlConfig(
+
config.get(ExecutionConfigOptions.IDLE_STATE_RETENTION).toMillis()),
+ InternalSerializers.create(rightRowType),
+ lookupKeyContainsPrimaryKey);
+
+ KeyedProcessOperator<RowData, RowData, RowData> operator =
+ new KeyedProcessOperator<>(keyedLookupJoinWrapper);
+
+ List<Integer> refKeys =
+ allLookupKeys.values().stream()
+ .filter(key -> key instanceof
LookupJoinUtil.FieldRefLookupKey)
+ .map(key -> ((LookupJoinUtil.FieldRefLookupKey)
key).index)
+ .collect(Collectors.toList());
+ RowDataKeySelector keySelector;
+
+ // use single parallelism for empty key shuffle
+ boolean singleParallelism = refKeys.isEmpty();
+ if (singleParallelism) {
+ // all lookup keys are constants, then use an empty key selector
+ keySelector = EmptyRowDataKeySelector.INSTANCE;
+ } else {
+ // make it a deterministic asc order
+ Collections.sort(refKeys);
+ keySelector =
+ KeySelectorUtil.getRowDataSelector(
+ classLoader,
+
refKeys.stream().mapToInt(Integer::intValue).toArray(),
+ InternalTypeInfo.of(inputRowType));
+ }
+ final KeyGroupStreamPartitioner<RowData, RowData> partitioner =
+ new KeyGroupStreamPartitioner<>(
+ keySelector,
KeyGroupRangeAssignment.DEFAULT_LOWER_BOUND_MAX_PARALLELISM);
+ Transformation<RowData> partitionedTransform =
+ new PartitionTransformation<>(inputTransformation,
partitioner);
+ if (singleParallelism) {
+ setSingletonParallelism(partitionedTransform);
+ } else {
+
partitionedTransform.setParallelism(inputTransformation.getParallelism());
+ }
+
+ OneInputTransformation<RowData, RowData> transform =
+ ExecNodeUtil.createOneInputTransformation(
+ partitionedTransform,
+
createTransformationMeta(LOOKUP_JOIN_WITH_STATE_TRANSFORMATION, config),
+ operator,
+ InternalTypeInfo.of(resultRowType),
+ partitionedTransform.getParallelism());
+ transform.setStateKeySelector(keySelector);
+ transform.setStateKeyType(keySelector.getProducedType());
+ if (singleParallelism) {
+ setSingletonParallelism(transform);
+ }
+ return transform;
}
private LogicalType getLookupKeyLogicalType(
Review Comment:
this method can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
