Copilot commented on code in PR #18165: URL: https://github.com/apache/pinot/pull/18165#discussion_r3067239643
########## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/pipeline/PartitionFunctionExprCompiler.java: ########## @@ -0,0 +1,903 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.segment.spi.partition.pipeline; + +import com.google.common.base.Preconditions; +import java.lang.reflect.Array; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; +import java.util.ArrayList; +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; +import org.apache.pinot.segment.spi.function.scalar.PartitionInternalScalarFunctions; +import org.apache.pinot.spi.annotations.ScalarFunction; +import org.apache.pinot.spi.config.table.ColumnPartitionConfig; +import org.apache.pinot.spi.utils.PinotReflectionUtils; + +import static java.nio.charset.StandardCharsets.UTF_8; + + +/** + * Compiles a restricted partition-function expression into a typed {@link PartitionPipeline} backed by deterministic + * scalar functions. + */ +public final class PartitionFunctionExprCompiler { + private static final int MAX_EXPRESSION_LENGTH = 256; + private static final int MAX_DEPTH = 8; + private static final int MAX_ARGUMENTS = 32; + private static final String FUNCTION_REGEX = ".*\\.function\\..*"; + private static final Object DYNAMIC_ARGUMENT = new Object(); + + private static final Map<String, List<ScalarFunctionMethod>> FUNCTION_METHODS = loadFunctions(); + + private PartitionFunctionExprCompiler() { + } + + public static PartitionPipeline compile(String rawColumn, String functionExpr) { + return compile(rawColumn, functionExpr, null); + } Review Comment: The overload compile(rawColumn, functionExpr) always passes a null PartitionIntNormalizer. For expressions that produce INT/LONG, this will fail later when constructing PartitionPipeline (integral-output pipelines require a normalizer), which is surprising given expression-mode defaults to POSITIVE_MODULO elsewhere. Consider either defaulting to POSITIVE_MODULO here as well, or making this overload reject integral outputs with a clearer error (or removing the overload to force callers to choose a normalizer explicitly). ########## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/LegacyExpressionPartitionFunction.java: ########## @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.segment.spi.partition; + +import com.fasterxml.jackson.annotation.JsonIgnore; +import org.apache.pinot.segment.spi.partition.pipeline.PartitionFunctionExprCompiler; +import org.apache.pinot.segment.spi.partition.pipeline.PartitionPipelineFunction; + + +/** + * Shared helper for legacy partition functions that preserve their public identity while delegating partition + * computation to the expression compiler and pipeline runtime. + */ +abstract class LegacyExpressionPartitionFunction implements PartitionFunction { + private static final String RAW_VALUE_IDENTIFIER = "__partition_value"; + + private final int _numPartitions; + private transient volatile PartitionPipelineFunction _delegate; + + LegacyExpressionPartitionFunction(int numPartitions) { + _numPartitions = numPartitions; + } + + protected final int computePartition(String value) { + return getDelegate().getPartition(value); + } Review Comment: Legacy partition functions now delegate per-call partition computation through PartitionFunctionExprCompiler/PartitionPipelineFunction, which relies on reflection (Method.invoke) and allocates PartitionValue/step objects. Partition functions are used in hot paths (e.g., MutableSegmentImpl and AbstractColumnStatisticsCollector call getPartition per row), so this indirection can cause a significant ingestion/query performance regression and adds a costly reflection scan on first use. Consider keeping legacy partition functions’ direct implementations for getPartition(), and reserve the expression compiler/runtime for functionExpr mode; or switch the pipeline runtime to pre-bound MethodHandles/lambdas with minimal allocations for legacy delegates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
