yashmayya commented on code in PR #19990:
URL: https://github.com/apache/pinot/pull/19990#discussion_r4381238078
##########
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AnyValueAggregationFunction.java:
##########
@@ -200,7 +249,7 @@ private void aggregateHelper(int length, BlockValSet bvs,
ValueProcessor<Object>
// Fall back to direct value access based on type
forEachNotNull(length, bvs, (from, to) -> {
for (int i = from; i < to; i++) {
- Object value = getDirectValue(bvs, i);
+ Object value = getDirectValue(bvs, i, storedType);
Review Comment:
Perf: `getDirectValue` calls `bvs.getXxxValuesSV()` once per row, and the
group-by path never stops early. `TransformBlockValSet`, `RowBasedBlockValSet`
and `DataBlockValSet` build the full array on each call, so one block costs
O(n²).
This is older than this PR, but the typed `BIG_DECIMAL` read makes it much
worse. One `RowBasedBlockValSet` block, 10 groups, local test:
| Rows | Untyped | `'DOUBLE'` | `'BIG_DECIMAL'` |
|---|---|---|---|
| 5k | 35 ms | 41 ms | 1,514 ms |
| 10k | 129 ms | 107 ms | 5,377 ms |
Suggestion: read the array once per block in `aggregateHelper`, and check
`holder.getResult(g) == null` before you read or box the value. A follow-up PR
is fine for this one.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/InferAnyValueTypeProvider.java:
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.spi.config.provider.PinotClusterConfigChangeListener;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants.Broker;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Resolves the default of the `inferAnyValueType` query option from
+/// [Broker#CONFIG_OF_BROKER_QUERY_INFER_ANY_VALUE_TYPE] in the broker conf
and in the Helix cluster config. The cluster
+/// config wins where it sets the key, and it is applied live: registered as a
[PinotClusterConfigChangeListener], a
+/// change takes effect on the next query without a broker restart.
+///
+/// Thread-safe.
+public class InferAnyValueTypeProvider implements
PinotClusterConfigChangeListener {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(InferAnyValueTypeProvider.class);
+
+ private final boolean _brokerConfEnabled;
+ private volatile boolean _enabled;
+
+ public InferAnyValueTypeProvider(PinotConfiguration config) {
+ _brokerConfEnabled =
config.getProperty(Broker.CONFIG_OF_BROKER_QUERY_INFER_ANY_VALUE_TYPE,
+ Broker.DEFAULT_BROKER_QUERY_INFER_ANY_VALUE_TYPE);
+ _enabled = _brokerConfEnabled;
+ }
+
+ @Override
+ public void onChange(Set<String> changedConfigs, Map<String, String>
clusterConfigs) {
+ if
(!changedConfigs.contains(Broker.CONFIG_OF_BROKER_QUERY_INFER_ANY_VALUE_TYPE)) {
+ return;
+ }
+ // A removed or invalid value falls back to the broker conf, so that a
broker that restarts resolves the same.
Review Comment:
This fallback does not work if the cluster config has the key when the
broker starts. `ServiceStartableUtils.applyClusterConfig` copies cluster keys
into the broker conf first, so `_brokerConfEnabled` is `true`. Then removing
the key keeps the rewrite on until a restart, and after a restart it is off.
Removing the key is the natural way to turn this off when old servers reject
the 2-arg form. Setting it to `false` does work.
Suggestion: pin the key in `BaseBrokerStarter.init()` before
`applyClusterConfig`, like #20038 does for fingerprinting:
```java
_brokerConf.setProperty(Broker.CONFIG_OF_BROKER_QUERY_INFER_ANY_VALUE_TYPE,
_brokerConf.getProperty(Broker.CONFIG_OF_BROKER_QUERY_INFER_ANY_VALUE_TYPE,
Broker.DEFAULT_BROKER_QUERY_INFER_ANY_VALUE_TYPE));
```
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateTypeArgumentRewriteRule.java:
##########
@@ -119,17 +131,33 @@ public void onMatch(RelOptRuleCall call) {
/// Returns the data type argument to append, or `null` when the call
already has one or is not such a call. Throws
/// for a data argument without an explicit form, such as an array from a
multi-value column.
@Nullable
- private static String getDataTypeArgument(AggregateCall aggCall, RelNode
input) {
- if (aggCall.getArgList().size() != 2) {
- return null;
- }
+ private String getDataTypeArgument(AggregateCall aggCall, RelNode input) {
String name =
AggregationFunctionType.getNormalizedAggregationFunctionName(aggCall.getAggregation().getName());
- if (!name.equals(AggregationFunctionType.FIRSTWITHTIME.name())
- && !name.equals(AggregationFunctionType.LASTWITHTIME.name())) {
+ List<Integer> argList = aggCall.getArgList();
+ String arguments;
+ if (name.equals(AggregationFunctionType.FIRSTWITHTIME.name())
+ || name.equals(AggregationFunctionType.LASTWITHTIME.name())) {
+ if (argList.size() != 2) {
+ return null;
+ }
+ arguments = "dataColumn, timeColumn";
+ } else if (name.equals(AggregationFunctionType.ANYVALUE.name())) {
+ if (argList.size() == 2) {
+ checkAnyValueDataTypeArgument(name, argList, input);
+ return null;
+ }
+ if (!_rewriteAnyValue) {
+ return null;
+ }
+ arguments = "expr";
+ } else {
return null;
}
- SqlTypeName dataType =
-
input.getRowType().getFieldList().get(aggCall.getArgList().get(0)).getType().getSqlTypeName();
+ return inferDataType(name, arguments,
input.getRowType().getFieldList().get(argList.get(0)).getType());
Review Comment:
For a DECIMAL expression this picks `BIG_DECIMAL`. But the plan types
DECIMAL with scale > 0 as DOUBLE (`resolveDecimal`), and the leaf computes
DOUBLE. So the server converts the whole block to BigDecimal. If any row in the
block is NaN or Infinity, the query fails, even when another row is picked.
Repro with rows `(num=1, den=1)` and `(num=2, den=0)`:
```sql
SET inferAnyValueType=true;
SELECT ANY_VALUE(num / (den * 1.0)) FROM t
```
This fails with `Infinite or NaN`. GROUP BY and skip-leaf fail the same way.
Without the flag it works.
Suggestion: for DECIMAL, use the type that the plan uses, for example
`RelToPlanNodeConverter.convertToColumnDataType(relDataType).toDataType().name()`.
This gives DOUBLE for scale > 0, and INT, LONG or BIG_DECIMAL for scale 0.
Another option: rewrite only in SSE. MSE already returns the ARG0 type, so
in MSE the rewrite adds the old-server risk for little gain.
Note: the same mapping came from #19986, so `FIRST_WITH_TIME(num / (den *
1.0), ts)` may have the same problem. I did not check that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
