This is an automated email from the ASF dual-hosted git repository.
gian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 61ace5ccfc0 SQL: Avoid extractionFns by default. (#18092)
61ace5ccfc0 is described below
commit 61ace5ccfc0c7c02f47963d74c96547abb5c1909
Author: Gian Merlino <[email protected]>
AuthorDate: Thu Aug 21 18:07:52 2025 -0700
SQL: Avoid extractionFns by default. (#18092)
* SQL: Avoid extractionFns by default.
The SQL planner now avoids using extractionFns unless "sqlUseExtractionFns"
is set to "true" is set. This promotes more usage of query vectorization.
This affects the SQL functions LOOKUP, REGEXP_EXTRACT, and SUBSTRING.
* Fix BloomFilterSqlAggregatorTest.
* More idiomatic.
* Remove cannotVectorize() from a bunch of tests.
* Remove dangling comment.
---
docs/querying/sql-query-context.md | 1 +
.../bloom/sql/BloomFilterSqlAggregatorTest.java | 16 +-
.../query/expression/RegexpExtractExprMacro.java | 29 +-
.../sql/calcite/expression/SimpleExtraction.java | 14 +-
.../builtin/QueryLookupOperatorConversion.java | 2 +-
.../builtin/RegexpExtractOperatorConversion.java | 6 +-
.../builtin/SubstringOperatorConversion.java | 4 +-
.../druid/sql/calcite/planner/PlannerContext.java | 23 ++
.../druid/sql/calcite/CalciteArraysQueryTest.java | 15 +-
.../druid/sql/calcite/CalciteExplainQueryTest.java | 2 +-
.../druid/sql/calcite/CalciteJoinQueryTest.java | 44 ++-
.../calcite/CalciteLookupFunctionQueryTest.java | 312 +++++++++------------
.../calcite/CalciteMultiValueStringQueryTest.java | 49 +++-
.../apache/druid/sql/calcite/CalciteQueryTest.java | 252 +++++++++++++----
.../druid/sql/calcite/CalciteSubqueryTest.java | 15 +-
...tUsingSubqueryWithExtractionFns@all_disabled.iq | 17 +-
...stUsingSubqueryWithExtractionFns@all_enabled.iq | 17 +-
.../[email protected] | 17 +-
...xtractionFns@filter-on-value-column_disabled.iq | 17 +-
[email protected] | 17 +-
[email protected] | 17 +-
[email protected] | 17 +-
.../testGroupByLimitPushdownExtraction.iq | 17 +-
.../testRequireTimeConditionPositive3.iq | 17 +-
24 files changed, 581 insertions(+), 356 deletions(-)
diff --git a/docs/querying/sql-query-context.md
b/docs/querying/sql-query-context.md
index 854d3838474..31ec12338aa 100644
--- a/docs/querying/sql-query-context.md
+++ b/docs/querying/sql-query-context.md
@@ -47,6 +47,7 @@ The table below lists the query context parameters you can
use with Druid SQL.
|`useNativeQueryExplain`|If `true`, `EXPLAIN PLAN FOR` returns the explain
plan as a JSON representation of equivalent native query, else it returns the
original version of explain plan generated by Calcite.<br /><br />This property
is provided for backwards compatibility. We don't recommend setting this
parameter unless your application depends on the older behavior.|`true`|
|`sqlFinalizeOuterSketches`|If `false` (default behavior in Druid 25.0.0 and
later), `DS_HLL`, `DS_THETA`, and `DS_QUANTILES_SKETCH` return sketches in
query results. If `true` (default behavior in Druid 24.0.1 and earlier), Druid
finalizes sketches from these functions when they appear in query results.<br
/><br />This property is provided for backwards compatibility with behavior in
Druid 24.0.1 and earlier. We don't recommend setting this parameter unless your
application uses Druid 2 [...]
|`sqlUseBoundAndSelectors`|If `false` (default behavior in Druid 27.0.0 and
later), the SQL planner uses [equality](./filters.md#equality-filter),
[null](./filters.md#null-filter), and [range](./filters.md#range-filter)
filters instead of [selector](./filters.md#selector-filter) and
[bounds](./filters.md#bound-filter). For filtering `ARRAY` typed values,
`sqlUseBoundAndSelectors` must be `false`. | `false`.|
+|`sqlUseExtractionFns`|If false, the SQL planner avoids using
[`extractionFn`](dimensionspecs.md#extraction-functions) in favor of using
other constructs such as [virtual columns](virtual-columns.md). This parameter
is provided for compatibility with prior behavior, and may be removed in a
future release.|false|
|`sqlReverseLookup`|Whether to consider the [reverse-lookup
rewrite](lookups.md#reverse-lookup) of the `LOOKUP` function during SQL
planning.<br /><br />Druid reverses calls to `LOOKUP` only when the number of
matching keys is lower than both `inSubQueryThreshold` and
`sqlReverseLookupThreshold`.|`true`|
|`sqlReverseLookupThreshold`|Maximum size of `IN` filter to create when
applying a [reverse-lookup rewrite](lookups.md#reverse-lookup). If a `LOOKUP`
call matches more keys than the specified threshold, it remains unchanged.<br
/><br />If `inSubQueryThreshold` is lower than `sqlReverseLookupThreshold`,
Druid uses `inSubQueryThreshold` threshold instead.|10000|
|`sqlPullUpLookup`|Whether to consider the [pull-up
rewrite](lookups.md#pull-up) of the `LOOKUP` function during SQL
planning.|`true`|
diff --git
a/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java
b/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java
index cb9e04de583..857521fbf9f 100644
---
a/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java
+++
b/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java
@@ -32,9 +32,7 @@ import
org.apache.druid.query.aggregation.FilteredAggregatorFactory;
import org.apache.druid.query.aggregation.bloom.BloomFilterAggregatorFactory;
import
org.apache.druid.query.aggregation.bloom.sql.BloomFilterSqlAggregatorTest.BloomFilterSqlAggComponentSupplier;
import org.apache.druid.query.dimension.DefaultDimensionSpec;
-import org.apache.druid.query.dimension.ExtractionDimensionSpec;
import org.apache.druid.query.expression.TestExprMacroTable;
-import org.apache.druid.query.extraction.SubstringDimExtractionFn;
import org.apache.druid.query.filter.BloomKFilter;
import org.apache.druid.query.groupby.GroupByQuery;
import org.apache.druid.query.spec.MultipleIntervalSegmentSpec;
@@ -246,15 +244,19 @@ public class BloomFilterSqlAggregatorTest extends
BaseCalciteQueryTest
.dataSource(CalciteTests.DATASOURCE3)
.intervals(new
MultipleIntervalSegmentSpec(ImmutableList.of(Filtration.eternity())))
.granularity(Granularities.ALL)
+ .virtualColumns(
+ new ExpressionVirtualColumn(
+ "v0",
+ "substring(\"dim1\", 0, 1)",
+ ColumnType.STRING,
+ TestExprMacroTable.INSTANCE
+ )
+ )
.aggregators(
ImmutableList.of(
new BloomFilterAggregatorFactory(
"a0",
- new ExtractionDimensionSpec(
- "dim1",
- "a0:dim1",
- new SubstringDimExtractionFn(0, 1)
- ),
+ new DefaultDimensionSpec("v0", "a0:v0"),
TEST_NUM_ENTRIES
)
)
diff --git
a/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java
b/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java
index e4eddc5857b..9d95ec1e9f1 100644
---
a/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java
+++
b/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java
@@ -19,6 +19,7 @@
package org.apache.druid.query.expression;
+import org.apache.druid.error.InvalidInput;
import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.math.expr.Expr;
import org.apache.druid.math.expr.ExprEval;
@@ -30,6 +31,7 @@ import javax.annotation.Nullable;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
public class RegexpExtractExprMacro implements ExprMacroTable.ExprMacro
{
@@ -59,9 +61,7 @@ public class RegexpExtractExprMacro implements
ExprMacroTable.ExprMacro
}
// Precompile the pattern.
- final Pattern pattern = Pattern.compile(
- StringUtils.nullToEmptyNonDruidDataString((String)
patternExpr.getLiteralValue())
- );
+ final Pattern pattern = compilePattern((String)
patternExpr.getLiteralValue());
final int index = indexExpr == null ? 0 : ((Number)
indexExpr.getLiteralValue()).intValue();
@@ -79,7 +79,7 @@ public class RegexpExtractExprMacro implements
ExprMacroTable.ExprMacro
final String s = arg.eval(bindings).asString();
if (s == null) {
- // True nulls do not match anything. Note: this branch only executes
in SQL-compatible null handling mode.
+ // True nulls do not match anything.
return ExprEval.of(null);
} else {
final Matcher matcher = pattern.matcher(s);
@@ -97,4 +97,25 @@ public class RegexpExtractExprMacro implements
ExprMacroTable.ExprMacro
}
return new RegexpExtractExpr(args);
}
+
+ /**
+ * Compile the provided pattern, or provide a nice error message if it
cannot be compiled.
+ */
+ private static Pattern compilePattern(@Nullable String pattern)
+ {
+ try {
+ return
Pattern.compile(StringUtils.nullToEmptyNonDruidDataString(pattern));
+ }
+ catch (PatternSyntaxException e) {
+ throw InvalidInput.exception(
+ e,
+ StringUtils.format(
+ "An invalid pattern [%s] was provided for the %s function,
error: [%s]",
+ e.getPattern(),
+ FN_NAME,
+ e.getMessage()
+ )
+ );
+ }
+ }
}
diff --git
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java
index 81d19d1b767..c66404fd979 100644
---
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java
+++
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java
@@ -26,6 +26,7 @@ import org.apache.druid.query.dimension.DimensionSpec;
import org.apache.druid.query.dimension.ExtractionDimensionSpec;
import org.apache.druid.query.extraction.ExtractionFn;
import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
import javax.annotation.Nullable;
import java.util.Objects;
@@ -41,12 +42,19 @@ public class SimpleExtraction
@Nullable
private final ExtractionFn extractionFn;
- public SimpleExtraction(String column, @Nullable ExtractionFn extractionFn)
+ private SimpleExtraction(String column, @Nullable ExtractionFn extractionFn)
{
this.column = Preconditions.checkNotNull(column, "column");
this.extractionFn = extractionFn;
}
+ /**
+ * Create an instance.
+ *
+ * @param column column
+ * @param extractionFn extraction fn, if any. Callers should only provide a
function if
+ * {@link PlannerContext#isUseExtractionFns()} is true.
+ */
public static SimpleExtraction of(String column, @Nullable ExtractionFn
extractionFn)
{
return new SimpleExtraction(column, extractionFn);
@@ -63,6 +71,10 @@ public class SimpleExtraction
return extractionFn;
}
+ /**
+ * Create a new instance where the provided {@link ExtractionFn} is composed
with the existing {@link ExtractionFn}
+ * of this instance. Callers should only use this method if {@link
PlannerContext#isUseExtractionFns()} is true.
+ */
public SimpleExtraction cascade(final ExtractionFn nextExtractionFn)
{
return new SimpleExtraction(
diff --git
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java
index af4ecd9426d..b9f01648c5f 100644
---
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java
+++
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java
@@ -86,7 +86,7 @@ public class QueryLookupOperatorConversion implements
SqlOperatorConversion
// Add the lookup name to the set of lookups to selectively load.
plannerContext.addLookupToLoad(lookupExtractorFactoryContainerProvider.getCanonicalLookupName(lookupName));
- if (arg.isSimpleExtraction() && lookupNameExpr.isLiteral()) {
+ if (arg.isSimpleExtraction() && lookupNameExpr.isLiteral() &&
plannerContext.isUseExtractionFns()) {
return arg.getSimpleExtraction().cascade(
new RegisteredLookupExtractionFn(
lookupExtractorFactoryContainerProvider,
diff --git
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java
index 0de60954130..cd7fcd00b13 100644
---
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java
+++
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java
@@ -74,10 +74,12 @@ public class RegexpExtractOperatorConversion implements
SqlOperatorConversion
?
plannerContext.parseExpression(inputExpressions.get(2).getExpression())
: null;
- if (arg.isSimpleExtraction() && patternExpr.isLiteral() &&
(indexExpr == null || indexExpr.isLiteral())) {
+ if (arg.isSimpleExtraction()
+ && patternExpr.isLiteral()
+ && (indexExpr == null || indexExpr.isLiteral())
+ && plannerContext.isUseExtractionFns()) {
final String pattern = (String) patternExpr.getLiteralValue();
-
try {
return arg.getSimpleExtraction().cascade(
new RegexDimExtractionFn(
diff --git
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java
index b03a74d9fbb..d4573421cb0 100644
---
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java
+++
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java
@@ -106,7 +106,9 @@ public class SubstringOperatorConversion implements
SqlOperatorConversion
return input.map(
simpleExtraction -> {
- if (adjustedIndexLiteral != null && (lengthNode == null ||
lengthLiteral != null)) {
+ if (adjustedIndexLiteral != null
+ && (lengthNode == null || lengthLiteral != null)
+ && plannerContext.isUseExtractionFns()) {
return simpleExtraction.cascade(new
SubstringDimExtractionFn(adjustedIndexLiteral, lengthLiteral));
} else {
return null;
diff --git
a/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java
b/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java
index 812b888ce76..859d1d5715c 100644
--- a/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java
+++ b/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java
@@ -41,6 +41,7 @@ import org.apache.druid.query.JoinAlgorithm;
import org.apache.druid.query.QueryContext;
import org.apache.druid.query.QueryContexts;
import org.apache.druid.query.explain.ExplainAttributes;
+import org.apache.druid.query.extraction.ExtractionFn;
import org.apache.druid.query.filter.InDimFilter;
import org.apache.druid.query.filter.TypedInFilter;
import org.apache.druid.query.lookup.LookupExtractor;
@@ -105,6 +106,12 @@ public class PlannerContext
public static final String CTX_SQL_USE_BOUNDS_AND_SELECTORS =
"sqlUseBoundAndSelectors";
public static final boolean DEFAULT_SQL_USE_BOUNDS_AND_SELECTORS = false;
+ /**
+ * Context key for {@link PlannerContext#isUseExtractionFns()}.
+ */
+ public static final String CTX_SQL_USE_EXTRACTION_FNS =
"sqlUseExtractionFns";
+ public static final boolean DEFAULT_SQL_USE_EXTRACTION_FNS = false;
+
/**
* Context key for {@link PlannerContext#isPullUpLookup()}.
*/
@@ -141,6 +148,7 @@ public class PlannerContext
private String sqlQueryId;
private boolean stringifyArrays;
private boolean useBoundsAndSelectors;
+ private boolean useExtractionFns;
private boolean pullUpLookup;
private boolean reverseLookup;
private boolean useGranularity;
@@ -356,6 +364,14 @@ public class PlannerContext
return useBoundsAndSelectors;
}
+ /**
+ * Whether we should use {@link ExtractionFn} in planning.
+ */
+ public boolean isUseExtractionFns()
+ {
+ return useExtractionFns;
+ }
+
/**
* Whether we should use {@link InDimFilter} (true) or {@link TypedInFilter}
(false).
*/
@@ -689,6 +705,13 @@ public class PlannerContext
useBoundsAndSelectors = DEFAULT_SQL_USE_BOUNDS_AND_SELECTORS;
}
+ final Object useExtractionFnsParam =
queryContext.get(CTX_SQL_USE_EXTRACTION_FNS);
+ if (useExtractionFnsParam != null) {
+ useExtractionFns = Numbers.parseBoolean(useExtractionFnsParam);
+ } else {
+ useExtractionFns = DEFAULT_SQL_USE_EXTRACTION_FNS;
+ }
+
final Object pullUpLookupParam = queryContext.get(CTX_SQL_PULL_UP_LOOKUP);
if (pullUpLookupParam != null) {
pullUpLookup = Numbers.parseBoolean(pullUpLookupParam);
diff --git
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
index 63dc9946815..1d754142555 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
@@ -73,6 +73,7 @@ import
org.apache.druid.sql.calcite.DecoupledTestConfig.QuidemTestCaseReason;
import org.apache.druid.sql.calcite.NotYetSupported.Modes;
import org.apache.druid.sql.calcite.filtration.Filtration;
import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
import org.apache.druid.sql.calcite.util.CalciteTests;
import
org.apache.druid.sql.calcite.util.SqlTestFramework.StandardComponentSupplier;
import org.apache.druid.sql.http.SqlParameter;
@@ -1014,8 +1015,14 @@ public class CalciteArraysQueryTest extends
BaseCalciteQueryTest
@Test
public void testArrayOverlapFilterWithExtractionFn()
{
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ Map.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
testQuery(
"SELECT dim3 FROM druid.numfoo WHERE ARRAY_OVERLAP(SUBSTRING(dim3, 1,
1), ARRAY['a','b']) LIMIT 5",
+ queryContext,
ImmutableList.of(
newScanQueryBuilder()
.dataSource(CalciteTests.DATASOURCE3)
@@ -1027,7 +1034,7 @@ public class CalciteArraysQueryTest extends
BaseCalciteQueryTest
.columnTypes(ColumnType.STRING)
.resultFormat(ScanQuery.ResultFormat.RESULT_FORMAT_COMPACTED_LIST)
.limit(5)
- .context(QUERY_CONTEXT_DEFAULT)
+ .context(queryContext)
.build()
),
ImmutableList.of(
@@ -1293,6 +1300,11 @@ public class CalciteArraysQueryTest extends
BaseCalciteQueryTest
@Test
public void testArrayContainsFilterWithExtractionFn()
{
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
Druids.ScanQueryBuilder builder = newScanQueryBuilder()
.dataSource(CalciteTests.DATASOURCE3)
.intervals(querySegmentSpec(Filtration.eternity()))
@@ -1310,6 +1322,7 @@ public class CalciteArraysQueryTest extends
BaseCalciteQueryTest
);
testQuery(
"SELECT dim3 FROM druid.numfoo WHERE ARRAY_CONTAINS(SUBSTRING(dim3, 1,
1), ARRAY['a','b']) LIMIT 5",
+ queryContext,
ImmutableList.of(builder.build()),
ImmutableList.of(
new Object[]{"[\"a\",\"b\"]"}
diff --git
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java
index d57eb5d8f97..c4231fe5661 100644
---
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java
+++
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java
@@ -103,7 +103,7 @@ public class CalciteExplainQueryTest extends
BaseCalciteQueryTest
+ " SELECT SUBSTRING(dim1, 1, 1) FROM druid.foo
WHERE dim1 IS NOT NULL\n"
+ " )\n"
+ ")";
- final String explanation =
"[{\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"join\",\"left\":{\"type\":\"table\",\"name\":\"foo\"},\"right\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"table\",\"name\":\"foo\"},\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z\"]},\"filter\":{\"type\":\"not\"
[...]
+ final String explanation =
"[{\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"join\",\"left\":{\"type\":\"table\",\"name\":\"foo\"},\"right\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"table\",\"name\":\"foo\"},\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z\"]},\"virtualColumns\":[{\"type\
[...]
final String resources = "[{\"name\":\"foo\",\"type\":\"DATASOURCE\"}]";
final String attributes = "{\"statementType\":\"SELECT\"}";
diff --git
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java
index eba5d515884..650135ef43d 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java
@@ -62,8 +62,6 @@ import
org.apache.druid.query.aggregation.cardinality.CardinalityAggregatorFacto
import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
import org.apache.druid.query.dimension.DefaultDimensionSpec;
-import org.apache.druid.query.dimension.ExtractionDimensionSpec;
-import org.apache.druid.query.extraction.SubstringDimExtractionFn;
import org.apache.druid.query.filter.LikeDimFilter;
import org.apache.druid.query.groupby.GroupByQuery;
import org.apache.druid.query.groupby.ResultRow;
@@ -1535,13 +1533,14 @@ public class CalciteJoinQueryTest extends
BaseCalciteQueryTest
.setDataSource(new
LookupDataSource("lookyloo"))
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
- .setDimensions(
- new ExtractionDimensionSpec(
- "k",
- "d0",
- new SubstringDimExtractionFn(0, 1)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "substring(\"k\", 0, 1)",
+ ColumnType.STRING
)
)
+ .setDimensions(new DefaultDimensionSpec("v0",
"d0"))
.setAggregatorSpecs(new
StringAnyAggregatorFactory("a0", "v", 10, true))
.build()
),
@@ -2765,19 +2764,17 @@ public class CalciteJoinQueryTest extends
BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "substring(\"dim1\",
0, 1)",
+ ColumnType.STRING
+ )
+ )
.setDimFilter(
not(equality("dim1", "",
ColumnType.STRING))
)
- .setDimensions(
- dimensions(new
ExtractionDimensionSpec(
- "dim1",
- "d0",
- new
SubstringDimExtractionFn(
- 0,
- 1
- )
- ))
- )
+ .setDimensions(dimensions(new
DefaultDimensionSpec("v0", "d0")))
.setContext(QUERY_CONTEXT_DEFAULT)
.build()
),
@@ -4570,16 +4567,9 @@ public class CalciteJoinQueryTest extends
BaseCalciteQueryTest
)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
- .setDimensions(
- dimensions(
- new ExtractionDimensionSpec(
- "dim1",
- "d0",
- ColumnType.STRING,
- new SubstringDimExtractionFn(0, 10)
- )
- )
- )
+ .setVirtualColumns(
+ expressionVirtualColumn("v0", "substring(\"dim1\",
0, 10)", ColumnType.STRING))
+ .setDimensions(new DefaultDimensionSpec("v0", "d0",
ColumnType.STRING))
.setAggregatorSpecs(aggregators(new
LongSumAggregatorFactory("a0", "cnt")))
.setContext(queryContext)
.build()
diff --git
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java
index 2faeae3fda4..0553164a312 100644
---
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java
+++
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java
@@ -47,6 +47,7 @@ import org.apache.druid.segment.VirtualColumn;
import org.apache.druid.segment.VirtualColumns;
import org.apache.druid.segment.column.ColumnType;
import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
import org.apache.druid.sql.calcite.filtration.Filtration;
import org.apache.druid.sql.calcite.planner.PlannerConfig;
import org.apache.druid.sql.calcite.planner.PlannerContext;
@@ -74,12 +75,19 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
.put(ReverseLookupRule.CTX_MAX_OPTIMIZE_COUNT, 1)
.build();
+ /**
+ * For tests that use the lookup extraction function rather than expressions.
+ */
+ private static final Map<String, Object> QUERY_CONTEXT_WITH_EXTRACTION_FNS =
+ ImmutableMap.<String, Object>builder()
+ .putAll(QUERY_CONTEXT)
+ .put(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ .build();
+
+ private static final String LOOKUP_EXPRESSION =
"lookup(\"dim1\",'lookyloo')";
private static final ExtractionFn EXTRACTION_FN =
new RegisteredLookupExtractionFn(null, "lookyloo", false, null, null,
false);
- private static final ExtractionFn EXTRACTION_FN_121 =
- new RegisteredLookupExtractionFn(null, "lookyloo121", false, null, null,
false);
-
@Test
public void testFilterEquals()
{
@@ -143,8 +151,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInLookupOfConcat()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(CONCAT(dim1, 'a', dim2), 'lookyloo') IN
('xa', 'xabc')"),
QUERY_CONTEXT,
@@ -167,8 +173,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterScalarInArrayLookupOfConcat()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("SCALAR_IN_ARRAY(LOOKUP(CONCAT(dim1, 'a', dim2),
'lookyloo'), ARRAY['xa', 'xabc'])"),
QUERY_CONTEXT,
@@ -205,7 +209,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInConcatOfLookup()
{
- cannotVectorize();
// One optimize call is needed for each "IN" value, because this
expression is decomposed into a sequence of
// [(LOOKUP(dim1, 'lookyloo') = 'xabc' AND dim1 = 'abc') OR ...]. They
can't be collected and combined.
@@ -246,8 +249,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInConcatOfLookupOfConcat()
{
- cannotVectorize();
-
// One optimize call is needed for each "IN" value, because this
expression is decomposed into a sequence of
// [(LOOKUP(dim1, 'lookyloo') = 'xabc' AND dim1 = 'abc') OR ...]. They
can't be collected and combined.
@@ -356,8 +357,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMultipleIsNotDistinctFrom()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS NOT DISTINCT FROM
'xabc' OR "
+ "LOOKUP(dim1, 'lookyloo') IS NOT DISTINCT FROM
'x6' OR "
@@ -371,8 +370,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterIn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6',
'nonexistent')"),
QUERY_CONTEXT,
@@ -384,8 +381,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterScalarInArray()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("SCALAR_IN_ARRAY(LOOKUP(dim1, 'lookyloo'),
ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
@@ -397,7 +392,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInOverScalarInArrayThreshold()
{
- cannotVectorize();
// Set inFunctionThreshold = 1 to cause the IN to be converted to
SCALAR_IN_ARRAY.
final ImmutableMap<String, Object> queryContext =
@@ -418,7 +412,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInOverMaxSize()
{
- cannotVectorize();
// Set sqlReverseLookupThreshold = 1 to stop the LOOKUP call from being
reversed.
final ImmutableMap<String, Object> queryContext =
@@ -432,7 +425,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6',
'nonexistent')"),
queryContext,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
in("v0", ImmutableList.of("nonexistent", "x6", "xabc"))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -442,7 +435,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInOverMaxSize2()
{
- cannotVectorize();
// Set inSubQueryThreshold = 1 to stop the LOOKUP call from being reversed.
final ImmutableMap<String, Object> queryContext =
@@ -456,7 +448,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') = 'xabc' OR LOOKUP(dim1,
'lookyloo') = 'x6'"),
queryContext,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
in("v0", ImmutableList.of("x6", "xabc"))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -466,14 +458,12 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInOrIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent') OR
LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
or(
in("dim1", Arrays.asList("6", "abc")),
isNull("v0")
@@ -486,8 +476,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInAndIsNotNull()
{
- cannotVectorize();
-
// Ideally we'd be able to eliminate "AND LOOKUP(dim1, 'lookyloo') IS NOT
NULL", because it's implied by
// "LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent')". We're not
currently able to do that.
@@ -496,7 +484,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
"LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent') AND
LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
and(
in("dim1", ImmutableList.of("6", "abc")),
not(isNull("v0"))
@@ -509,8 +497,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInOrIsNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo121') IN ('xabc', 'x6', 'nonexistent') OR
LOOKUP(dim1, 'lookyloo121') IS NULL"),
@@ -525,14 +511,12 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotInAndIsNotNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo') NOT IN ('x6', 'nonexistent') AND
LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
and(
not(equality("v0", "x6", ColumnType.STRING)),
not(equality("v0", "nonexistent", ColumnType.STRING)),
@@ -546,15 +530,13 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrueAndIsNotNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"(LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent')) IS
NOT TRUE "
+ "AND LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
and(
not(istrue(in("dim1", ImmutableList.of("6", "abc")))),
notNull("v0")
@@ -567,8 +549,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotInAndIsNotNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo121') NOT IN ('xabc', 'xdef',
'nonexistent') "
@@ -587,14 +567,12 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotInOrIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo') NOT IN ('x6', 'nonexistent') OR
LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
or(
and(
not(equality("v0", "x6", ColumnType.STRING)),
@@ -613,14 +591,12 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrueOrIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"(LOOKUP(dim1, 'lookyloo') IN ('x6', 'nonexistent')) IS NOT TRUE
OR LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
or(
not(istrue(equality("dim1", "6", ColumnType.STRING))),
isNull("v0")
@@ -636,13 +612,11 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotIn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') NOT IN ('x6',
'nonexistent')"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
and(not(equality("v0", "x6", ColumnType.STRING)),
not(equality("v0", "nonexistent", ColumnType.STRING)))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -652,8 +626,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IN ('x6', 'nonexistent')
IS NOT TRUE"),
QUERY_CONTEXT,
@@ -670,8 +642,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotInInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') NOT IN ('xabc',
'xdef', 'nonexistent')"),
QUERY_CONTEXT,
@@ -683,8 +653,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotInWithReplaceMissingValue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo', 'xyzzy') NOT IN ('xabc',
'x6', 'nonexistent')"),
QUERY_CONTEXT,
@@ -696,8 +664,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrueWithReplaceMissingValue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo', 'xyzzy') IN ('xabc',
'x6', 'nonexistent') IS NOT TRUE"),
QUERY_CONTEXT,
@@ -709,8 +675,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMvContains()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_CONTAINS(LOOKUP(dim1, 'lookyloo'), 'xabc')"),
QUERY_CONTEXT,
@@ -749,8 +713,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMvOverlap()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo'), ARRAY['xabc',
'x6', 'nonexistent'])"),
QUERY_CONTEXT,
@@ -768,7 +730,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo'), ARRAY['xabc',
'x6', 'nonexistent', NULL])"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- in("dim1", Arrays.asList(null, "nonexistent", "x6", "xabc"),
EXTRACTION_FN)
+
expressionFilter("mv_overlap(lookup(\"dim1\",'lookyloo'),array('xabc','x6','nonexistent',null))")
),
ImmutableList.of(
new Object[]{null, 5L},
@@ -780,8 +742,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMvOverlapNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo121'),
ARRAY['xabc', 'x6', 'nonexistent', NULL])"),
QUERY_CONTEXT,
@@ -801,18 +761,18 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
buildFilterTestSql("NOT MV_CONTAINS(lookup(dim1, 'lookyloo'),
'xabc')"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
- not(equality("v0", "xabc", ColumnType.STRING))
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
+
not(expressionFilter("mv_contains(lookup(\"dim1\",'lookyloo'),'xabc')"))
),
- ImmutableList.of()
+ ImmutableList.of(
+ new Object[]{null, 5L}
+ )
);
}
@Test
public void testFilterMvContainsIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_CONTAINS(lookup(dim1, 'lookyloo'), 'xabc') IS
NOT TRUE"),
QUERY_CONTEXT,
@@ -826,8 +786,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotMvContainsInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT MV_CONTAINS(LOOKUP(dim1, 'lookyloo121'),
'xabc')"),
QUERY_CONTEXT,
@@ -845,7 +803,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
buildFilterTestSql("NOT MV_OVERLAP(lookup(dim1, 'lookyloo'),
ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- not(in("dim1", ImmutableList.of("nonexistent", "x6", "xabc"),
EXTRACTION_FN))
+
not(expressionFilter("mv_overlap(lookup(\"dim1\",'lookyloo'),array('xabc','x6','nonexistent'))"))
),
Collections.emptyList()
);
@@ -854,8 +812,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMvOverlapIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo'), ARRAY['xabc',
'x6', 'nonexistent']) IS NOT TRUE"),
QUERY_CONTEXT,
@@ -869,8 +825,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotMvOverlapInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT MV_OVERLAP(lookup(dim1, 'lookyloo121'),
ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
@@ -882,7 +836,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMultipleIsDistinctFrom()
{
- cannotVectorize();
// One optimize call is needed for each "IS DISTINCT FROM", because "x IS
DISTINCT FROM y" is sugar for
// "(x = y) IS NOT TRUE", and ReverseLookupRule doesn't peek into the "IS
NOT TRUE" calls nested beneatth
@@ -915,13 +868,11 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
isNull("v0")
),
ImmutableList.of(new Object[]{null, 5L})
@@ -944,13 +895,11 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterIsNotNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
not(isNull("v0"))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -960,8 +909,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterIsNotNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') IS NOT NULL"),
QUERY_CONTEXT,
@@ -978,13 +925,11 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotEquals()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') <> 'x6'"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
not(equality("v0", "x6", ColumnType.STRING))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -994,8 +939,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotEqualsInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') <> 'xabc'"),
QUERY_CONTEXT,
@@ -1007,8 +950,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterEqualsIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') = 'x6' IS NOT TRUE"),
QUERY_CONTEXT,
@@ -1025,8 +966,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterEqualsIsNotTrueInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') = 'xabc' IS NOT TRUE"),
QUERY_CONTEXT,
@@ -1040,8 +979,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterIsDistinctFrom()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS DISTINCT FROM 'x6'"),
QUERY_CONTEXT,
@@ -1058,11 +995,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterIsDistinctFromReplaceMissingValueWithSameLiteral()
{
- cannotVectorize();
-
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null,
false);
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo', 'x6') IS DISTINCT FROM
'x6'"),
QUERY_CONTEXT,
@@ -1077,13 +1009,11 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotEquals2()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT (LOOKUP(dim1, 'lookyloo') = 'x6' OR cnt = 2)"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')",
ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING),
and(
not(equality("v0", "x6", ColumnType.STRING)),
not(equality("cnt", 2L, ColumnType.LONG))
@@ -1096,8 +1026,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterEqualsIsNotTrue2()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("(LOOKUP(dim1, 'lookyloo') = 'x6' OR cnt = 2) IS
NOT TRUE"),
QUERY_CONTEXT,
@@ -1117,8 +1045,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterNotEquals2Injective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT (LOOKUP(dim1, 'lookyloo121') = 'xdef' OR cnt =
2)"),
QUERY_CONTEXT,
@@ -1136,8 +1062,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceSameLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6') = 'x6'"),
QUERY_CONTEXT,
@@ -1165,11 +1089,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInCoalesceSameLiteral()
{
- cannotVectorize();
-
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null,
false);
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6') IN ('xa',
'xabc', 'x6')"),
QUERY_CONTEXT,
@@ -1187,8 +1106,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInCoalesceSameLiteralInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo121'), 'x2') IN
('xabc', 'xdef', 'x2')"),
QUERY_CONTEXT,
@@ -1204,15 +1121,11 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
{
cannotVectorize();
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null,
false);
-
testQuery(
buildFilterTestSql("MV_CONTAINS(COALESCE(LOOKUP(dim1, 'lookyloo'),
'x6'), 'x6')"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo','x6')",
ColumnType.STRING),
- equality("v0", "x6", ColumnType.STRING)
+
expressionFilter("mv_contains(lookup(\"dim1\",'lookyloo','x6'),'x6')")
),
ImmutableList.of(new Object[]{null, 5L})
);
@@ -1223,25 +1136,23 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
{
cannotVectorize();
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null,
false);
-
testQuery(
buildFilterTestSql("MV_OVERLAP(COALESCE(LOOKUP(dim1, 'lookyloo'),
'x6'), ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
- buildFilterTestExpectedQuery(in("dim1", ImmutableList.of("xabc", "x6",
"nonexistent"), extractionFn)),
- ImmutableList.of(new Object[]{null, 5L}, new Object[]{"xabc", 1L})
+ buildFilterTestExpectedQuery(
+
+
expressionFilter("mv_overlap(lookup(\"dim1\",'lookyloo','x6'),array('xabc','x6','nonexistent'))")
+ ),
+ ImmutableList.of(
+ new Object[]{null, 5L},
+ new Object[]{"xabc", 1L}
+ )
);
}
@Test
public void testFilterCoalesceSameLiteralNotEquals()
{
- cannotVectorize();
-
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null,
false);
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6') <> 'x6'"),
QUERY_CONTEXT,
@@ -1256,8 +1167,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceSameLiteralNotEqualsInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo121'), 'xabc') <>
'xabc'"),
QUERY_CONTEXT,
@@ -1302,8 +1211,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceCastBigintDifferentLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(CAST(LOOKUP(dim1, 'lookyloo') AS BIGINT),
1) = 6"),
QUERY_CONTEXT,
@@ -1318,8 +1225,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMvContainsCoalesceDifferentLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_CONTAINS(COALESCE(LOOKUP(dim1, 'lookyloo'),
'xyzzy'), 'x6')"),
QUERY_CONTEXT,
@@ -1331,8 +1236,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterMvOverlapCoalesceDifferentLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"MV_OVERLAP(COALESCE(LOOKUP(dim1, 'lookyloo'), 'xyzzy'),
ARRAY['xabc', 'x6', 'nonexistent'])"),
@@ -1345,8 +1248,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceDifferentLiteralNotEquals()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'xyzzy') <>
'x6'"),
QUERY_CONTEXT,
@@ -1361,8 +1262,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceDifferentLiteralNotEqualsAlwaysTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'xyzzy') <>
'nonexistent'"),
QUERY_CONTEXT,
@@ -1377,8 +1276,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceSameColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim1) = 'x6'"),
QUERY_CONTEXT,
@@ -1393,8 +1290,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterInCoalesceSameColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim1) IN
('xabc', '10.1')"),
QUERY_CONTEXT,
@@ -1412,8 +1307,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceFunctionOfSameColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim1 || '') =
'x6'"),
QUERY_CONTEXT,
@@ -1428,8 +1321,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testFilterCoalesceDifferentColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim2) = 'x6'"),
QUERY_CONTEXT,
@@ -1465,7 +1356,49 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testLookupReplaceMissingValueWith()
{
- // Cannot vectorize due to extraction dimension specs.
+ testQuery(
+ "SELECT\n"
+ + " LOOKUP(dim1, 'lookyloo', 'Missing_Value'),\n"
+ + " COALESCE(LOOKUP(dim1, 'lookyloo'), 'Missing_Value'), -- converted
to the first form\n"
+ + " LOOKUP(dim1, 'lookyloo', null) as rmvNull,\n"
+ + " COUNT(*)\n"
+ + "FROM foo\n"
+ + "GROUP BY 1,2,3",
+ QUERY_CONTEXT,
+ ImmutableList.of(
+ GroupByQuery.builder()
+ .setDataSource(CalciteTests.DATASOURCE1)
+ .setInterval(querySegmentSpec(Filtration.eternity()))
+ .setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "lookup(\"dim1\",'lookyloo','Missing_Value')",
+ ColumnType.STRING
+ ),
+ expressionVirtualColumn("v1",
"lookup(\"dim1\",'lookyloo',null)", ColumnType.STRING)
+ )
+ .setDimensions(
+ dimensions(
+ new DefaultDimensionSpec("v0", "d0",
ColumnType.STRING),
+ new DefaultDimensionSpec("v0", "d1",
ColumnType.STRING),
+ new DefaultDimensionSpec("v1", "d2",
ColumnType.STRING)
+ )
+ )
+ .setAggregatorSpecs(new CountAggregatorFactory("a0"))
+ .setContext(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
+ ImmutableList.of(
+ new Object[]{"Missing_Value", "Missing_Value", null, 5L},
+ new Object[]{"xabc", "xabc", "xabc", 1L}
+ )
+ );
+ }
+
+ @Test
+ public void testLookupReplaceMissingValueWithExtractionFns()
+ {
cannotVectorize();
final RegisteredLookupExtractionFn extractionFn1 =
@@ -1478,7 +1411,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
+ " COUNT(*)\n"
+ "FROM foo\n"
+ "GROUP BY 1,2,3",
- QUERY_CONTEXT,
+ QUERY_CONTEXT_WITH_EXTRACTION_FNS,
ImmutableList.of(
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
@@ -1507,7 +1440,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
)
)
.setAggregatorSpecs(new CountAggregatorFactory("a0"))
- .setContext(QUERY_CONTEXT_DEFAULT)
+ .setContext(QUERY_CONTEXT_WITH_EXTRACTION_FNS)
.build()
),
ImmutableList.of(
@@ -1520,9 +1453,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testCountDistinctOfLookup()
{
- // Cannot vectorize due to extraction dimension spec.
- cannotVectorize();
-
testQuery(
"SELECT COUNT(DISTINCT LOOKUP(dim1, 'lookyloo')) FROM foo",
QUERY_CONTEXT,
@@ -1531,11 +1461,14 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
.dataSource(CalciteTests.DATASOURCE1)
.intervals(querySegmentSpec(Filtration.eternity()))
.granularity(Granularities.ALL)
+ .virtualColumns(
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING)
+ )
.aggregators(aggregators(
new CardinalityAggregatorFactory(
"a0",
null,
- ImmutableList.of(new ExtractionDimensionSpec("dim1",
null, EXTRACTION_FN)),
+ ImmutableList.of(new DefaultDimensionSpec("v0",
"v0", ColumnType.STRING)),
false,
true
)
@@ -1615,9 +1548,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
+ "SUM(foo.cnt) FROM druid.foo "
+ "GROUP BY 1";
- // ExtractionDimensionSpec cannot be vectorized
- cannotVectorize();
-
testQuery(
PLANNER_CONFIG_NO_HLL.withOverrides(
ImmutableMap.of(
@@ -1636,13 +1566,15 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ LOOKUP_EXPRESSION,
+ ColumnType.STRING
+ )
+ )
.setDimensions(dimensions(
- new ExtractionDimensionSpec(
- "dim1",
- "d0",
- ColumnType.STRING,
- EXTRACTION_FN
- ),
+ new DefaultDimensionSpec("v0",
"d0", ColumnType.STRING),
new
DefaultDimensionSpec("dim2", "d1", ColumnType.STRING)
))
.setAggregatorSpecs(
@@ -1650,7 +1582,7 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
new
LongSumAggregatorFactory("a0", "cnt"),
new
GroupingAggregatorFactory(
"a1",
- Arrays.asList("dim1",
"dim2")
+ Arrays.asList("v0",
"dim2")
)
)
)
@@ -1756,15 +1688,9 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
.setVirtualColumns(
- new VirtualColumn[]{
- expressionVirtualColumn(
- "v0",
- "lookup(\"dim1\",'lookyloo121')",
- ColumnType.STRING
- )
- }
+ expressionVirtualColumn("v0",
"lookup(\"dim1\",'lookyloo121')", ColumnType.STRING)
)
- .setDimensions(dimensions(new
ExtractionDimensionSpec("dim1", "d0", EXTRACTION_FN_121)))
+ .setDimensions(dimensions(new
DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
.setAggregatorSpecs(
aggregators(
new FilteredAggregatorFactory(
@@ -1849,8 +1775,6 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Test
public void testPullUpLookupOneInjectiveOneNot()
{
- cannotVectorize();
-
testQuery(
"SELECT COUNT(*), LOOKUP(dim1, 'lookyloo'), LOOKUP(dim1,
'lookyloo121') FROM druid.foo GROUP BY 2, 3",
ImmutableList.of(
@@ -1858,8 +1782,11 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION,
ColumnType.STRING)
+ )
.setDimensions(dimensions(
- new ExtractionDimensionSpec("dim1", "d0",
EXTRACTION_FN),
+ new DefaultDimensionSpec("v0", "d0",
ColumnType.STRING),
new DefaultDimensionSpec("dim1", "d1",
ColumnType.STRING)
))
.setAggregatorSpecs(aggregators(new
CountAggregatorFactory("a0")))
@@ -1891,16 +1818,39 @@ public class CalciteLookupFunctionQueryTest extends
BaseCalciteQueryTest
@Nullable final DimFilter expectedFilter
)
{
+ final VirtualColumns virtualColumns;
+ final String groupByDimension;
+
+ if (expectedVirtualColumn != null) {
+ // The filter uses a specified virtual column. GROUP BY always uses a
virtual column for LOOKUP(dim1, 'lookyloo').
+ // Determine if they are the same.
+ if (expectedVirtualColumn instanceof ExpressionVirtualColumn &&
+ LOOKUP_EXPRESSION.equals(((ExpressionVirtualColumn)
expectedVirtualColumn).getExpression())) {
+ // Only need one virtual column. GROUP BY will reuse the filter's
expectedVirtualColumn.
+ groupByDimension = expectedVirtualColumn.getOutputName();
+ virtualColumns = VirtualColumns.create(expectedVirtualColumn);
+ } else {
+ // Need both virtual columns: one for the filter, one for the GROUP BY.
+ groupByDimension = "v1";
+ virtualColumns = VirtualColumns.create(
+ expectedVirtualColumn,
+ expressionVirtualColumn(groupByDimension, LOOKUP_EXPRESSION,
ColumnType.STRING)
+ );
+ }
+ } else {
+ // The filter does not need its own virtual column.
+ groupByDimension = "v0";
+ virtualColumns = VirtualColumns.create(expressionVirtualColumn("v0",
LOOKUP_EXPRESSION, ColumnType.STRING));
+ }
+
return ImmutableList.of(
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Intervals.of("2000/3000")))
- .setVirtualColumns(expectedVirtualColumn != null
- ?
VirtualColumns.create(expectedVirtualColumn)
- : VirtualColumns.EMPTY)
+ .setVirtualColumns(virtualColumns)
.setGranularity(Granularities.ALL)
.setDimFilter(expectedFilter)
- .setDimensions(new ExtractionDimensionSpec("dim1", "d0",
ColumnType.STRING, EXTRACTION_FN))
+ .setDimensions(new DefaultDimensionSpec(groupByDimension,
"d0", ColumnType.STRING))
.setAggregatorSpecs(new CountAggregatorFactory("a0"))
.setContext(QUERY_CONTEXT)
.build()
diff --git
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java
index 8b65c5731ef..ae3a6a0eec3 100644
---
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java
+++
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java
@@ -20,11 +20,13 @@
package org.apache.druid.sql.calcite;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.java.util.common.granularity.Granularities;
import org.apache.druid.math.expr.ExpressionProcessing;
import org.apache.druid.query.Druids;
+import org.apache.druid.query.QueryContexts;
import org.apache.druid.query.aggregation.CountAggregatorFactory;
import org.apache.druid.query.aggregation.ExpressionLambdaAggregatorFactory;
import org.apache.druid.query.aggregation.FilteredAggregatorFactory;
@@ -47,6 +49,7 @@ import
org.apache.druid.segment.virtual.ListFilteredVirtualColumn;
import org.apache.druid.segment.virtual.PrefixFilteredVirtualColumn;
import org.apache.druid.segment.virtual.RegexFilteredVirtualColumn;
import org.apache.druid.sql.calcite.filtration.Filtration;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
import org.apache.druid.sql.calcite.util.CalciteTests;
import org.hamcrest.CoreMatchers;
import org.junit.internal.matchers.ThrowableMessageMatcher;
@@ -2117,6 +2120,43 @@ public class CalciteMultiValueStringQueryTest extends
BaseCalciteQueryTest
.sql("SELECT \n"
+ " MV_FILTER_ONLY(LOOKUP(dim3,'lookyloo'),ARRAY[null]),count(1)
\n"
+ "FROM druid.foo AS t group by 1\n")
+ .expectedQuery(
+ GroupByQuery.builder()
+ .setDataSource(CalciteTests.DATASOURCE1)
+ .setInterval(querySegmentSpec(Filtration.eternity()))
+ .setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "filter((x) -> array_contains(array(null), x),
lookup(\"dim3\",'lookyloo'))",
+ ColumnType.STRING
+ )
+ )
+ .setDimensions(dimensions(new
DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
+ .setAggregatorSpecs(aggregators(new
CountAggregatorFactory("a0")))
+ .setContext(QUERY_CONTEXT_DEFAULT)
+ .build())
+ .expectedResults(
+ ImmutableList.of(new Object[]{null, 7L}))
+ .run();
+
+ }
+
+ @Test
+ public void
testMultiValuedFilterOnlyWhenLookupPullsInDuplicatesWithExtractionFn()
+ {
+ cannotVectorize();
+
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
+ testBuilder()
+ .sql("SELECT \n"
+ + " MV_FILTER_ONLY(LOOKUP(dim3,'lookyloo'),ARRAY[null]),count(1)
\n"
+ + "FROM druid.foo AS t group by 1\n")
+ .queryContext(queryContext)
.expectedQuery(
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
@@ -2137,17 +2177,21 @@ public class CalciteMultiValueStringQueryTest extends
BaseCalciteQueryTest
)
.setDimensions(dimensions(new
DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
.setAggregatorSpecs(aggregators(new
CountAggregatorFactory("a0")))
- .setContext(QUERY_CONTEXT_DEFAULT)
+ .setContext(queryContext)
.build())
.expectedResults(
ImmutableList.of(new Object[]{null, 7L}))
.run();
-
}
@Test
public void testMvContainsFilterWithExtractionFn()
{
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
Druids.ScanQueryBuilder builder = newScanQueryBuilder()
.dataSource(CalciteTests.DATASOURCE3)
.intervals(querySegmentSpec(Filtration.eternity()))
@@ -2165,6 +2209,7 @@ public class CalciteMultiValueStringQueryTest extends
BaseCalciteQueryTest
);
testQuery(
"SELECT dim3 FROM druid.numfoo WHERE MV_CONTAINS(SUBSTRING(dim3, 1,
1), ARRAY['a','b']) LIMIT 5",
+ queryContext,
ImmutableList.of(builder.build()),
ImmutableList.of(
new Object[]{"[\"a\",\"b\"]"}
diff --git
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
index 628232d4464..f749ad0cc52 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
@@ -26,6 +26,7 @@ import com.google.common.collect.ImmutableSet;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.runtime.CalciteContextException;
import org.apache.druid.error.DruidException;
+import org.apache.druid.error.DruidExceptionMatcher;
import org.apache.druid.java.util.common.DateTimes;
import org.apache.druid.java.util.common.HumanReadableBytes;
import org.apache.druid.java.util.common.Intervals;
@@ -4172,24 +4173,25 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
public void testCountStarOnCommonTableExpression()
{
cannotVectorizeUnlessFallback();
- Druids.TimeseriesQueryBuilder builder =
- Druids.newTimeseriesQueryBuilder()
- .dataSource(CalciteTests.DATASOURCE1)
- .intervals(querySegmentSpec(Filtration.eternity()))
- .granularity(Granularities.ALL)
- .aggregators(aggregators(new CountAggregatorFactory("a0")))
- .context(QUERY_CONTEXT_DEFAULT)
- .virtualColumns(expressionVirtualColumn("v0",
"substring(\"dim1\", 0, 1)", ColumnType.STRING))
- .filters(
- and(
- equality("dim2", "a", ColumnType.STRING),
- not(equality("v0", "z", ColumnType.STRING))
- )
- );
testQuery(
"WITH beep (dim1_firstchar) AS (SELECT SUBSTRING(dim1, 1, 1) FROM foo
WHERE dim2 = 'a')\n"
+ "SELECT COUNT(*) FROM beep WHERE dim1_firstchar <> 'z'",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(expressionVirtualColumn("v0",
"substring(\"dim1\", 0, 1)", ColumnType.STRING))
+ .filters(
+ and(
+ equality("dim2", "a", ColumnType.STRING),
+ not(equality("v0", "z", ColumnType.STRING))
+ )
+ )
+ .aggregators(aggregators(new CountAggregatorFactory("a0")))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{1L}
)
@@ -4216,7 +4218,22 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
);
testQuery(
"SELECT COUNT(*) FROM view.aview WHERE dim1_firstchar <> 'z'",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(expressionVirtualColumn("v0",
"substring(\"dim1\", 0, 1)", ColumnType.STRING))
+ .filters(
+ and(
+ equality("dim2", "a", ColumnType.STRING),
+ not(equality("v0", "z", ColumnType.STRING))
+ )
+ )
+ .aggregators(aggregators(new CountAggregatorFactory("a0")))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{1L}
)
@@ -4245,7 +4262,24 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
);
testQuery(
"SELECT COUNT(*) FROM view.dview as druid WHERE druid.numfoo <> 'z'",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(
+ expressionVirtualColumn("v0", "substring(\"dim1\", 0,
1)", ColumnType.STRING)
+ )
+ .filters(
+ and(
+ equality("dim2", "a", ColumnType.STRING),
+ not(equality("v0", "z", ColumnType.STRING))
+ )
+ )
+ .aggregators(aggregators(new CountAggregatorFactory("a0")))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{1L}
)
@@ -7490,7 +7524,8 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
.intervals(querySegmentSpec(Filtration.eternity()))
.granularity(Granularities.ALL)
.virtualColumns(
- expressionVirtualColumn("v0",
"concat(substring(\"dim2\", 0, 1),'x')", ColumnType.STRING)
+ expressionVirtualColumn("v0", "substring(\"dim2\", 0,
1)", ColumnType.STRING),
+ expressionVirtualColumn("v1",
"concat(substring(\"dim2\", 0, 1),'x')", ColumnType.STRING)
)
.aggregators(
aggregators(
@@ -7498,7 +7533,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
new CardinalityAggregatorFactory(
"a1",
null,
- dimensions(new DefaultDimensionSpec("dim2",
"dim2")),
+ dimensions(DefaultDimensionSpec.of("dim2")),
false,
true
),
@@ -7506,7 +7541,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
new CardinalityAggregatorFactory(
"a2",
null,
- dimensions(new DefaultDimensionSpec("dim2",
"dim2")),
+ dimensions(DefaultDimensionSpec.of("dim2")),
false,
true
),
@@ -7515,21 +7550,14 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
new CardinalityAggregatorFactory(
"a3",
null,
- dimensions(
- new ExtractionDimensionSpec(
- "dim2",
- "dim2",
- ColumnType.STRING,
- new SubstringDimExtractionFn(0, 1)
- )
- ),
+ dimensions(DefaultDimensionSpec.of("v0")),
false,
true
),
new CardinalityAggregatorFactory(
"a4",
null,
- dimensions(new DefaultDimensionSpec("v0", "v0",
ColumnType.STRING)),
+ dimensions(DefaultDimensionSpec.of("v1")),
false,
true
),
@@ -8146,6 +8174,47 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
testQuery(
"SELECT COUNT(DISTINCT SUBSTRING(dim1, 1, 1)) FROM druid.foo WHERE
dim1 <> ''",
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .virtualColumns(expressionVirtualColumn("v0",
"substring(\"dim1\", 0, 1)", ColumnType.STRING))
+ .filters(not(equality("dim1", "", ColumnType.STRING)))
+ .granularity(Granularities.ALL)
+ .aggregators(
+ aggregators(
+ new CardinalityAggregatorFactory(
+ "a0",
+ null,
+ dimensions(new DefaultDimensionSpec("v0", "v0",
ColumnType.STRING)),
+ false,
+ true
+ )
+ )
+ )
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
+ ImmutableList.of(
+ new Object[]{4L}
+ )
+ );
+ }
+
+ @Test
+ public void testCountDistinctOfSubstringWithExtractionFn()
+ {
+ // Cannot vectorize due to extraction dimension spec.
+ cannotVectorize();
+
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
+ testQuery(
+ "SELECT COUNT(DISTINCT SUBSTRING(dim1, 1, 1)) FROM druid.foo WHERE
dim1 <> ''",
+ queryContext,
ImmutableList.of(
Druids.newTimeseriesQueryBuilder()
.dataSource(CalciteTests.DATASOURCE1)
@@ -8169,7 +8238,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
)
)
)
- .context(QUERY_CONTEXT_DEFAULT)
+ .context(queryContext)
.build()
),
ImmutableList.of(
@@ -8244,10 +8313,58 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
@Test
public void testRegexpExtract()
+ {
+ // Cannot vectorize due to regexp_extract function.
+ cannotVectorize();
+
+ testQuery(
+ "SELECT DISTINCT\n"
+ + " REGEXP_EXTRACT(dim1, '^.'),\n"
+ + " REGEXP_EXTRACT(dim1, '^(.)', 1)\n"
+ + "FROM foo\n"
+ + "WHERE REGEXP_EXTRACT(dim1, '^(.)', 1) <> 'x'",
+ ImmutableList.of(
+ GroupByQuery
+ .builder()
+ .setDataSource(CalciteTests.DATASOURCE1)
+ .setInterval(querySegmentSpec(Filtration.eternity()))
+ .setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn("v0",
"regexp_extract(\"dim1\",'^(.)',1)", ColumnType.STRING),
+ expressionVirtualColumn("v1",
"regexp_extract(\"dim1\",'^.')", ColumnType.STRING)
+ )
+ .setDimFilter(
+ not(equality("v0", "x", ColumnType.STRING))
+ )
+ .setDimensions(
+ dimensions(
+ new DefaultDimensionSpec("v1", "d0"),
+ new DefaultDimensionSpec("v0", "d1")
+ )
+ )
+ .setContext(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
+ ImmutableList.of(
+ new Object[]{"1", "1"},
+ new Object[]{"2", "2"},
+ new Object[]{"a", "a"},
+ new Object[]{"d", "d"}
+ )
+ );
+ }
+
+ @Test
+ public void testRegexpExtractWithExtractionFn()
{
// Cannot vectorize due to extractionFn in dimension spec.
cannotVectorize();
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
GroupByQuery.Builder builder =
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
@@ -8281,6 +8398,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
+ " REGEXP_EXTRACT(dim1, '^(.)', 1)\n"
+ "FROM foo\n"
+ "WHERE REGEXP_EXTRACT(dim1, '^(.)', 1) <> 'x'",
+ queryContext,
ImmutableList.of(builder.build()),
ImmutableList.of(
new Object[]{"1", "1"},
@@ -8301,6 +8419,28 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
"SELECT DISTINCT\n"
+ " REGEXP_EXTRACT(dim1, '^(.))', 1)\n"
+ "FROM foo",
+ DruidExceptionMatcher.invalidInput().expectMessageContains(
+ "An invalid pattern [^(.))] was provided for the regexp_extract
function, " +
+ "error: [Unmatched closing ')' near index 3\n^(.))\n ^]"
+ )
+ );
+ }
+
+ @Test
+ public void testRegexpExtractWithBadRegexPatternWithExtractionFn()
+ {
+ // Cannot vectorize due to extractionFn in dimension spec.
+ cannotVectorize();
+
+ testQueryThrows(
+ "SELECT DISTINCT\n"
+ + " REGEXP_EXTRACT(dim1, '^(.))', 1)\n"
+ + "FROM foo",
+ QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ ),
+ DruidException.class,
invalidSqlContains(
"An invalid pattern [^(.))] was provided for the REGEXP_EXTRACT
function, " +
"error: [Unmatched closing ')' near index 3\n^(.))\n ^]"
@@ -8333,7 +8473,25 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
"SELECT COUNT(*)\n"
+ "FROM foo\n"
+ "WHERE REGEXP_EXTRACT(dim1, '^1') IS NOT NULL OR REGEXP_EXTRACT('Z'
|| dim1, '^Z2') IS NOT NULL",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(
+ expressionVirtualColumn("v0",
"regexp_extract(\"dim1\",'^1')", ColumnType.STRING),
+ expressionVirtualColumn("v1",
"regexp_extract(concat('Z',\"dim1\"),'^Z2')", ColumnType.STRING)
+ )
+ .filters(
+ or(
+ notNull("v0"),
+ notNull("v1")
+ )
+ )
+ .aggregators(new CountAggregatorFactory("a0"))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{3L}
)
@@ -8474,13 +8632,16 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE3)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn("v0", "'a'",
ColumnType.STRING),
+ expressionVirtualColumn("v1", "substring(\"dim5\",
0, 1)", ColumnType.STRING)
+ )
.setDimensions(
dimensions(
new DefaultDimensionSpec("v0", "d0"),
- new ExtractionDimensionSpec("dim5", "d1", new
SubstringDimExtractionFn(0, 1))
+ new DefaultDimensionSpec("v1", "d1")
)
)
- .setVirtualColumns(expressionVirtualColumn("v0",
"'a'", ColumnType.STRING))
.setDimFilter(equality("dim4", "a", ColumnType.STRING))
.setAggregatorSpecs(
aggregators(
@@ -9210,15 +9371,8 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
.setDataSource(new LookupDataSource("lookyloo"))
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
- .setDimensions(
- dimensions(
- new ExtractionDimensionSpec(
- "v",
- "d0",
- new SubstringDimExtractionFn(0, 1)
- )
- )
- )
+ .setVirtualColumns(expressionVirtualColumn("v0",
"substring(\"v\", 0, 1)", ColumnType.STRING))
+ .setDimensions(dimensions(new
DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
.setAggregatorSpecs(aggregators(new
CountAggregatorFactory("a0")))
.setContext(QUERY_CONTEXT_DEFAULT)
.build()
@@ -12526,7 +12680,7 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
@Test
public void testRequireTimeConditionPositive3()
{
- // Cannot vectorize next test due to extraction dimension spec.
+ // Cannot vectorize next test due to substring function.
cannotVectorize();
// semi-join requires time condition on both left and right query
@@ -12556,14 +12710,14 @@ public class CalciteQueryTest extends
BaseCalciteQueryTest
)
.setDimFilter(not(equality("dim1",
"", ColumnType.STRING)))
.setGranularity(Granularities.ALL)
- .setDimensions(
- new ExtractionDimensionSpec(
- "dim1",
- "d0",
- ColumnType.STRING,
- new
SubstringDimExtractionFn(0, 1)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "substring(\"dim1\", 0, 1)",
+ ColumnType.STRING
)
)
+ .setDimensions(new
DefaultDimensionSpec("v0", "d0", ColumnType.STRING))
.setContext(QUERY_CONTEXT_DEFAULT)
.build()
),
diff --git
a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java
b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java
index c13dcdab62f..05bd18f0be2 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java
@@ -54,9 +54,7 @@ import
org.apache.druid.query.aggregation.firstlast.first.StringFirstAggregatorF
import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
import org.apache.druid.query.dimension.DefaultDimensionSpec;
-import org.apache.druid.query.dimension.ExtractionDimensionSpec;
import org.apache.druid.query.expression.TestExprMacroTable;
-import org.apache.druid.query.extraction.SubstringDimExtractionFn;
import org.apache.druid.query.filter.DimFilter;
import org.apache.druid.query.filter.TypedInFilter;
import org.apache.druid.query.groupby.GroupByQuery;
@@ -253,14 +251,17 @@ public class CalciteSubqueryTest extends
BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+
expressionVirtualColumn(
+ "v0",
+
"substring(\"dim1\", 0, 1)",
+
ColumnType.STRING
+ )
+ )
.setDimFilter(not(equality("dim1", "", ColumnType.STRING)))
.setDimensions(
dimensions(
- new
ExtractionDimensionSpec(
- "dim1",
- "d0",
- new
SubstringDimExtractionFn(0, 1)
- )
+ new
DefaultDimensionSpec("v0", "d0")
)
)
.setContext(QUERY_CONTEXT_DEFAULT)
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq
index 742859de627..71f75680c4c 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq
index 060c6cebfe6..af1cc7c9708 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
index 141df36c597..fb972a91be2 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
@@ -81,6 +81,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -94,15 +100,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq
index 5fbb15e9710..4e553b80ca3 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
index ea1ef086fa6..85cb83409b6 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
index 072f68d2bf8..0f664b798a6 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
index a51c482ab8c..4e41f33603b 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/[email protected]
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq
index 665cf5ef26b..147ef51b6be 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq
@@ -44,6 +44,12 @@ DruidProject(dim4=[CAST('a':VARCHAR):VARCHAR], EXPR$1=[$0],
EXPR$2=[$1], druid=[
"type" : "intervals",
"intervals" : [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim5\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "equals",
"column" : "dim4",
@@ -54,15 +60,10 @@ DruidProject(dim4=[CAST('a':VARCHAR):VARCHAR], EXPR$1=[$0],
EXPR$2=[$1], druid=[
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim5",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"aggregations" : [ {
"type" : "count",
diff --git
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq
index 487561db05c..9d180c80b7c 100644
---
a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq
+++
b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq
@@ -87,6 +87,12 @@ DruidAggregate(group=[{}], EXPR$0=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [
"2000-01-01T00:00:00.000Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -100,15 +106,10 @@ DruidAggregate(group=[{}], EXPR$0=[COUNT()],
druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
