This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new a523b7beae2 Expose common segment metadata as built-in virtual columns
(#19179)
a523b7beae2 is described below
commit a523b7beae27a25c4e18f43ccac38921b3e894ce
Author: Xiang Fu <[email protected]>
AuthorDate: Mon Aug 10 01:08:58 2026 -0700
Expose common segment metadata as built-in virtual columns (#19179)
* Expose common segment metadata as built-in virtual columns
Adds five built-in virtual columns that surface segment metadata to
queries, alongside the existing $docId / $hostName / $segmentName /
$partitionId:
| Column | Type | Value |
|-----------------|--------|----------------------------------------------|
| $creationTime | LONG | segment creation time, epoch millis |
| $startTimeMs | LONG | segment time range start, epoch millis |
| $endTimeMs | LONG | segment time range end, epoch millis |
| $totalDocs | INT | documents stored in the segment |
| $segmentCrc | STRING | segment CRC |
Example:
SELECT $segmentName, $segmentCrc, $totalDocs, $creationTime
FROM myTable GROUP BY 1, 2, 3, 4
Each column is a constant single-value column within a segment. Values
are read from SegmentMetadata every time the column is built rather than
baked into the field spec, so mutable segments - which rebuild their
virtual data sources on every access - always observe current metadata.
Metadata that genuinely does not exist yet reads as SQL NULL rather than
a sentinel: a CONSUMING segment has no time range and no CRC until it is
committed. This required teaching the virtual column path to carry a null
value vector - VirtualColumnIndexContainer now serves
StandardIndexes.nullValueVector() and VirtualColumnProvider gained a
buildNullValueVector hook (defaulting to no nulls, so existing providers
are unaffected). Without it the placeholder stored in the forward index
would be indistinguishable from a real value once null handling is on,
e.g. MIN($startTimeMs) returning Long.MIN_VALUE on a consuming segment.
$totalDocs counts the documents physically stored in the segment, so for
an upsert table it also includes documents that have been replaced and
are no longer returned by queries.
Naming note: only the time range columns carry the "Ms" suffix.
SegmentMetadata#getStartTime()/#getEndTime() return values in the time
column's own unit, so unsuffixed names would be ambiguous; a creation
time is always epoch millis throughout Pinot.
Supporting changes:
- New BuiltInVirtualColumns in pinot-spi is the single source of each
column's name, data type and single-value/multi-value shape. The broker
side (TableCache#addBuiltInVirtualColumns) and the server side
(VirtualColumnProviderFactory#addBuiltInVirtualColumnsToSegmentSchema)
now both build their field specs from it, so the two can no longer
disagree on a type. Previously these specs were declared twice by hand.
- DefaultNullValueVirtualColumnProvider grew an overridable
getValue(context) so it can back any per-segment constant column, plus
a type check that names the offending column and provider instead of
throwing a bare ClassCastException from inside segment loading.
Behavior is unchanged for existing callers.
- Fixes a pre-existing bug in SchemaInfo, which computed
getDimensionFieldSpecs().size() - 3 with a comment naming three virtual
columns. That has been off by one since $partitionId was added, and
these five would have made GET /schemas/info over-report user dimension
counts by six in the controller UI. It now excludes built-in virtual
columns by name.
* Use TIMESTAMP for the segment time columns, and address review feedback
Follow-up to the segment metadata virtual columns, covering the review
comments and the CI failures on the first revision.
Column types and names
----------------------
The three time columns are now TIMESTAMP rather than LONG, so the unit is
carried by the type instead of by the column name and query results render
them as readable timestamps instead of raw millis. That removes the reason
the "Ms" suffix existed, so the columns are back to their natural names:
$creationTime TIMESTAMP
$startTime TIMESTAMP
$endTime TIMESTAMP
$totalDocs INT
$crc STRING
$segmentCrc is renamed to $crc so that the five names are consistent: none
of them repeats the "segment" scope, which every one of them shares.
Correctness
-----------
The segment metadata used to be read three separate times per build - once
for the dictionary, once for the column metadata, and once for the null
value vector - with nothing tying the three reads together. On a mutable
segment, whose metadata is live, they could disagree and the column would
then serve its placeholder as if it were a real value. The value is now
resolved exactly once per build and shared by all three.
VirtualColumnIndexContainer#close now also closes the null value vector.
Structure
---------
- The constant-value machinery moves out of
DefaultNullValueVirtualColumnProvider
into a new BaseConstantValueVirtualColumnProvider in the virtualcolumn
package. DefaultNullValueVirtualColumnProvider becomes a thin subclass;
its
fully-qualified name is stored in field specs and resolved reflectively,
so
it keeps its name and package.
- VirtualColumnProviderFactory resolves providers through a map whose key
set
is checked against the column definitions in a static initializer. A
column
added without a provider now fails at class load instead of aborting every
segment load on every server.
- BuiltInVirtualColumns is renamed to BuiltInVirtualColumnDefinitions: the
old
name differed from the pre-existing BuiltInVirtualColumn by a single
trailing
letter, and VirtualColumnProviderFactory imports both.
Test fixes
----------
Calcite plans reference fields positionally, and all $-prefixed columns sort
first in a table's row type, so five new virtual columns shift every ordinal
in an EXPLAIN assertion by five. Updated in NullHandlingIntegrationTest,
MultiStageEngineExplainIntegrationTest and OfflineClusterIntegrationTest.
MultiNodesOfflineClusterIntegrationTest inherits the latter's fixes.
The aggregate metadata API reports one entry per column of a loaded segment,
which includes the virtual columns, so its expected count grows from 83 to
88.
Test coverage
-------------
- BuiltInVirtualColumnDefinitionsTest pins the definitions against the
declared names, and pins that field specs are never shared between
schemas.
- A negative test covers the value type check.
- The CONSUMING-segment queries in BaseClusterIntegrationTestSet now assert
their results instead of only checking that nothing throws. On a hybrid
table the time boundary can hide rows that are physically present in a
segment, so the assertion is that a segment's $totalDocs is at least the
number of rows the query returns from it.
- OfflineClusterIntegrationTest additionally checks the millisecond
normalization per segment, filtering by $crc, and that null handling sees
no nulls on a table where all the metadata is available.
* Address review: suppress min/max for NULL columns, expose $crc as LONG
Fixes found by review on top of the segment metadata virtual columns.
Do not publish min/max for a column that reads as NULL
------------------------------------------------------
When the segment metadata is unavailable the column stores a placeholder
and reports every document as null, but it was still publishing that
placeholder as the column's min/max. Segment pruners read min/max without
consulting the null value vector, so:
SET enableNullHandling=true;
SELECT $segmentName, $creationTime FROM myTable ORDER BY $creationTime
ASC LIMIT 10
let a CONSUMING segment - whose $creationTime placeholder is the epoch -
sort first in SelectionQuerySegmentPruner, consume the whole LIMIT, and
prune away every committed segment whose min was greater. The query then
returned rows that are all SQL NULL, which under NULLS-LAST ordering should
have sorted last. $crc had the mirror problem for ORDER BY ... DESC.
The min/max are now left unset whenever the value is a placeholder. Both
SelectionQuerySegmentPruner and ColumnValueSegmentPruner already keep a
segment that reports no min/max, which is the correct conservative
behavior for an all-null column.
Resolve the metadata once per data source, not once per index
-------------------------------------------------------------
buildDataSource calls buildMetadata and buildColumnIndexContainer
separately, so the column metadata came from a different read of
SegmentMetadata than the dictionary and the null value vector. Only the
container was internally consistent.
BaseSegmentMetadataVirtualColumnProvider
now overrides buildDataSource and resolves the value once for all three.
Apply the value type check on the path that is actually taken
------------------------------------------------------------
The check was only reachable through the no-argument buildDictionary /
buildMetadata. Every one of the five new providers goes through the
value-taking overloads, which skipped it, so the diagnostic it exists to
give never applied to them. The check now runs on every path.
$crc is a LONG
--------------
A CRC is a long everywhere else in Pinot - SegmentZKMetadata#getCrc returns
one, and SegmentMetadata#getCrc merely renders that long as a String. Typing
the column STRING to match the latter left users unable to write
WHERE $crc = 12345, and contradicted the reasoning used for the time
columns, which are TIMESTAMP precisely so the type carries the semantics.
Other review fixes
------------------
- AllNullValueVector was a byte-for-byte duplicate of a nested class in
OpenStructNullDataSource. Both now share
AllNullValueVectorReader in the readers package.
- SchemaUtils.validate rejects a user column named after a built-in virtual
column. Such a column is filtered out of the segment's physical columns
when the metadata is read, and the virtual provider then takes over the
name, so queries would silently return segment metadata instead of the
user's data.
- The built-in providers are stateless, so they are now shared instances
rather than reflectively constructed once per virtual column per segment.
Segment load did that 9 times per segment after this feature, up from 4.
- Removed the static initializer asserting provider coverage. The factory is
only ever loaded during segment load, so it did not fail any earlier than
the per-column check, and it turned every subsequent load into a bare
NoClassDefFoundError. The invariant is now asserted in a test.
- Corrected the AllNullValueVectorReader publication comment:
toImmutableRoaringBitmap() returns `this`, so safety rests on the volatile
write alone and the returned bitmap must be treated as read-only.
- Imported SegmentMetadata rather than naming it inline in Javadoc.
Naming note: the reviewers proposed prefixing all five columns with
"segment" ($segmentCreationTime, ...) to match $segmentName. The terse forms
are a deliberate choice, recorded here so it is not re-litigated as an
oversight.
---
.../pinot/common/config/provider/TableCache.java | 17 +-
.../PinotQueryResourceStaticValidationTest.java | 5 +-
.../pinot/controller/helix/TableCacheTest.java | 12 +-
.../tests/BaseClusterIntegrationTestSet.java | 40 ++
.../MultiStageEngineExplainIntegrationTest.java | 6 +-
.../tests/NullHandlingIntegrationTest.java | 8 +-
.../tests/OfflineClusterIntegrationTest.java | 82 ++++-
.../DefaultNullValueVirtualColumnProvider.java | 118 +-----
.../index/openstruct/OpenStructNullDataSource.java | 29 +-
.../index/readers/AllNullValueVectorReader.java | 60 +++
.../BaseConstantValueVirtualColumnProvider.java | 220 +++++++++++
.../BaseSegmentMetadataVirtualColumnProvider.java | 97 +++++
.../SegmentCrcVirtualColumnProvider.java | 49 +++
.../SegmentCreationTimeVirtualColumnProvider.java | 38 ++
.../SegmentEndTimeVirtualColumnProvider.java | 38 ++
.../SegmentStartTimeVirtualColumnProvider.java | 38 ++
.../SegmentTotalDocsVirtualColumnProvider.java | 35 ++
.../virtualcolumn/VirtualColumnIndexContainer.java | 18 +
.../virtualcolumn/VirtualColumnProvider.java | 14 +-
.../VirtualColumnProviderFactory.java | 86 ++++-
.../pinot/segment/local/utils/SchemaUtils.java | 7 +
.../mutable/MutableSegmentImplRawMVTest.java | 13 +-
.../mutable/MutableSegmentImplTest.java | 37 +-
.../local/segment/index/loader/LoaderTest.java | 38 +-
.../SegmentMetadataVirtualColumnProviderTest.java | 405 +++++++++++++++++++++
.../spi/data/BuiltInVirtualColumnDefinitions.java | 107 ++++++
.../java/org/apache/pinot/spi/data/SchemaInfo.java | 13 +-
.../apache/pinot/spi/utils/CommonConstants.java | 31 +-
.../data/BuiltInVirtualColumnDefinitionsTest.java | 99 +++++
.../org/apache/pinot/spi/data/SchemaInfoTest.java | 24 ++
30 files changed, 1590 insertions(+), 194 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java
b/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java
index 02c5e7cf313..2e62e80909d 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java
@@ -31,11 +31,9 @@ import
org.apache.pinot.spi.config.provider.SchemaChangeListener;
import org.apache.pinot.spi.config.provider.TableConfigChangeListener;
import org.apache.pinot.spi.config.table.QueryConfig;
import org.apache.pinot.spi.config.table.TableConfig;
-import org.apache.pinot.spi.data.DimensionFieldSpec;
-import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
import org.apache.pinot.spi.data.LogicalTableConfig;
import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
import org.apache.pinot.spi.utils.TimestampIndexUtils;
import org.apache.pinot.sql.parsers.CalciteSqlParser;
import org.slf4j.Logger;
@@ -119,18 +117,7 @@ public interface TableCache extends PinotConfigProvider {
/// Adds the built-in virtual columns to the schema.
/// NOTE: The virtual column provider class is not added.
default void addBuiltInVirtualColumns(Schema schema) {
- if (!schema.hasColumn(BuiltInVirtualColumn.DOCID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.DOCID,
FieldSpec.DataType.INT, true));
- }
- if (!schema.hasColumn(BuiltInVirtualColumn.HOSTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.HOSTNAME,
FieldSpec.DataType.STRING, true));
- }
- if (!schema.hasColumn(BuiltInVirtualColumn.SEGMENTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.SEGMENTNAME,
FieldSpec.DataType.STRING, true));
- }
- if (!schema.hasColumn(BuiltInVirtualColumn.PARTITIONID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.PARTITIONID,
FieldSpec.DataType.STRING, false));
- }
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
}
static Map<Expression, Expression> createExpressionOverrideMap(String
physicalOrLogicalTableName,
diff --git
a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java
b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java
index 45dde807c67..0bd7222cb1c 100644
---
a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java
+++
b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java
@@ -26,6 +26,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.mockito.MockitoAnnotations;
import org.testng.Assert;
@@ -62,7 +63,9 @@ public class PinotQueryResourceStaticValidationTest {
Assert.assertNotNull(provider.getTableConfig("testTable_OFFLINE"));
Assert.assertNotNull(provider.getSchema("testTable"));
Assert.assertNotNull(provider.getColumnNameMap("testTable"));
- Assert.assertEquals(provider.getColumnNameMap("testTable").size(), 6); //
2 columns + 4 built-in virtual columns
+ // 2 columns + all built-in virtual columns
+ Assert.assertEquals(provider.getColumnNameMap("testTable").size(),
+ 2 +
CommonConstants.Segment.BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS.size());
Assert.assertTrue(provider.getTableNameMap().containsKey("testTable_OFFLINE"));
Assert.assertTrue(provider.getTableNameMap().containsKey("testTable"));
diff --git
a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java
index 9eeff07c69f..ca1e4861512 100644
---
a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java
+++
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java
@@ -303,6 +303,11 @@ public class TableCacheTest {
expectedColumnMap.put(isCaseInsensitive ? "$hostname" : "$hostName",
"$hostName");
expectedColumnMap.put(isCaseInsensitive ? "$segmentname" : "$segmentName",
"$segmentName");
expectedColumnMap.put(isCaseInsensitive ? "$partitionid" : "$partitionId",
"$partitionId");
+ expectedColumnMap.put(isCaseInsensitive ? "$creationtime" :
"$creationTime", "$creationTime");
+ expectedColumnMap.put(isCaseInsensitive ? "$starttime" : "$startTime",
"$startTime");
+ expectedColumnMap.put(isCaseInsensitive ? "$endtime" : "$endTime",
"$endTime");
+ expectedColumnMap.put(isCaseInsensitive ? "$totaldocs" : "$totalDocs",
"$totalDocs");
+ expectedColumnMap.put("$crc", "$crc");
return expectedColumnMap;
}
@@ -311,7 +316,12 @@ public class TableCacheTest {
.addSingleValueDimension(BuiltInVirtualColumn.DOCID, DataType.INT)
.addSingleValueDimension(BuiltInVirtualColumn.HOSTNAME,
DataType.STRING)
.addSingleValueDimension(BuiltInVirtualColumn.SEGMENTNAME,
DataType.STRING)
- .addMultiValueDimension(BuiltInVirtualColumn.PARTITIONID,
DataType.STRING).build();
+ .addMultiValueDimension(BuiltInVirtualColumn.PARTITIONID,
DataType.STRING)
+ .addSingleValueDimension(BuiltInVirtualColumn.CREATIONTIME,
DataType.TIMESTAMP)
+ .addSingleValueDimension(BuiltInVirtualColumn.STARTTIME,
DataType.TIMESTAMP)
+ .addSingleValueDimension(BuiltInVirtualColumn.ENDTIME,
DataType.TIMESTAMP)
+ .addSingleValueDimension(BuiltInVirtualColumn.TOTALDOCS, DataType.INT)
+ .addSingleValueDimension(BuiltInVirtualColumn.CRC,
DataType.LONG).build();
}
@DataProvider(name = "testTableCacheDataProvider")
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
index 8a52029a2ab..d13a9723214 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
@@ -22,6 +22,7 @@ import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
+import java.sql.Timestamp;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
@@ -465,6 +466,45 @@ public abstract class BaseClusterIntegrationTestSet
extends BaseClusterIntegrati
"select $docId, $segmentName, $hostName, $partitionId from mytable
where $docId = 5 limit 50");
getPinotConnection().execute(
"select $docId, $segmentName, $hostName, $partitionId from mytable
where $docId > 19998 limit 50");
+
+ // Segment metadata virtual columns. This method is the only place they
are exercised against a table that can
+ // have CONSUMING segments, so assert the results rather than just
checking that nothing throws.
+ long numTotalDocs = getCountStarResult();
+
+ // $totalDocs is the number of documents stored in the segment. On a
hybrid table the broker's time boundary can
+ // hide some of them, so a segment's $totalDocs is at least the number of
rows the query sees from it, never less.
+ ResultSet perSegment = getPinotConnection().execute(
+ "select $segmentName, max($totalDocs), count(*) from mytable group
by $segmentName limit 10000")
+ .getResultSet(0);
+ assertTrue(perSegment.getRowCount() > 0);
+ long visibleRows = 0;
+ for (int i = 0; i < perSegment.getRowCount(); i++) {
+ String segmentName = perSegment.getString(i, 0);
+ // MAX()/COUNT() render as floating point values in the single-stage
engine, so read them as doubles in both
+ long totalDocsInSegment = (long)
Double.parseDouble(perSegment.getString(i, 1));
+ long rowsFromSegment = (long) Double.parseDouble(perSegment.getString(i,
2));
+ assertTrue(totalDocsInSegment > 0, "Unexpected $totalDocs: " +
totalDocsInSegment + " for: " + segmentName);
+ assertTrue(totalDocsInSegment >= rowsFromSegment,
+ "$totalDocs: " + totalDocsInSegment + " is below the " +
rowsFromSegment + " rows returned by: "
+ + segmentName);
+ visibleRows += rowsFromSegment;
+ }
+ assertEquals(visibleRows, numTotalDocs, "Grouping by $segmentName should
cover every row exactly once");
+
+ // Every segment - CONSUMING included - is created with a creation time,
so none of them falls back to the epoch
+ // placeholder used when the metadata is unavailable
+ ResultSet creationTimes = getPinotConnection()
+ .execute("select $segmentName, $creationTime from mytable group by
$segmentName, $creationTime limit 10000")
+ .getResultSet(0);
+ assertTrue(creationTimes.getRowCount() > 0);
+ for (int i = 0; i < creationTimes.getRowCount(); i++) {
+ String creationTime = creationTimes.getString(i, 1);
+ assertTrue(Timestamp.valueOf(creationTime).getTime() > 0,
+ "Unexpected $creationTime: " + creationTime + " for segment: " +
creationTimes.getString(i, 0));
+ }
+
+ // Selecting them must not fail on any segment type
+ getPinotConnection().execute("select $creationTime, $startTime, $endTime,
$totalDocs, $crc from mytable limit 50");
}
/// Test queries from the query file.
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java
index 69d9861a5b4..1873fe8d1db 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java
@@ -178,7 +178,7 @@ public class MultiStageEngineExplainIntegrationTest extends
BaseClusterIntegrati
"Execution Plan\n"
+ "PinotLogicalAggregate(group=[{}], agg#0=[SUM($0)],
aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash])\n"
- + " PinotLogicalAggregate(group=[{}], agg#0=[SUM($6)],
aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{}], agg#0=[SUM($11)],
aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n");
// Enable PinotAggregateFunctionRewriteRule through query option, ensure
it overrides the broker config
@@ -186,7 +186,7 @@ public class MultiStageEngineExplainIntegrationTest extends
BaseClusterIntegrati
"Execution Plan\n"
+ "PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($0)],
aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash])\n"
- + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($6)],
aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($11)],
aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of("usePlannerRules",
CommonConstants.Broker.PlannerRuleNames.AGGREGATE_FUNCTION_REWRITE));
@@ -197,7 +197,7 @@ public class MultiStageEngineExplainIntegrationTest extends
BaseClusterIntegrati
+ "LogicalProject(EXPR$0=[CASE(=($1, 0), null:BIGINT, $0)])\n"
+ " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($0)],
agg#1=[COUNT($1)], aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash])\n"
- + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($6)],
agg#1=[COUNT()], aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($11)],
agg#1=[COUNT()], aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of("usePlannerRules",
CommonConstants.Broker.PlannerRuleNames.AGGREGATE_FUNCTION_REWRITE + ","
+
CommonConstants.Broker.PlannerRuleNames.AGGREGATE_REDUCE_FUNCTIONS));
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java
index 8388fc46722..c158d6913cf 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java
@@ -434,7 +434,7 @@ public class NullHandlingIntegrationTest extends
BaseClusterIntegrationTestSet
explainLogical(query,
"Execution Plan\n"
+ "LogicalProject(EXPR$0=[1])\n"
- + " LogicalFilter(condition=[AND(IS NOT NULL($8), <>($8, 0))])\n"
+ + " LogicalFilter(condition=[AND(IS NOT NULL($13), <>($13,
0))])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of(CommonConstants.Broker.Request.QueryOptionKey.ENABLE_NULL_HANDLING,
"false"));
}
@@ -452,7 +452,7 @@ public class NullHandlingIntegrationTest extends
BaseClusterIntegrationTestSet
explainLogical(query,
"Execution Plan\n"
+ "LogicalProject(EXPR$0=[1])\n"
- + " LogicalFilter(condition=[<>($8, 0)])\n"
+ + " LogicalFilter(condition=[<>($13, 0)])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of(CommonConstants.Broker.Request.QueryOptionKey.ENABLE_NULL_HANDLING,
"true"));
}
@@ -470,7 +470,7 @@ public class NullHandlingIntegrationTest extends
BaseClusterIntegrationTestSet
explainLogical(query,
"Execution Plan\n"
+ "LogicalProject(EXPR$0=[1])\n"
- + " LogicalFilter(condition=[AND(IS NULL($8), <>($8, 0))])\n"
+ + " LogicalFilter(condition=[AND(IS NULL($13), <>($13, 0))])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of(CommonConstants.Broker.Request.QueryOptionKey.ENABLE_NULL_HANDLING,
"false"));
}
@@ -537,7 +537,7 @@ public class NullHandlingIntegrationTest extends
BaseClusterIntegrationTestSet
+ "PinotLogicalAggregate(group=[{0}], agg#0=[COUNT($1)],
agg#1=[COUNT($2)], aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash[0]])\n"
+ " PinotLogicalAggregate(group=[{0}], agg#0=[COUNT()],
agg#1=[COUNT() FILTER $1], aggType=[LEAF])\n"
- + " LogicalProject(city=[$5], $f1=[IS TRUE(=($7,
_UTF-8'unknown'))])\n"
+ + " LogicalProject(city=[$10], $f1=[IS TRUE(=($12,
_UTF-8'unknown'))])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n");
// IS_TRUE should be trimmed off, then the filter becomes always false
in the server execution plan
explainAskingServers(query,
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
index 8984ca7027f..6872360a874 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
@@ -2449,6 +2449,78 @@ public class OfflineClusterIntegrationTest extends
BaseClusterIntegrationTestSet
assertEquals(response.get("numSegmentsMatched").asInt(),
numSegmentsToQuery);
}
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testSegmentMetadataVirtualColumns(boolean
useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+
+ String query = "SELECT $creationTime, $startTime, $endTime, $totalDocs,
$crc FROM mytable LIMIT 10";
+ JsonNode response = postQuery(query);
+ JsonNode resultTable = response.get("resultTable");
+ JsonNode dataSchema = resultTable.get("dataSchema");
+ assertEquals(dataSchema.get("columnNames").toString(),
+
"[\"$creationTime\",\"$startTime\",\"$endTime\",\"$totalDocs\",\"$crc\"]");
+ // The three time columns are TIMESTAMP, so they render as formatted
timestamps rather than raw millis
+ assertEquals(dataSchema.get("columnDataTypes").toString(),
+ "[\"TIMESTAMP\",\"TIMESTAMP\",\"TIMESTAMP\",\"INT\",\"LONG\"]");
+ JsonNode rows = resultTable.get("rows");
+ assertEquals(rows.size(), 10);
+ for (int i = 0; i < 10; i++) {
+ JsonNode row = rows.get(i);
+ long creationTime = Timestamp.valueOf(row.get(0).asText()).getTime();
+ assertTrue(creationTime > 0, "Unexpected segment creation time: " +
row.get(0).asText());
+ long startTime = Timestamp.valueOf(row.get(1).asText()).getTime();
+ long endTime = Timestamp.valueOf(row.get(2).asText()).getTime();
+ assertTrue(startTime > 0, "Unexpected segment start time: " +
row.get(1).asText());
+ assertTrue(startTime <= endTime, "Segment start time: " + startTime + "
is after end time: " + endTime);
+ assertTrue(row.get(3).asInt() > 0, "Unexpected total docs: " +
row.get(3).asInt());
+ // A real CRC, not the placeholder the column falls back to when the
metadata is unavailable
+ long crc = row.get(4).asLong();
+ assertNotEquals(crc, (long)
FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG, "Segment CRC should be a real
CRC");
+ }
+
+ // $totalDocs is constant within a segment, so the per-segment values must
add up to the total number of rows
+ query = "SELECT $segmentName, MAX($totalDocs) FROM mytable GROUP BY
$segmentName LIMIT 10000";
+ rows = postQuery(query).get("resultTable").get("rows");
+ long totalDocs = 0;
+ for (JsonNode row : rows) {
+ totalDocs += row.get(1).asLong();
+ }
+ assertEquals(totalDocs, getCountStarResult());
+
+ // $startTime/$endTime are normalized from the time column's own unit.
mytable's time column is DaysSinceEpoch in
+ // DAYS, so each segment's values must be its own day boundaries expressed
as timestamps - this is what
+ // distinguishes a correct implementation from one leaking the raw DAYS
value.
+ query = "SELECT $segmentName, $startTime, $endTime, MIN(DaysSinceEpoch),
MAX(DaysSinceEpoch) FROM mytable "
+ + "GROUP BY $segmentName, $startTime, $endTime LIMIT 10000";
+ rows = postQuery(query).get("resultTable").get("rows");
+ assertFalse(rows.isEmpty());
+ long daysToMillis = TimeUnit.DAYS.toMillis(1);
+ for (JsonNode row : rows) {
+ assertEquals(Timestamp.valueOf(row.get(1).asText()).getTime(),
row.get(3).asLong() * daysToMillis,
+ "Unexpected $startTime for segment: " + row.get(0).asText());
+ assertEquals(Timestamp.valueOf(row.get(2).asText()).getTime(),
row.get(4).asLong() * daysToMillis,
+ "Unexpected $endTime for segment: " + row.get(0).asText());
+ }
+
+ // Filtering on a single segment's CRC must return exactly the documents
grouped under that CRC
+ query = "SELECT $crc, COUNT(*) FROM mytable GROUP BY $crc LIMIT 10000";
+ JsonNode crcRow = postQuery(query).get("resultTable").get("rows").get(0);
+ long segmentCrc = crcRow.get(0).asLong();
+ long docsForCrc = crcRow.get(1).asLong();
+ assertEquals(postQuery("SELECT COUNT(*) FROM mytable WHERE $crc = " +
segmentCrc).get("resultTable")
+ .get("rows").get(0).get(0).asLong(), docsForCrc);
+
+ // With null handling on, the engine must consult the null value vector.
Every segment of this table has a time
+ // range, a CRC and a creation time, so none of these columns may report a
null.
+ for (String column : List.of("$creationTime", "$startTime", "$endTime",
"$totalDocs", "$crc")) {
+ assertEquals(postQuery("SET enableNullHandling=true; SELECT COUNT(*)
FROM mytable WHERE " + column
+ + " IS NOT
NULL").get("resultTable").get("rows").get(0).get(0).asLong(),
getCountStarResult(), column);
+ assertEquals(postQuery("SET enableNullHandling=true; SELECT COUNT(*)
FROM mytable WHERE " + column
+ + " IS NULL").get("resultTable").get("rows").get(0).get(0).asLong(),
0, column);
+ }
+ }
+
@Test(dataProvider = "useBothQueryEngines")
public void testGroupByUDF(boolean useMultiStageQueryEngine)
throws Exception {
@@ -3581,7 +3653,7 @@ public class OfflineClusterIntegrationTest extends
BaseClusterIntegrationTestSet
+ " LogicalProject(count=[$1], name=[$0])\n"
+ " PinotLogicalAggregate(group=[{0}], agg#0=[COUNT($1)],
aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash[0]])\n"
- + " PinotLogicalAggregate(group=[{18}], agg#0=[COUNT()],
aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{23}], agg#0=[COUNT()],
aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n");
assertEquals(response1Json.get("rows").get(0).get(2).asText(), "Rule
Execution Times\n"
+ "Rule: SortRemove -> Time:*\n"
@@ -3870,22 +3942,22 @@ public class OfflineClusterIntegrationTest extends
BaseClusterIntegrationTestSet
JsonNode starColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()), "*"));
- validateMetadataResponse(starColumnResponse, 83, 10);
+ validateMetadataResponse(starColumnResponse, 88, 10);
JsonNode starEncodedColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()), "*"));
- validateMetadataResponse(starEncodedColumnResponse, 83, 10);
+ validateMetadataResponse(starEncodedColumnResponse, 88, 10);
JsonNode starWithExtraColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()),
"CRSElapsedTime,*,OriginStateName"));
- validateMetadataResponse(starWithExtraColumnResponse, 83, 10);
+ validateMetadataResponse(starWithExtraColumnResponse, 88, 10);
JsonNode starWithExtraEncodedColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()),
"CRSElapsedTime,*,OriginStateName"));
- validateMetadataResponse(starWithExtraEncodedColumnResponse, 83, 10);
+ validateMetadataResponse(starWithExtraEncodedColumnResponse, 88, 10);
}
private void validateMetadataResponse(JsonNode response, int numTotalColumn,
int numMVColumn) {
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java
index f8db37712bc..05ee8c774d7 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java
@@ -18,116 +18,22 @@
*/
package org.apache.pinot.segment.local.segment.index.column;
-import java.math.BigDecimal;
-import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueBigDecimalDictionary;
-import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueBytesDictionary;
-import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueDoubleDictionary;
-import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueFloatDictionary;
-import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueIntDictionary;
-import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueLongDictionary;
-import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueStringDictionary;
-import
org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVForwardIndexReader;
-import
org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVInvertedIndexReader;
-import
org.apache.pinot.segment.local.segment.index.readers.constant.ConstantSortedIndexReader;
+import
org.apache.pinot.segment.local.segment.virtualcolumn.BaseConstantValueVirtualColumnProvider;
import
org.apache.pinot.segment.local.segment.virtualcolumn.VirtualColumnContext;
-import
org.apache.pinot.segment.local.segment.virtualcolumn.VirtualColumnProvider;
-import org.apache.pinot.segment.spi.index.metadata.ColumnMetadataImpl;
-import org.apache.pinot.segment.spi.index.reader.Dictionary;
-import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
-import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.utils.ByteArray;
-/// Provide the default null value.
-public class DefaultNullValueVirtualColumnProvider implements
VirtualColumnProvider {
+/// Provides the column's default null value for every document.
+///
+/// This is also how `$hostName` and `$segmentName` are served: their value is
constant for the segment and is carried
+/// as the field's default null value.
+///
+/// NOTE: The fully-qualified name of this class is stored in field specs (see
+/// `DimensionFieldSpec#DimensionFieldSpec(String, DataType, boolean, Class)`)
and resolved reflectively, so it must
+/// not be moved or renamed.
+public class DefaultNullValueVirtualColumnProvider extends
BaseConstantValueVirtualColumnProvider {
@Override
- public ForwardIndexReader<?> buildForwardIndex(VirtualColumnContext context)
{
- if (context.getFieldSpec().isSingleValueField()) {
- return new ConstantSortedIndexReader(context.getTotalDocCount());
- } else {
- return new ConstantMVForwardIndexReader();
- }
- }
-
- @Override
- public Dictionary buildDictionary(VirtualColumnContext context) {
- FieldSpec fieldSpec = context.getFieldSpec();
- switch (fieldSpec.getDataType().getStoredType()) {
- case INT:
- return new ConstantValueIntDictionary((int)
fieldSpec.getDefaultNullValue());
- case LONG:
- return new ConstantValueLongDictionary((long)
fieldSpec.getDefaultNullValue());
- case FLOAT:
- return new ConstantValueFloatDictionary((float)
fieldSpec.getDefaultNullValue());
- case DOUBLE:
- return new ConstantValueDoubleDictionary((double)
fieldSpec.getDefaultNullValue());
- case BIG_DECIMAL:
- return new ConstantValueBigDecimalDictionary((BigDecimal)
fieldSpec.getDefaultNullValue());
- case STRING:
- return new ConstantValueStringDictionary((String)
fieldSpec.getDefaultNullValue());
- case BYTES:
- case UUID:
- return new ConstantValueBytesDictionary((byte[])
fieldSpec.getDefaultNullValue());
- default:
- throw new IllegalStateException();
- }
- }
-
- @Override
- public InvertedIndexReader<?> buildInvertedIndex(VirtualColumnContext
context) {
- if (context.getFieldSpec().isSingleValueField()) {
- return new ConstantSortedIndexReader(context.getTotalDocCount());
- } else {
- return new ConstantMVInvertedIndexReader(context.getTotalDocCount());
- }
- }
-
- @Override
- public ColumnMetadataImpl buildMetadata(VirtualColumnContext context) {
- FieldSpec fieldSpec = context.getFieldSpec();
- ColumnMetadataImpl.Builder builder = new
ColumnMetadataImpl.Builder().setFieldSpec(fieldSpec)
- .setTotalDocs(context.getTotalDocCount())
- .setCardinality(1)
- .setHasDictionary(true);
- if (fieldSpec.isSingleValueField()) {
- builder.setSorted(true);
- } else {
- // When there is no value for a multi-value column, the
maxNumberOfMultiValues and cardinality should be
- // set as 1 because the MV column bitmap uses 1 to delimit the rows for
a MV column. Each MV column will have a
- // default null value based on column's data type
- builder.setMaxNumberOfMultiValues(1);
- }
-
- Object defaultNullValue = fieldSpec.getDefaultNullValue();
- switch (fieldSpec.getDataType().getStoredType()) {
- case INT:
- builder.setMinValue((int) defaultNullValue).setMaxValue((int)
defaultNullValue);
- break;
- case LONG:
- builder.setMinValue((long) defaultNullValue).setMaxValue((long)
defaultNullValue);
- break;
- case FLOAT:
- builder.setMinValue((float) defaultNullValue).setMaxValue((float)
defaultNullValue);
- break;
- case DOUBLE:
- builder.setMinValue((double) defaultNullValue).setMaxValue((double)
defaultNullValue);
- break;
- case BIG_DECIMAL:
- builder.setMinValue((BigDecimal)
defaultNullValue).setMaxValue((BigDecimal) defaultNullValue);
- break;
- case STRING:
- builder.setMinValue((String) defaultNullValue).setMaxValue((String)
defaultNullValue);
- break;
- case BYTES:
- builder.setMinValue(new ByteArray((byte[]) defaultNullValue))
- .setMaxValue(new ByteArray((byte[]) defaultNullValue));
- break;
- default:
- throw new IllegalStateException();
- }
-
- return builder.build();
+ protected Object getValue(VirtualColumnContext context) {
+ return context.getFieldSpec().getDefaultNullValue();
}
}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java
index be4ad585650..1b36dceff4d 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java
@@ -23,19 +23,17 @@ import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
import org.apache.pinot.segment.local.segment.index.datasource.BaseDataSource;
+import
org.apache.pinot.segment.local.segment.index.readers.AllNullValueVectorReader;
import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
import org.apache.pinot.segment.spi.datasource.OpenStructDataSource;
import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
-import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.segment.spi.partition.PartitionFunction;
import org.apache.pinot.spi.data.DimensionFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.FieldSpec.DataType;
-import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
-import org.roaringbitmap.buffer.MutableRoaringBitmap;
/// Typed all-null {@link org.apache.pinot.segment.spi.datasource.DataSource}
for an OPEN_STRUCT key
@@ -54,7 +52,7 @@ public class OpenStructNullDataSource extends BaseDataSource {
public OpenStructNullDataSource(FieldSpec fieldSpec, int numDocs) {
super(new AllNullMetadata(fieldSpec, numDocs), new
ColumnIndexContainer.FromMap(Map.of(
StandardIndexes.forward(), new
TypedNullForwardIndex(fieldSpec.getDataType().getStoredType()),
- StandardIndexes.nullValueVector(), new AllNullValueVector(numDocs))));
+ StandardIndexes.nullValueVector(), new
AllNullValueVectorReader(numDocs))));
}
/// Creates an all-null DataSource for a key absent from this OPEN_STRUCT
segment.
@@ -131,29 +129,6 @@ public class OpenStructNullDataSource extends
BaseDataSource {
}
}
- /// {@link NullValueVectorReader} where every document is null.
- static class AllNullValueVector implements NullValueVectorReader {
- private final ImmutableRoaringBitmap _nullBitmap;
-
- AllNullValueVector(int numDocs) {
- MutableRoaringBitmap bm = new MutableRoaringBitmap();
- if (numDocs > 0) {
- bm.add(0L, numDocs);
- }
- _nullBitmap = bm.toImmutableRoaringBitmap();
- }
-
- @Override
- public boolean isNull(int docId) {
- return true;
- }
-
- @Override
- public ImmutableRoaringBitmap getNullBitmap() {
- return _nullBitmap;
- }
- }
-
private static class AllNullMetadata implements DataSourceMetadata {
private final FieldSpec _fieldSpec;
private final int _numDocs;
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/AllNullValueVectorReader.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/AllNullValueVectorReader.java
new file mode 100644
index 00000000000..5c8d6a50e06
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/AllNullValueVectorReader.java
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.readers;
+
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+
+
+/// [NullValueVectorReader] where every document of the column is null.
+///
+/// Used by columns that carry no value at all: a materialized-but-absent
OPEN_STRUCT child, and the segment metadata
+/// virtual columns whose metadata is not available yet (e.g. the time range
of a CONSUMING segment).
+///
+/// The bitmap is materialized lazily, because null value vectors are only
consulted when null handling is enabled
+/// while the reader itself is created on every data source build for a
mutable segment.
+public class AllNullValueVectorReader implements NullValueVectorReader {
+ private final int _numDocs;
+ private volatile ImmutableRoaringBitmap _nullBitmap;
+
+ public AllNullValueVectorReader(int numDocs) {
+ _numDocs = numDocs;
+ }
+
+ @Override
+ public boolean isNull(int docId) {
+ return true;
+ }
+
+ @Override
+ public ImmutableRoaringBitmap getNullBitmap() {
+ ImmutableRoaringBitmap nullBitmap = _nullBitmap;
+ if (nullBitmap == null) {
+ // Benign race: concurrent callers may each build an equivalent bitmap,
and publication is safe because the
+ // volatile write happens-after the bitmap is fully built. NOTE:
toImmutableRoaringBitmap() returns `this`, so
+ // the returned instance is shared and must be treated as read-only by
callers.
+ MutableRoaringBitmap bitmap = _numDocs > 0 ?
MutableRoaringBitmap.bitmapOfRange(0, _numDocs)
+ : new MutableRoaringBitmap();
+ nullBitmap = bitmap.toImmutableRoaringBitmap();
+ _nullBitmap = nullBitmap;
+ }
+ return nullBitmap;
+ }
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseConstantValueVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseConstantValueVirtualColumnProvider.java
new file mode 100644
index 00000000000..d4d7c9fcc10
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseConstantValueVirtualColumnProvider.java
@@ -0,0 +1,220 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import javax.annotation.Nullable;
+import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueBigDecimalDictionary;
+import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueBytesDictionary;
+import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueDoubleDictionary;
+import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueFloatDictionary;
+import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueIntDictionary;
+import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueLongDictionary;
+import
org.apache.pinot.segment.local.segment.index.readers.ConstantValueStringDictionary;
+import
org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVForwardIndexReader;
+import
org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVInvertedIndexReader;
+import
org.apache.pinot.segment.local.segment.index.readers.constant.ConstantSortedIndexReader;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.metadata.ColumnMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+/// Base for virtual columns holding a single constant value for every
document of a segment.
+///
+/// Subclasses supply the value by overriding
[#getValue(VirtualColumnContext)]; everything else - a constant sorted
+/// forward index, a single-entry dictionary and matching [ColumnMetadataImpl]
- is derived from it here.
+public abstract class BaseConstantValueVirtualColumnProvider implements
VirtualColumnProvider {
+
+ /// Returns the constant value to be stored for every document of the
column. The returned object must match the
+ /// stored type of the field spec (e.g. a `Long` for a LONG or TIMESTAMP
column).
+ protected abstract Object getValue(VirtualColumnContext context);
+
+ /// Reads the value via [#getValue(VirtualColumnContext)] and checks it
against the field's stored type, so that a
+ /// provider returning the wrong box type fails with the offending column
and provider named rather than with a bare
+ /// `ClassCastException` from deep inside segment loading.
+ private Object getCheckedValue(VirtualColumnContext context) {
+ return checkValue(context, getValue(context));
+ }
+
+ /// Checks an already-resolved value against the field's stored type. Every
path that turns a value into an index
+ /// goes through here, including the one taken by subclasses that resolve
the value themselves.
+ private Object checkValue(VirtualColumnContext context, Object value) {
+ FieldSpec fieldSpec = context.getFieldSpec();
+ Class<?> expectedClass = getValueClass(fieldSpec);
+ Preconditions.checkState(expectedClass.isInstance(value),
+ "Virtual column provider: %s returned value of type: %s for column:
%s, expecting: %s", getClass().getName(),
+ value != null ? value.getClass().getName() : "null",
fieldSpec.getName(), expectedClass.getName());
+ return value;
+ }
+
+ private static Class<?> getValueClass(FieldSpec fieldSpec) {
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ return Integer.class;
+ case LONG:
+ return Long.class;
+ case FLOAT:
+ return Float.class;
+ case DOUBLE:
+ return Double.class;
+ case BIG_DECIMAL:
+ return BigDecimal.class;
+ case STRING:
+ return String.class;
+ case BYTES:
+ case UUID:
+ return byte[].class;
+ default:
+ throw new IllegalStateException(unsupportedStoredType(fieldSpec));
+ }
+ }
+
+ private static String unsupportedStoredType(FieldSpec fieldSpec) {
+ return "Unsupported stored type: " +
fieldSpec.getDataType().getStoredType() + " for virtual column: "
+ + fieldSpec.getName();
+ }
+
+ @Override
+ public ForwardIndexReader<?> buildForwardIndex(VirtualColumnContext context)
{
+ if (context.getFieldSpec().isSingleValueField()) {
+ return new ConstantSortedIndexReader(context.getTotalDocCount());
+ } else {
+ return new ConstantMVForwardIndexReader();
+ }
+ }
+
+ @Override
+ public Dictionary buildDictionary(VirtualColumnContext context) {
+ return buildDictionary(context, getCheckedValue(context));
+ }
+
+ /// Builds the dictionary from an already-resolved value, so that callers
which need the value more than once can
+ /// resolve it a single time.
+ private Dictionary buildDictionary(VirtualColumnContext context, Object
value) {
+ FieldSpec fieldSpec = context.getFieldSpec();
+ checkValue(context, value);
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ return new ConstantValueIntDictionary((int) value);
+ case LONG:
+ return new ConstantValueLongDictionary((long) value);
+ case FLOAT:
+ return new ConstantValueFloatDictionary((float) value);
+ case DOUBLE:
+ return new ConstantValueDoubleDictionary((double) value);
+ case BIG_DECIMAL:
+ return new ConstantValueBigDecimalDictionary((BigDecimal) value);
+ case STRING:
+ return new ConstantValueStringDictionary((String) value);
+ case BYTES:
+ case UUID:
+ return new ConstantValueBytesDictionary((byte[]) value);
+ default:
+ throw new IllegalStateException(unsupportedStoredType(fieldSpec));
+ }
+ }
+
+ @Override
+ public InvertedIndexReader<?> buildInvertedIndex(VirtualColumnContext
context) {
+ if (context.getFieldSpec().isSingleValueField()) {
+ return new ConstantSortedIndexReader(context.getTotalDocCount());
+ } else {
+ return new ConstantMVInvertedIndexReader(context.getTotalDocCount());
+ }
+ }
+
+ @Override
+ public ColumnMetadataImpl buildMetadata(VirtualColumnContext context) {
+ return buildMetadata(context, getCheckedValue(context));
+ }
+
+ /// Builds the column metadata from an already-resolved value.
+ protected final ColumnMetadataImpl buildMetadata(VirtualColumnContext
context, Object value) {
+ return buildMetadata(context, value, true);
+ }
+
+ /// Builds the column metadata from an already-resolved value.
+ ///
+ /// @param hasValue `false` when `value` is only a placeholder standing in
for a value that is not available, in
+ /// which case the min/max are left unset. Segment pruners
read min/max without consulting the null
+ /// value vector, so publishing the placeholder there would
let an all-null column prune or reorder
+ /// segments as if it held a real extreme value.
+ protected final ColumnMetadataImpl buildMetadata(VirtualColumnContext
context, Object value, boolean hasValue) {
+ FieldSpec fieldSpec = context.getFieldSpec();
+ checkValue(context, value);
+ ColumnMetadataImpl.Builder builder = new
ColumnMetadataImpl.Builder().setFieldSpec(fieldSpec)
+ .setTotalDocs(context.getTotalDocCount())
+ .setCardinality(1)
+ .setHasDictionary(true);
+ if (fieldSpec.isSingleValueField()) {
+ builder.setSorted(true);
+ } else {
+ // When there is no value for a multi-value column, the
maxNumberOfMultiValues and cardinality should be
+ // set as 1 because the MV column bitmap uses 1 to delimit the rows for
a MV column. Each MV column will have a
+ // default null value based on column's data type
+ builder.setMaxNumberOfMultiValues(1);
+ }
+
+ if (!hasValue) {
+ return builder.build();
+ }
+
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ builder.setMinValue((int) value).setMaxValue((int) value);
+ break;
+ case LONG:
+ builder.setMinValue((long) value).setMaxValue((long) value);
+ break;
+ case FLOAT:
+ builder.setMinValue((float) value).setMaxValue((float) value);
+ break;
+ case DOUBLE:
+ builder.setMinValue((double) value).setMaxValue((double) value);
+ break;
+ case BIG_DECIMAL:
+ builder.setMinValue((BigDecimal) value).setMaxValue((BigDecimal)
value);
+ break;
+ case STRING:
+ builder.setMinValue((String) value).setMaxValue((String) value);
+ break;
+ case BYTES:
+ builder.setMinValue(new ByteArray((byte[]) value)).setMaxValue(new
ByteArray((byte[]) value));
+ break;
+ default:
+ throw new IllegalStateException(unsupportedStoredType(fieldSpec));
+ }
+
+ return builder.build();
+ }
+
+ /// Builds the column index container from an already-resolved value, so
that a subclass whose value comes from
+ /// mutable state can resolve it a single time and keep every component
consistent.
+ protected final ColumnIndexContainer
buildColumnIndexContainer(VirtualColumnContext context, Object value,
+ @Nullable NullValueVectorReader nullValueVector) {
+ return new VirtualColumnIndexContainer(buildForwardIndex(context),
buildInvertedIndex(context),
+ buildDictionary(context, value), nullValueVector);
+ }
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseSegmentMetadataVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseSegmentMetadataVirtualColumnProvider.java
new file mode 100644
index 00000000000..9e28ba5c836
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseSegmentMetadataVirtualColumnProvider.java
@@ -0,0 +1,97 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import
org.apache.pinot.segment.local.segment.index.datasource.ImmutableDataSource;
+import
org.apache.pinot.segment.local.segment.index.readers.AllNullValueVectorReader;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.metadata.ColumnMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/// Base class for the built-in virtual columns that expose a piece of the
segment metadata (creation time, time range,
+/// CRC, etc.) as a constant single-value column.
+///
+/// The value is read from [SegmentMetadata] when the column is built rather
than baked into the field spec, so that a
+/// mutable segment - which rebuilds its virtual data sources on every access
- picks up metadata that was not yet
+/// available when the segment was created. Within a single build the value is
resolved exactly once (see
+/// [BaseConstantValueVirtualColumnProvider#buildColumnIndexContainer]), so
the dictionary and the null value vector
+/// can never disagree about whether the column has a value.
+///
+/// When the metadata is not available at all, or the specific piece of
metadata is not set (e.g. the time range of a
+/// CONSUMING segment), the column stores the default null value of its data
type *and* reports every document as null,
+/// so that the placeholder is not mistaken for a real value once null
handling is enabled.
+public abstract class BaseSegmentMetadataVirtualColumnProvider extends
BaseConstantValueVirtualColumnProvider {
+
+ @Override
+ protected Object getValue(VirtualColumnContext context) {
+ return valueOrPlaceholder(context, extractValueOrNull(context));
+ }
+
+ @Nullable
+ @Override
+ public NullValueVectorReader buildNullValueVector(VirtualColumnContext
context) {
+ return extractValueOrNull(context) == null ? new
AllNullValueVectorReader(context.getTotalDocCount()) : null;
+ }
+
+ @Override
+ public ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext
context) {
+ return buildColumnIndexContainer(context, extractValueOrNull(context));
+ }
+
+ @Override
+ public ColumnMetadataImpl buildMetadata(VirtualColumnContext context) {
+ Object extracted = extractValueOrNull(context);
+ return buildMetadata(context, valueOrPlaceholder(context, extracted),
extracted != null);
+ }
+
+ @Override
+ public DataSource buildDataSource(VirtualColumnContext context) {
+ // Resolve the metadata exactly once for the whole data source, so the
dictionary, the column metadata's min/max
+ // and the null value vector all agree about whether this column has a
value. buildDataSource is the path a
+ // mutable segment takes, and it rebuilds its virtual data sources on
every access.
+ Object extracted = extractValueOrNull(context);
+ return new ImmutableDataSource(buildMetadata(context,
valueOrPlaceholder(context, extracted), extracted != null),
+ buildColumnIndexContainer(context, extracted));
+ }
+
+ private ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext
context, @Nullable Object extracted) {
+ return buildColumnIndexContainer(context, valueOrPlaceholder(context,
extracted),
+ extracted == null ? new
AllNullValueVectorReader(context.getTotalDocCount()) : null);
+ }
+
+ /// Falls back to the column's default null value, which is only ever a
placeholder: whenever it is used, the column
+ /// also reports every document as null.
+ private static Object valueOrPlaceholder(VirtualColumnContext context,
@Nullable Object extracted) {
+ return extracted != null ? extracted :
context.getFieldSpec().getDefaultNullValue();
+ }
+
+ @Nullable
+ private Object extractValueOrNull(VirtualColumnContext context) {
+ SegmentMetadata segmentMetadata = context.getSegmentMetadata();
+ return segmentMetadata != null ? extractValue(segmentMetadata) : null;
+ }
+
+ /// Extracts the value for this column from the given segment metadata, or
`null` when it is not available.
+ @Nullable
+ protected abstract Object extractValue(SegmentMetadata segmentMetadata);
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCrcVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCrcVirtualColumnProvider.java
new file mode 100644
index 00000000000..d64d0683a61
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCrcVirtualColumnProvider.java
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+
+
+/// Virtual column provider for `$crc`, the CRC of the segment.
+///
+/// The CRC is exposed as a LONG, matching how it is stored everywhere else in
Pinot ([SegmentMetadata#getCrc()]
+/// renders the same `long` as a String). It reads as NULL for CONSUMING
segments, which have no CRC until they are
+/// committed. Grouping by `$segmentName` and `$crc` is a convenient way to
detect replicas of a segment that have
+/// diverged.
+public class SegmentCrcVirtualColumnProvider extends
BaseSegmentMetadataVirtualColumnProvider {
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ String crc = segmentMetadata.getCrc();
+ if (crc == null) {
+ return null;
+ }
+ long crcValue;
+ try {
+ crcValue = Long.parseLong(crc);
+ } catch (NumberFormatException e) {
+ // The CRC is always rendered from a long, but never fail a segment load
over an unreadable one
+ return null;
+ }
+ // SegmentMetadataImpl renders an unset CRC as Long.MIN_VALUE
+ return crcValue != Long.MIN_VALUE ? crcValue : null;
+ }
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCreationTimeVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCreationTimeVirtualColumnProvider.java
new file mode 100644
index 00000000000..3d564876ab3
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCreationTimeVirtualColumnProvider.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+
+
+/// Virtual column provider for `$creationTime`, the time the segment was
created.
+///
+/// For a CONSUMING segment this is the time the consuming segment was
created, not the time it was committed.
+public class SegmentCreationTimeVirtualColumnProvider extends
BaseSegmentMetadataVirtualColumnProvider {
+
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ // An unset creation time is represented as Long.MIN_VALUE in the segment
metadata and as -1 in the segment ZK
+ // metadata a CONSUMING segment is created from, so treat any non-positive
value as unavailable.
+ long creationTime = segmentMetadata.getIndexCreationTime();
+ return creationTime > 0 ? creationTime : null;
+ }
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentEndTimeVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentEndTimeVirtualColumnProvider.java
new file mode 100644
index 00000000000..ee9d79fe671
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentEndTimeVirtualColumnProvider.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.joda.time.Interval;
+
+
+/// Virtual column provider for `$endTime`, the end of the segment time range.
+///
+/// The value is normalized from the time unit of the segment's time column,
and exposed as a TIMESTAMP. It reads as
+/// NULL for segments without a time range, such as CONSUMING segments and
segments of tables without a time column.
+public class SegmentEndTimeVirtualColumnProvider extends
BaseSegmentMetadataVirtualColumnProvider {
+
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ return timeInterval != null ? timeInterval.getEndMillis() : null;
+ }
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentStartTimeVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentStartTimeVirtualColumnProvider.java
new file mode 100644
index 00000000000..50600c974a7
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentStartTimeVirtualColumnProvider.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.joda.time.Interval;
+
+
+/// Virtual column provider for `$startTime`, the start of the segment time
range.
+///
+/// The value is normalized from the time unit of the segment's time column,
and exposed as a TIMESTAMP. It reads as
+/// NULL for segments without a time range, such as CONSUMING segments and
segments of tables without a time column.
+public class SegmentStartTimeVirtualColumnProvider extends
BaseSegmentMetadataVirtualColumnProvider {
+
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ return timeInterval != null ? timeInterval.getStartMillis() : null;
+ }
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentTotalDocsVirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentTotalDocsVirtualColumnProvider.java
new file mode 100644
index 00000000000..2b44503a2ba
--- /dev/null
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentTotalDocsVirtualColumnProvider.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+
+/// Virtual column provider for `$totalDocs`, the number of documents in the
segment.
+///
+/// The count is taken from the virtual column context rather than from
`SegmentMetadata`,
+/// so that a CONSUMING segment reports the number of documents indexed so far
instead of `0`.
+///
+/// This is the number of documents physically stored in the segment, so for
an upsert table it also includes the
+/// documents that have been replaced and are no longer returned by queries.
+public class SegmentTotalDocsVirtualColumnProvider extends
BaseConstantValueVirtualColumnProvider {
+
+ @Override
+ protected Object getValue(VirtualColumnContext context) {
+ return context.getTotalDocCount();
+ }
+}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java
index d64f7291269..fc6d40beff0 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java
@@ -27,6 +27,7 @@ import
org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
/// Column index container for virtual columns.
@@ -34,12 +35,23 @@ public class VirtualColumnIndexContainer implements
ColumnIndexContainer {
private final ForwardIndexReader<?> _forwardIndex;
private final InvertedIndexReader<?> _invertedIndex;
private final Dictionary _dictionary;
+ private final NullValueVectorReader _nullValueVector;
public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex,
InvertedIndexReader<?> invertedIndex,
Dictionary dictionary) {
+ this(forwardIndex, invertedIndex, dictionary, null);
+ }
+
+ /// @param nullValueVector marks which documents of the virtual column are
null, or `null` when the column has no
+ /// null value. A virtual column whose value is
genuinely unavailable (e.g. the time range of
+ /// a CONSUMING segment) must supply one, otherwise
the engine treats the placeholder value
+ /// stored in the forward index as a real value.
+ public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex,
InvertedIndexReader<?> invertedIndex,
+ Dictionary dictionary, @Nullable NullValueVectorReader nullValueVector) {
_forwardIndex = forwardIndex;
_invertedIndex = invertedIndex;
_dictionary = dictionary;
+ _nullValueVector = nullValueVector;
}
@Nullable
@@ -54,6 +66,9 @@ public class VirtualColumnIndexContainer implements
ColumnIndexContainer {
if (indexType.equals(StandardIndexes.dictionary())) {
return (I) _dictionary;
}
+ if (indexType.equals(StandardIndexes.nullValueVector())) {
+ return (I) _nullValueVector;
+ }
return null;
}
@@ -67,5 +82,8 @@ public class VirtualColumnIndexContainer implements
ColumnIndexContainer {
if (_dictionary != null) {
_dictionary.close();
}
+ if (_nullValueVector != null) {
+ _nullValueVector.close();
+ }
}
}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java
index 194773d0332..0079cdaa2b0 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java
@@ -18,6 +18,7 @@
*/
package org.apache.pinot.segment.local.segment.virtualcolumn;
+import javax.annotation.Nullable;
import
org.apache.pinot.segment.local.segment.index.datasource.ImmutableDataSource;
import org.apache.pinot.segment.spi.ColumnMetadata;
import org.apache.pinot.segment.spi.datasource.DataSource;
@@ -25,6 +26,7 @@ import
org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
/// Virtual column provider interface, which is used to instantiate the
various components (dictionary, reader, etc)
@@ -39,9 +41,19 @@ public interface VirtualColumnProvider {
ColumnMetadata buildMetadata(VirtualColumnContext context);
+ /// Returns the null value vector of the virtual column, or `null` when the
column has no null value.
+ ///
+ /// Virtual columns almost always carry a real value for every document,
hence the default. A provider whose value
+ /// can be genuinely unavailable must override this, otherwise the
placeholder stored in the forward index is
+ /// indistinguishable from a real value once null handling is enabled.
+ @Nullable
+ default NullValueVectorReader buildNullValueVector(VirtualColumnContext
context) {
+ return null;
+ }
+
default ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext
context) {
return new VirtualColumnIndexContainer(buildForwardIndex(context),
buildInvertedIndex(context),
- buildDictionary(context));
+ buildDictionary(context), buildNullValueVector(context));
}
default DataSource buildDataSource(VirtualColumnContext context) {
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java
index b60d9a3cf00..4d9f04908c3 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java
@@ -18,9 +18,12 @@
*/
package org.apache.pinot.segment.local.segment.virtualcolumn;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.stream.Collectors;
import
org.apache.pinot.segment.local.segment.index.column.DefaultNullValueVirtualColumnProvider;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
import org.apache.pinot.spi.data.DimensionFieldSpec;
-import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.plugin.PluginManager;
import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
@@ -29,11 +32,46 @@ import org.apache.pinot.spi.utils.NetUtils;
/// Factory for virtual column providers.
public class VirtualColumnProviderFactory {
+ /// Provider for each built-in virtual column. Its key set is asserted to
cover
+ /// [BuiltInVirtualColumnDefinitions#DEFINITIONS] by
`SegmentMetadataVirtualColumnProviderTest`, so a column added
+ /// there without a provider here is caught at build time; at runtime
`getProviderClass` names the offending column.
+ private static final Map<String, Class<? extends VirtualColumnProvider>>
PROVIDER_CLASSES =
+ Map.of(BuiltInVirtualColumn.DOCID, DocIdVirtualColumnProvider.class,
+ BuiltInVirtualColumn.HOSTNAME,
DefaultNullValueVirtualColumnProvider.class,
+ BuiltInVirtualColumn.SEGMENTNAME,
DefaultNullValueVirtualColumnProvider.class,
+ BuiltInVirtualColumn.PARTITIONID,
PartitionIdVirtualColumnProvider.class,
+ BuiltInVirtualColumn.CREATIONTIME,
SegmentCreationTimeVirtualColumnProvider.class,
+ BuiltInVirtualColumn.STARTTIME,
SegmentStartTimeVirtualColumnProvider.class,
+ BuiltInVirtualColumn.ENDTIME,
SegmentEndTimeVirtualColumnProvider.class,
+ BuiltInVirtualColumn.TOTALDOCS,
SegmentTotalDocsVirtualColumnProvider.class,
+ BuiltInVirtualColumn.CRC, SegmentCrcVirtualColumnProvider.class);
+
+ /// Shared instances of the built-in providers, keyed by the class name
stored in the field spec.
+ private static final Map<String, VirtualColumnProvider> BUILT_IN_PROVIDERS =
PROVIDER_CLASSES.values()
+ .stream()
+ .distinct()
+ .collect(Collectors.toUnmodifiableMap(Class::getName,
VirtualColumnProviderFactory::newInstance));
+
private VirtualColumnProviderFactory() {
}
+ private static VirtualColumnProvider newInstance(Class<? extends
VirtualColumnProvider> providerClass) {
+ try {
+ return providerClass.getDeclaredConstructor().newInstance();
+ } catch (Exception e) {
+ throw new IllegalStateException("Caught exception while creating
instance of: " + providerClass.getName(), e);
+ }
+ }
+
public static VirtualColumnProvider buildProvider(VirtualColumnContext
virtualColumnContext) {
String virtualColumnProvider =
virtualColumnContext.getFieldSpec().getVirtualColumnProvider();
+ // The built-in providers are stateless - everything they need comes from
the VirtualColumnContext - so a single
+ // shared instance serves every column of every segment. This matters at
segment load, which resolves a provider
+ // per virtual column per segment, and on the mutable path, which resolves
one per query.
+ VirtualColumnProvider builtInProvider =
BUILT_IN_PROVIDERS.get(virtualColumnProvider);
+ if (builtInProvider != null) {
+ return builtInProvider;
+ }
try {
return PluginManager.get().createInstance(virtualColumnProvider);
} catch (Exception e) {
@@ -41,25 +79,37 @@ public class VirtualColumnProviderFactory {
}
}
+ /// Adds the built-in virtual columns to the schema of a segment, together
with the provider that produces their
+ /// values.
+ ///
+ /// The shape of each column (name, data type, single-value vs multi-value)
comes from
+ /// [BuiltInVirtualColumnDefinitions#DEFINITIONS], which the broker side
uses as well, so the two can never
+ /// disagree on a type.
+ /// This method only layers on the provider class, and the constant value
for the columns whose value is already
+ /// known here.
public static void addBuiltInVirtualColumnsToSegmentSchema(Schema schema,
String segmentName) {
- if (!schema.hasColumn(BuiltInVirtualColumn.DOCID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.DOCID,
FieldSpec.DataType.INT, true,
- DocIdVirtualColumnProvider.class));
- }
-
- if (!schema.hasColumn(BuiltInVirtualColumn.HOSTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.HOSTNAME,
FieldSpec.DataType.STRING, true,
- DefaultNullValueVirtualColumnProvider.class,
NetUtils.getHostnameOrAddress()));
- }
-
- if (!schema.hasColumn(BuiltInVirtualColumn.SEGMENTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.SEGMENTNAME,
FieldSpec.DataType.STRING, true,
- DefaultNullValueVirtualColumnProvider.class, segmentName));
+ for (BuiltInVirtualColumnDefinitions.Definition definition :
BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ String column = definition.getName();
+ if (schema.hasColumn(column)) {
+ continue;
+ }
+ DimensionFieldSpec fieldSpec = definition.createFieldSpec();
+ fieldSpec.setVirtualColumnProvider(getProviderClass(column).getName());
+ // $hostName and $segmentName are constants known at schema construction
time, and are carried as the field's
+ // default null value, which DefaultNullValueVirtualColumnProvider reads
back.
+ if (BuiltInVirtualColumn.HOSTNAME.equals(column)) {
+ fieldSpec.setDefaultNullValue(NetUtils.getHostnameOrAddress());
+ } else if (BuiltInVirtualColumn.SEGMENTNAME.equals(column)) {
+ fieldSpec.setDefaultNullValue(segmentName);
+ }
+ schema.addField(fieldSpec);
}
+ }
- if (!schema.hasColumn(BuiltInVirtualColumn.PARTITIONID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.PARTITIONID,
FieldSpec.DataType.STRING, false,
- PartitionIdVirtualColumnProvider.class));
- }
+ private static Class<? extends VirtualColumnProvider>
getProviderClass(String column) {
+ Class<? extends VirtualColumnProvider> providerClass =
PROVIDER_CLASSES.get(column);
+ Preconditions.checkState(providerClass != null, "No virtual column
provider registered for built-in column: %s",
+ column);
+ return providerClass;
}
}
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java
index 9c257b06eee..1481a5b26d8 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java
@@ -31,6 +31,7 @@ import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.pinot.common.evaluator.FunctionEvaluatorFactory;
import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
import org.apache.pinot.spi.data.DateTimeFieldSpec;
import org.apache.pinot.spi.data.DateTimeFormatSpec;
import org.apache.pinot.spi.data.DateTimeGranularitySpec;
@@ -162,6 +163,12 @@ public class SchemaUtils {
String column = fieldSpec.getName();
Preconditions.checkState(!StringUtils.containsWhitespace(column),
"The column name \"%s\" should not contain blank space.", column);
+ // A user column of the same name would be shadowed at query time: the
built-in virtual column is filtered out
+ // of the segment's physical columns when the segment metadata is read,
and the virtual provider then takes over
+ // the name, so queries would silently return segment metadata instead
of the user's data.
+ Preconditions.checkState(
+ fieldSpec.isVirtualColumn() ||
!BuiltInVirtualColumnDefinitions.NAMES.contains(column),
+ "The column name \"%s\" is reserved for a built-in virtual column.",
column);
if (!fieldSpec.isVirtualColumn()) {
primaryKeyColumnCandidates.add(column);
}
diff --git
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java
index 448bfd242b9..82153f600e8 100644
---
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java
+++
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java
@@ -67,6 +67,14 @@ import static org.testng.Assert.assertNull;
public class MutableSegmentImplRawMVTest implements
PinotBuffersAfterClassCheckRule {
private static final String AVRO_FILE = "data/test_data-mv.avro";
private static final File TEMP_DIR = new File(FileUtils.getTempDirectory(),
"MutableSegmentImplRawMVTest");
+ /// Virtual columns describing the segment itself, which are expected to
differ between a mutable segment and an
+ /// immutable segment built from the same records.
+ private static final Set<String> SEGMENT_LEVEL_VIRTUAL_COLUMNS =
+ Set.of(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CREATIONTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.STARTTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.ENDTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CRC);
private Schema _schema;
private MutableSegmentImpl _mutableSegmentImpl;
@@ -166,8 +174,9 @@ public class MutableSegmentImplRawMVTest implements
PinotBuffersAfterClassCheckR
Dictionary expectedDictionary = expectedDataSource.getDictionary();
assertEquals(actualDictionary.length(), expectedDictionary.length());
- // Allow the segment name to be different
- if
(column.equals(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME)) {
+ // Allow the segment level metadata to be different between the
mutable segment and the immutable segment
+ // built from the same records
+ if (SEGMENT_LEVEL_VIRTUAL_COLUMNS.contains(column)) {
continue;
}
diff --git
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java
index c837922f05e..537704ed557 100644
---
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java
+++
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java
@@ -38,6 +38,7 @@ import
org.apache.pinot.segment.spi.index.creator.VectorIndexConfig;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.spi.config.instance.InstanceType;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
@@ -62,6 +63,14 @@ import static org.testng.Assert.assertEquals;
public class MutableSegmentImplTest {
private static final String AVRO_FILE = "data/test_data-mv.avro";
private static final File TEMP_DIR = new File(FileUtils.getTempDirectory(),
"MutableSegmentImplTest");
+ /// Virtual columns describing the segment itself, which are expected to
differ between a mutable segment and an
+ /// immutable segment built from the same records.
+ private static final Set<String> SEGMENT_LEVEL_VIRTUAL_COLUMNS =
+ Set.of(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CREATIONTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.STARTTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.ENDTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CRC);
private Schema _schema;
private MutableSegmentImpl _mutableSegmentImpl;
@@ -132,6 +141,29 @@ public class MutableSegmentImplTest {
}
}
+ /// The segment metadata virtual columns are skipped in the
mutable-vs-immutable comparisons above because they
+ /// legitimately differ, so their values on a real mutable segment are
pinned here instead. This mutable segment is
+ /// built from ZK metadata without a creation time and never gets a time
range or a CRC, which is exactly the shape
+ /// of a CONSUMING segment.
+ @Test
+ public void testSegmentMetadataVirtualColumnsOnMutableSegment() {
+ for (String column :
Set.of(CommonConstants.Segment.BuiltInVirtualColumn.CREATIONTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.STARTTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.ENDTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CRC)) {
+ DataSource dataSource = _mutableSegmentImpl.getDataSource(column);
+ NullValueVectorReader nullValueVector = dataSource.getNullValueVector();
+ Assert.assertNotNull(nullValueVector, "Expecting a null value vector for
virtual column: " + column);
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(),
_mutableSegmentImpl.getNumDocsIndexed());
+ }
+
+ // $totalDocs tracks the documents indexed so far, and is never null
+ DataSource totalDocsDataSource =
+
_mutableSegmentImpl.getDataSource(CommonConstants.Segment.BuiltInVirtualColumn.TOTALDOCS);
+ assertEquals(totalDocsDataSource.getDictionary().getIntValue(0),
_mutableSegmentImpl.getNumDocsIndexed());
+ Assert.assertNull(totalDocsDataSource.getNullValueVector());
+ }
+
@Test
public void testDataSourceForSVColumns() {
for (FieldSpec fieldSpec : _schema.getAllFieldSpecs()) {
@@ -148,8 +180,9 @@ public class MutableSegmentImplTest {
Dictionary expectedDictionary = expectedDataSource.getDictionary();
assertEquals(actualDictionary.length(), expectedDictionary.length());
- // Allow the segment name to be different
- if
(column.equals(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME)) {
+ // Allow the segment level metadata to be different between the
mutable segment and the immutable segment
+ // built from the same records
+ if (SEGMENT_LEVEL_VIRTUAL_COLUMNS.contains(column)) {
continue;
}
diff --git
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java
index cd50d0de804..ac22a9a3e56 100644
---
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java
+++
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java
@@ -20,7 +20,6 @@ package org.apache.pinot.segment.local.segment.index.loader;
import java.io.File;
import java.net.URL;
-import java.util.Arrays;
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
@@ -34,11 +33,14 @@ import
org.apache.pinot.segment.local.utils.SegmentOperationsThrottler;
import org.apache.pinot.segment.local.utils.SegmentOperationsThrottlerSet;
import org.apache.pinot.segment.spi.ImmutableSegment;
import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentMetadata;
import org.apache.pinot.segment.spi.V1Constants;
import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
import org.apache.pinot.segment.spi.creator.SegmentVersion;
+import org.apache.pinot.segment.spi.datasource.DataSource;
import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
import org.apache.pinot.spi.config.table.FieldConfig;
@@ -217,11 +219,35 @@ public class LoaderTest {
}
private void testBuiltInVirtualColumns(IndexSegment indexSegment) {
- assertTrue(indexSegment.getColumnNames().containsAll(
- Arrays.asList(BuiltInVirtualColumn.DOCID,
BuiltInVirtualColumn.HOSTNAME, BuiltInVirtualColumn.SEGMENTNAME)));
- assertNotNull(indexSegment.getDataSource(BuiltInVirtualColumn.DOCID));
- assertNotNull(indexSegment.getDataSource(BuiltInVirtualColumn.HOSTNAME));
-
assertNotNull(indexSegment.getDataSource(BuiltInVirtualColumn.SEGMENTNAME));
+ // Iterate the full set so that a newly added built-in virtual column is
covered automatically
+
assertTrue(indexSegment.getColumnNames().containsAll(BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS));
+ for (String column : BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS) {
+ assertNotNull(indexSegment.getDataSource(column), "Missing data source
for virtual column: " + column);
+ }
+
+ // Segment metadata that this segment carries is exposed as a real value,
and is not marked null
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+
assertEquals(indexSegment.getDataSource(BuiltInVirtualColumn.TOTALDOCS).getDictionary().get(0),
+ segmentMetadata.getTotalDocs());
+
assertEquals(indexSegment.getDataSource(BuiltInVirtualColumn.CRC).getDictionary().get(0),
+ Long.parseLong(segmentMetadata.getCrc()));
+
assertEquals(indexSegment.getDataSource(BuiltInVirtualColumn.CREATIONTIME).getDictionary().get(0),
+ segmentMetadata.getIndexCreationTime());
+ for (String column : List.of(BuiltInVirtualColumn.TOTALDOCS,
BuiltInVirtualColumn.CRC,
+ BuiltInVirtualColumn.CREATIONTIME)) {
+ assertNull(indexSegment.getDataSource(column).getNullValueVector(),
+ "Unexpected null value vector for virtual column: " + column);
+ }
+
+ // This table has no time column, so the segment has no time range and the
two time columns read as NULL
+ assertNull(segmentMetadata.getTimeInterval());
+ for (String column : List.of(BuiltInVirtualColumn.STARTTIME,
BuiltInVirtualColumn.ENDTIME)) {
+ DataSource dataSource = indexSegment.getDataSource(column);
+ assertEquals(dataSource.getDictionary().get(0),
FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ NullValueVectorReader nullValueVector = dataSource.getNullValueVector();
+ assertNotNull(nullValueVector, "Missing null value vector for virtual
column: " + column);
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(),
segmentMetadata.getTotalDocs());
+ }
}
/// Tests loading default string column with empty ("") default null value.
diff --git
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentMetadataVirtualColumnProviderTest.java
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentMetadataVirtualColumnProviderTest.java
new file mode 100644
index 00000000000..a336a2e831f
--- /dev/null
+++
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentMetadataVirtualColumnProviderTest.java
@@ -0,0 +1,405 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
+import org.apache.pinot.spi.data.DimensionFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
+import org.joda.time.DateTimeZone;
+import org.joda.time.Interval;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+
+/// Tests for the built-in virtual columns exposing segment metadata
(`$creationTime`, `$startTime`, `$endTime`,
+/// `$totalDocs` and `$crc`).
+public class SegmentMetadataVirtualColumnProviderTest {
+ private static final int NUM_DOCS = 17;
+ private static final long CREATION_TIME_MS = 1_700_000_000_000L;
+ private static final long START_TIME_MS = 1_690_000_000_000L;
+ private static final long END_TIME_MS = 1_695_000_000_000L;
+ private static final long CRC = 1234567890L;
+
+ @DataProvider(name = "unsetCreationTimes")
+ public Object[][] unsetCreationTimes() {
+ return new Object[][]{{Long.MIN_VALUE}, {-1L}, {0L}};
+ }
+
+ /// Returns the schema a segment gets after the built-in virtual columns are
added to it.
+ private static Schema buildSegmentSchema() {
+ Schema schema = new Schema();
+
VirtualColumnProviderFactory.addBuiltInVirtualColumnsToSegmentSchema(schema,
"testSegment");
+ return schema;
+ }
+
+ private static SegmentMetadata mockSegmentMetadata() {
+ SegmentMetadata segmentMetadata = mock(SegmentMetadata.class);
+ when(segmentMetadata.getIndexCreationTime()).thenReturn(CREATION_TIME_MS);
+ when(segmentMetadata.getTimeInterval()).thenReturn(new
Interval(START_TIME_MS, END_TIME_MS, DateTimeZone.UTC));
+ when(segmentMetadata.getCrc()).thenReturn(String.valueOf(CRC));
+ when(segmentMetadata.getTotalDocs()).thenReturn(NUM_DOCS);
+ return segmentMetadata;
+ }
+
+ /// Returns the metadata a real CONSUMING segment is created with: no time
range and no CRC yet. Uses the real
+ /// `SegmentMetadataImpl` rather than a mock so that the unset-value
contract stays pinned to what
+ /// `pinot-segment-spi` actually produces.
+ private static SegmentMetadata consumingSegmentMetadata(long creationTime) {
+ return new SegmentMetadataImpl("testTable",
"testTable__0__0__20240101T0000Z", new Schema(), creationTime);
+ }
+
+ private static Dictionary buildDictionary(Schema schema, String column,
SegmentMetadata segmentMetadata) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ assertNotNull(fieldSpec, column + " should be added to the segment
schema");
+ assertTrue(fieldSpec.isVirtualColumn(), column + " should be a virtual
column");
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, segmentMetadata);
+ return
VirtualColumnProviderFactory.buildProvider(context).buildDictionary(context);
+ }
+
+ @Nullable
+ private static NullValueVectorReader buildNullValueVector(Schema schema,
String column,
+ SegmentMetadata segmentMetadata) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, segmentMetadata);
+ return
VirtualColumnProviderFactory.buildProvider(context).buildNullValueVector(context);
+ }
+
+ /// Asserts the column reads as SQL NULL for every document, and that the
data source exposes the null vector (which
+ /// is what the query engine consults once null handling is enabled).
+ private static void assertColumnIsNull(Schema schema, String column,
SegmentMetadata segmentMetadata) {
+ NullValueVectorReader nullValueVector = buildNullValueVector(schema,
column, segmentMetadata);
+ assertNotNull(nullValueVector, column + " should report a null value
vector when its metadata is unavailable");
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(), NUM_DOCS);
+ for (int docId = 0; docId < NUM_DOCS; docId++) {
+ assertTrue(nullValueVector.isNull(docId));
+ }
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, segmentMetadata);
+
assertNotNull(VirtualColumnProviderFactory.buildProvider(context).buildDataSource(context).getNullValueVector(),
+ column + " data source should expose the null value vector");
+ }
+
+ private static ColumnMetadata buildColumnMetadata(Schema schema, String
column, SegmentMetadata segmentMetadata) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, segmentMetadata);
+ return
VirtualColumnProviderFactory.buildProvider(context).buildMetadata(context);
+ }
+
+ /// Replaces the class-load assertion that used to live in
VirtualColumnProviderFactory: a definition added without
+ /// a provider must be caught here at build time rather than at segment load.
+ @Test
+ public void testEveryDefinitionHasAProvider() {
+ Schema schema = buildSegmentSchema();
+ for (BuiltInVirtualColumnDefinitions.Definition definition :
BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(definition.getName());
+ assertNotNull(fieldSpec, "No field spec added for: " +
definition.getName());
+ assertTrue(fieldSpec.isVirtualColumn(), "No provider configured for: " +
definition.getName());
+ assertEquals(fieldSpec.getDataType(), definition.getDataType());
+ assertEquals(fieldSpec.isSingleValueField(),
definition.isSingleValueField());
+ }
+ }
+
+ @Test
+ public void testAllBuiltInVirtualColumnsAreAddedToSegmentSchema() {
+ Schema schema = buildSegmentSchema();
+ for (String column : BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ assertNotNull(fieldSpec, column + " should be added to the segment
schema");
+ assertTrue(fieldSpec.isVirtualColumn(), column + " should have a virtual
column provider configured");
+ // Every configured provider must be resolvable
+ assertNotNull(VirtualColumnProviderFactory.buildProvider(
+ new VirtualColumnContext(fieldSpec, NUM_DOCS,
mockSegmentMetadata())));
+ }
+ }
+
+ @Test
+ public void testSegmentMetadataColumnTypes() {
+ Schema schema = buildSegmentSchema();
+
assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.CREATIONTIME).getDataType(),
+ FieldSpec.DataType.TIMESTAMP);
+
assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.STARTTIME).getDataType(),
+ FieldSpec.DataType.TIMESTAMP);
+
assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.ENDTIME).getDataType(),
+ FieldSpec.DataType.TIMESTAMP);
+
assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.TOTALDOCS).getDataType(),
FieldSpec.DataType.INT);
+
assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.CRC).getDataType(),
FieldSpec.DataType.LONG);
+ for (String column : List.of(BuiltInVirtualColumn.CREATIONTIME,
BuiltInVirtualColumn.STARTTIME,
+ BuiltInVirtualColumn.ENDTIME, BuiltInVirtualColumn.TOTALDOCS,
BuiltInVirtualColumn.CRC)) {
+ assertTrue(schema.getFieldSpecFor(column).isSingleValueField(), column +
" should be single-value");
+ }
+ }
+
+ @Test
+ public void testValuesFromSegmentMetadata() {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata = mockSegmentMetadata();
+
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CREATIONTIME,
segmentMetadata).getLongValue(0),
+ CREATION_TIME_MS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.STARTTIME,
segmentMetadata).getLongValue(0),
+ START_TIME_MS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.ENDTIME,
segmentMetadata).getLongValue(0), END_TIME_MS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.TOTALDOCS,
segmentMetadata).getIntValue(0), NUM_DOCS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CRC,
segmentMetadata).getLongValue(0), CRC);
+ }
+
+ @Test
+ public void testColumnMetadataMatchesValue() {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata = mockSegmentMetadata();
+
+ ColumnMetadata creationTimeMetadata =
+ buildColumnMetadata(schema, BuiltInVirtualColumn.CREATIONTIME,
segmentMetadata);
+ assertEquals(creationTimeMetadata.getTotalDocs(), NUM_DOCS);
+ assertEquals(creationTimeMetadata.getCardinality(), 1);
+ assertTrue(creationTimeMetadata.isSorted());
+ assertTrue(creationTimeMetadata.hasDictionary());
+ assertEquals(creationTimeMetadata.getMinValue(), CREATION_TIME_MS);
+ assertEquals(creationTimeMetadata.getMaxValue(), CREATION_TIME_MS);
+
+ ColumnMetadata crcMetadata = buildColumnMetadata(schema,
BuiltInVirtualColumn.CRC, segmentMetadata);
+ assertEquals(crcMetadata.getMinValue(), CRC);
+ assertEquals(crcMetadata.getMaxValue(), CRC);
+
+ // When the metadata is available the column carries a real value, so
there is no null vector
+ for (String column : List.of(BuiltInVirtualColumn.CREATIONTIME,
BuiltInVirtualColumn.STARTTIME,
+ BuiltInVirtualColumn.ENDTIME, BuiltInVirtualColumn.TOTALDOCS,
BuiltInVirtualColumn.CRC)) {
+ assertNull(buildNullValueVector(schema, column, segmentMetadata), column
+ " should not be null");
+ }
+ }
+
+ /// A segment without a time range and without a CRC must fall back to the
default null value for those columns,
+ /// while `$creationTime` and `$totalDocs` remain meaningful. An unset
creation time is `Long.MIN_VALUE` in the
+ /// segment metadata of an immutable segment and `-1` in the ZK metadata a
CONSUMING segment is created from.
+ @Test(dataProvider = "unsetCreationTimes")
+ public void testUnavailableMetadataFallsBackToDefaultNullValue(long
unsetCreationTime) {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata =
consumingSegmentMetadata(unsetCreationTime);
+
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CREATIONTIME,
segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.STARTTIME,
segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.ENDTIME,
segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CRC,
segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ // The stored value is only a placeholder: the column must additionally
report every document as null
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CREATIONTIME,
segmentMetadata);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.STARTTIME,
segmentMetadata);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.ENDTIME, segmentMetadata);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CRC, segmentMetadata);
+
+ // $totalDocs comes from the context, so it is always available and never
null
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.TOTALDOCS,
segmentMetadata).getIntValue(0), NUM_DOCS);
+ assertNull(buildNullValueVector(schema, BuiltInVirtualColumn.TOTALDOCS,
segmentMetadata));
+ }
+
+ /// Segment pruners read `ColumnMetadata` min/max without consulting the
null value vector, so a column whose value
+ /// is unavailable must not publish the placeholder as its min/max.
Otherwise a CONSUMING segment reporting epoch 0
+ /// for `$creationTime` sorts first in `ORDER BY $creationTime ASC LIMIT n`
and prunes away the committed segments
+ /// that actually hold the answer.
+ @Test
+ public void testUnavailableMetadataPublishesNoMinMax() {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata = consumingSegmentMetadata(-1L);
+ for (String column : List.of(BuiltInVirtualColumn.CREATIONTIME,
BuiltInVirtualColumn.STARTTIME,
+ BuiltInVirtualColumn.ENDTIME, BuiltInVirtualColumn.CRC)) {
+ ColumnMetadata columnMetadata = buildColumnMetadata(schema, column,
segmentMetadata);
+ assertNull(columnMetadata.getMinValue(), "Unavailable " + column + "
should not publish a min value");
+ assertNull(columnMetadata.getMaxValue(), "Unavailable " + column + "
should not publish a max value");
+
+ // The same has to hold through buildDataSource, which is the path a
mutable segment takes
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, segmentMetadata);
+ DataSourceMetadata dataSourceMetadata =
+
VirtualColumnProviderFactory.buildProvider(context).buildDataSource(context).getDataSourceMetadata();
+ assertNull(dataSourceMetadata.getMinValue(), column);
+ assertNull(dataSourceMetadata.getMaxValue(), column);
+ }
+
+ // A column whose value IS available still publishes min/max, so pruning
keeps working
+ ColumnMetadata available = buildColumnMetadata(schema,
BuiltInVirtualColumn.CREATIONTIME, mockSegmentMetadata());
+ assertEquals(available.getMinValue(), CREATION_TIME_MS);
+ assertEquals(available.getMaxValue(), CREATION_TIME_MS);
+ }
+
+ /// A segment that has not indexed anything yet still has to produce a
well-formed, empty null bitmap rather than
+ /// failing or reporting a stale document count.
+ @Test
+ public void testNullValueVectorOnAnEmptySegment() {
+ Schema schema = buildSegmentSchema();
+ FieldSpec fieldSpec =
schema.getFieldSpecFor(BuiltInVirtualColumn.STARTTIME);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, 0,
consumingSegmentMetadata(-1L));
+ NullValueVectorReader nullValueVector =
+
VirtualColumnProviderFactory.buildProvider(context).buildNullValueVector(context);
+ assertNotNull(nullValueVector);
+ assertTrue(nullValueVector.getNullBitmap().isEmpty());
+ }
+
+ /// The null value vector is built on every data source access for a mutable
segment, so the bitmap is materialized
+ /// lazily. Repeated reads must return an equal - and stable - bitmap.
+ @Test
+ public void testNullValueVectorBitmapIsStableAcrossReads() {
+ Schema schema = buildSegmentSchema();
+ FieldSpec fieldSpec = schema.getFieldSpecFor(BuiltInVirtualColumn.ENDTIME);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, consumingSegmentMetadata(-1L));
+ NullValueVectorReader nullValueVector =
+
VirtualColumnProviderFactory.buildProvider(context).buildNullValueVector(context);
+ assertNotNull(nullValueVector);
+ assertEquals(nullValueVector.getNullBitmap(),
nullValueVector.getNullBitmap());
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(), NUM_DOCS);
+ }
+
+ /// The index container must hand the null value vector to the engine, and
must not hand one back for an index type
+ /// a virtual column does not have.
+ @Test
+ public void testIndexContainerExposesTheNullValueVector()
+ throws IOException {
+ Schema schema = buildSegmentSchema();
+ FieldSpec fieldSpec = schema.getFieldSpecFor(BuiltInVirtualColumn.CRC);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, consumingSegmentMetadata(-1L));
+ try (ColumnIndexContainer container =
+
VirtualColumnProviderFactory.buildProvider(context).buildColumnIndexContainer(context))
{
+ assertNotNull(container.getIndex(StandardIndexes.nullValueVector()));
+ assertNotNull(container.getIndex(StandardIndexes.forward()));
+ assertNotNull(container.getIndex(StandardIndexes.dictionary()));
+ assertNull(container.getIndex(StandardIndexes.range()));
+ }
+
+ // With the metadata available there is no null value vector at all
+ VirtualColumnContext availableContext = new
VirtualColumnContext(fieldSpec, NUM_DOCS, mockSegmentMetadata());
+ try (ColumnIndexContainer container =
+
VirtualColumnProviderFactory.buildProvider(availableContext).buildColumnIndexContainer(availableContext))
{
+ assertNull(container.getIndex(StandardIndexes.nullValueVector()));
+ }
+ }
+
+ /// Every stored type the constant-value base supports must round-trip
through the type check, so that a new virtual
+ /// column of any type is rejected loudly rather than silently mis-cast.
+ @Test(dataProvider = "storedTypeValues")
+ public void testValueTypeCheckAcceptsEveryStoredType(FieldSpec.DataType
dataType, Object value) {
+ FieldSpec fieldSpec = new DimensionFieldSpec("col", dataType, true);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, null);
+ BaseConstantValueVirtualColumnProvider provider = new
BaseConstantValueVirtualColumnProvider() {
+ @Override
+ protected Object getValue(VirtualColumnContext ctx) {
+ return value;
+ }
+ };
+ assertNotNull(provider.buildDictionary(context));
+ assertNotNull(provider.buildMetadata(context));
+
+ // ... and the same type check rejects a value of the wrong type
+ BaseConstantValueVirtualColumnProvider wrong = new
BaseConstantValueVirtualColumnProvider() {
+ @Override
+ protected Object getValue(VirtualColumnContext ctx) {
+ return new Object();
+ }
+ };
+ try {
+ wrong.buildDictionary(context);
+ fail("Expecting an IllegalStateException for data type: " + dataType);
+ } catch (IllegalStateException e) {
+ assertTrue(e.getMessage().contains("col"), e.getMessage());
+ }
+ }
+
+ @DataProvider(name = "storedTypeValues")
+ public Object[][] storedTypeValues() {
+ return new Object[][]{
+ {FieldSpec.DataType.INT, 1},
+ {FieldSpec.DataType.LONG, 1L},
+ {FieldSpec.DataType.FLOAT, 1.0f},
+ {FieldSpec.DataType.DOUBLE, 1.0d},
+ {FieldSpec.DataType.BIG_DECIMAL, BigDecimal.ONE},
+ {FieldSpec.DataType.STRING, "value"},
+ {FieldSpec.DataType.BYTES, new byte[]{1, 2}},
+ {FieldSpec.DataType.TIMESTAMP, 1L}
+ };
+ }
+
+ /// The type check exists so that a provider returning the wrong box type
names the column and the provider instead
+ /// of throwing a bare ClassCastException from deep inside segment loading.
+ @Test
+ public void testWrongValueTypeIsRejectedWithADiagnosticMessage() {
+ FieldSpec fieldSpec =
buildSegmentSchema().getFieldSpecFor(BuiltInVirtualColumn.CREATIONTIME);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec,
NUM_DOCS, mockSegmentMetadata());
+ // A LONG-stored column handed an Integer
+ BaseConstantValueVirtualColumnProvider provider = new
BaseConstantValueVirtualColumnProvider() {
+ @Override
+ protected Object getValue(VirtualColumnContext ctx) {
+ return 1;
+ }
+ };
+ try {
+ provider.buildDictionary(context);
+ fail("Expecting an IllegalStateException for a wrongly typed virtual
column value");
+ } catch (IllegalStateException e) {
+ String message = e.getMessage();
+ assertTrue(message.contains(BuiltInVirtualColumn.CREATIONTIME), message);
+ assertTrue(message.contains("java.lang.Integer"), message);
+ assertTrue(message.contains("java.lang.Long"), message);
+ }
+ }
+
+ /// The segment metadata is not always available (e.g. when the virtual
column is built for a column that is missing
+ /// from the segment). The providers must not fail in that case.
+ @Test
+ public void testMissingSegmentMetadataFallsBackToDefaultNullValue() {
+ Schema schema = buildSegmentSchema();
+
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CREATIONTIME,
null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.STARTTIME,
null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.ENDTIME,
null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CRC,
null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.TOTALDOCS,
null).getIntValue(0), NUM_DOCS);
+
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CREATIONTIME, null);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.STARTTIME, null);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.ENDTIME, null);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CRC, null);
+ }
+}
diff --git
a/pinot-spi/src/main/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitions.java
b/pinot-spi/src/main/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitions.java
new file mode 100644
index 00000000000..ab3903468bc
--- /dev/null
+++
b/pinot-spi/src/main/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitions.java
@@ -0,0 +1,107 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
+
+
+/// Single source of truth for the shape (name, data type, single-value vs
multi-value) of the built-in virtual columns.
+///
+/// The columns are declared here once because they are materialized in two
independent places that must agree:
+///
+/// - the broker/controller side adds them to the table schema used for query
planning, without a provider class
+/// (see `TableCache#addBuiltInVirtualColumns`), and
+/// - the server side adds them to the segment schema together with the
provider that produces the values
+/// (see
`VirtualColumnProviderFactory#addBuiltInVirtualColumnsToSegmentSchema`).
+///
+/// If the two sides disagreed on a data type or on single-value vs
multi-value, the broker would declare one type
+/// while the server produced another. Both sides therefore build their field
specs from [#DEFINITIONS].
+///
+/// The column *names* additionally live in [BuiltInVirtualColumn], which is
where code that only needs to recognize a
+/// built-in virtual column by name looks them up. [#NAMES] exposes the names
derived from [#DEFINITIONS] so the two
+/// can be asserted equal in a test.
+public class BuiltInVirtualColumnDefinitions {
+ private BuiltInVirtualColumnDefinitions() {
+ }
+
+ /// Shape of a single built-in virtual column. The provider class is
intentionally not part of this definition: it
+ /// lives in `pinot-segment-local` and is only known to the server side.
+ public static class Definition {
+ private final String _name;
+ private final DataType _dataType;
+ private final boolean _singleValueField;
+
+ private Definition(String name, DataType dataType, boolean
singleValueField) {
+ _name = name;
+ _dataType = dataType;
+ _singleValueField = singleValueField;
+ }
+
+ public String getName() {
+ return _name;
+ }
+
+ public DataType getDataType() {
+ return _dataType;
+ }
+
+ public boolean isSingleValueField() {
+ return _singleValueField;
+ }
+
+ /// Creates a new field spec for this column, without a virtual column
provider. Callers that can resolve a
+ /// provider should set it on the returned spec.
+ ///
+ /// NOTE: Returns a fresh instance on every call. Field specs are mutable
and are stored by reference in the
+ /// schema they are added to, so they must never be shared across schemas.
+ public DimensionFieldSpec createFieldSpec() {
+ return new DimensionFieldSpec(_name, _dataType, _singleValueField);
+ }
+ }
+
+ public static final List<Definition> DEFINITIONS = List.of(
+ new Definition(BuiltInVirtualColumn.DOCID, DataType.INT, true),
+ new Definition(BuiltInVirtualColumn.HOSTNAME, DataType.STRING, true),
+ new Definition(BuiltInVirtualColumn.SEGMENTNAME, DataType.STRING, true),
+ new Definition(BuiltInVirtualColumn.PARTITIONID, DataType.STRING, false),
+ new Definition(BuiltInVirtualColumn.CREATIONTIME, DataType.TIMESTAMP,
true),
+ new Definition(BuiltInVirtualColumn.STARTTIME, DataType.TIMESTAMP, true),
+ new Definition(BuiltInVirtualColumn.ENDTIME, DataType.TIMESTAMP, true),
+ new Definition(BuiltInVirtualColumn.TOTALDOCS, DataType.INT, true),
+ new Definition(BuiltInVirtualColumn.CRC, DataType.LONG, true));
+
+ /// Names of all the built-in virtual columns, derived from [#DEFINITIONS].
+ public static final Set<String> NAMES =
+
DEFINITIONS.stream().map(Definition::getName).collect(Collectors.toUnmodifiableSet());
+
+ /// Adds the built-in virtual columns to the given schema, without a virtual
column provider.
+ ///
+ /// Existing columns are left untouched, so a user-defined column of the
same name always wins.
+ public static void addToSchema(Schema schema) {
+ for (Definition definition : DEFINITIONS) {
+ if (!schema.hasColumn(definition.getName())) {
+ schema.addField(definition.createFieldSpec());
+ }
+ }
+ }
+}
diff --git a/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java
b/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java
index a0364117b6a..fa7c2de0335 100644
--- a/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java
+++ b/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java
@@ -66,8 +66,17 @@ public class SchemaInfo {
public SchemaInfo(Schema schema) {
_schemaName = schema.getSchemaName();
- //Removed virtual columns($docId, $hostName, $segmentName) from dimension
fields count
- _numDimensionFields = schema.getDimensionFieldSpecs().size() - 3;
+ // The schema handed in here has usually been through
TableCache#addBuiltInVirtualColumns, which adds the built-in
+ // virtual columns as plain dimension fields (without a provider class, so
isVirtualColumn() does not identify
+ // them). Exclude them by name so that the reported count only covers the
user's own dimensions, and stays correct
+ // as built-in virtual columns are added.
+ int numDimensionFields = 0;
+ for (DimensionFieldSpec fieldSpec : schema.getDimensionFieldSpecs()) {
+ if
(!BuiltInVirtualColumnDefinitions.NAMES.contains(fieldSpec.getName())) {
+ numDimensionFields++;
+ }
+ }
+ _numDimensionFields = numDimensionFields;
_numDateTimeFields = schema.getDateTimeFieldSpecs().size();
_numMetricFields = schema.getMetricFieldSpecs().size();
_numComplexFields = schema.getComplexFieldSpecs().size();
diff --git
a/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
b/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
index e00598b582e..c117713b8c8 100644
--- a/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
+++ b/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
@@ -2246,7 +2246,36 @@ public class CommonConstants {
public static final String HOSTNAME = "$hostName";
public static final String SEGMENTNAME = "$segmentName";
public static final String PARTITIONID = "$partitionId";
- public static final Set<String> BUILT_IN_VIRTUAL_COLUMNS = Set.of(DOCID,
HOSTNAME, SEGMENTNAME, PARTITIONID);
+
+ // Segment metadata virtual columns. Each of them is a constant
single-value column within a segment, exposing a
+ // piece of the segment metadata to queries. When the underlying
metadata is not available (e.g. on a CONSUMING
+ // segment, which has no time range and no CRC yet), the column reads as
NULL.
+ //
+ // The three time columns are TIMESTAMP rather than LONG, so the unit is
carried by the type instead of by the
+ // column name, and query results render them as readable timestamps.
+
+ /// Segment creation time (TIMESTAMP). This is the index creation time
recorded in the segment's creation
+ /// metadata; for a CONSUMING segment it is the time the consuming
segment was created.
+ public static final String CREATIONTIME = "$creationTime";
+ /// Start of the segment time range (TIMESTAMP), normalized from the
time column's own unit.
+ /// NULL for segments without a time range, such as CONSUMING segments
and tables without a time column.
+ public static final String STARTTIME = "$startTime";
+ /// End of the segment time range (TIMESTAMP), normalized from the time
column's own unit.
+ /// NULL for segments without a time range, such as CONSUMING segments
and tables without a time column.
+ public static final String ENDTIME = "$endTime";
+ /// Number of documents in the segment (INT). On a CONSUMING segment
this is the number of documents indexed so
+ /// far. NOTE: This counts all the documents physically stored in the
segment, so for an upsert table it also
+ /// includes the documents that have been replaced and are no longer
returned by queries.
+ public static final String TOTALDOCS = "$totalDocs";
+ /// Segment CRC (LONG). NULL on CONSUMING segments, which have no CRC
until they are committed.
+ /// NOTE: Do not confuse this constant with the enclosing [Segment#CRC],
which is the `segment.crc` metadata key.
+ public static final String CRC = "$crc";
+
+ /// NOTE: Kept in sync with
`BuiltInVirtualColumnDefinitions#DEFINITIONS`, which additionally carries the
data
+ /// type and single-value/multi-value shape of each column.
`BuiltInVirtualColumnDefinitionsTest` asserts the two
+ /// agree.
+ public static final Set<String> BUILT_IN_VIRTUAL_COLUMNS =
+ Set.of(DOCID, HOSTNAME, SEGMENTNAME, PARTITIONID, CREATIONTIME,
STARTTIME, ENDTIME, TOTALDOCS, CRC);
}
}
diff --git
a/pinot-spi/src/test/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitionsTest.java
b/pinot-spi/src/test/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitionsTest.java
new file mode 100644
index 00000000000..c8cc253d808
--- /dev/null
+++
b/pinot-spi/src/test/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitionsTest.java
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data;
+
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNotSame;
+import static org.testng.Assert.assertTrue;
+
+
+public class BuiltInVirtualColumnDefinitionsTest {
+
+ /// The names are declared twice - once as constants in
[BuiltInVirtualColumn] and once implicitly by
+ /// [BuiltInVirtualColumnDefinitions#DEFINITIONS]. A definition missing from
the name set would stop being filtered
+ /// out of a segment's physical columns and would be counted as a user
dimension by [SchemaInfo].
+ @Test
+ public void testDefinitionsMatchTheDeclaredNames() {
+ assertEquals(BuiltInVirtualColumnDefinitions.NAMES,
BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS);
+ assertEquals(BuiltInVirtualColumnDefinitions.DEFINITIONS.size(),
+ BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS.size(), "Duplicate name
in DEFINITIONS");
+ }
+
+ /// Every built-in virtual column name must start with `$`: that prefix is
what excludes them from `SELECT *` in
+ /// both query engines.
+ @Test
+ public void testNamesAreDollarPrefixed() {
+ for (String name : BuiltInVirtualColumnDefinitions.NAMES) {
+ assertTrue(name.startsWith("$"), "Built-in virtual column must be
$-prefixed: " + name);
+ }
+ }
+
+ /// Field specs are mutable and are stored by reference in the schema they
are added to, so each call must return a
+ /// fresh instance - the server side mutates them to attach the provider
class and, for `$segmentName`, a per-segment
+ /// value.
+ @Test
+ public void testCreateFieldSpecReturnsFreshInstances() {
+ for (BuiltInVirtualColumnDefinitions.Definition definition :
BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ DimensionFieldSpec first = definition.createFieldSpec();
+ DimensionFieldSpec second = definition.createFieldSpec();
+ assertNotSame(first, second, "Field spec must not be shared for: " +
definition.getName());
+ assertEquals(first, second);
+ assertEquals(first.getName(), definition.getName());
+ assertEquals(first.getDataType(), definition.getDataType());
+ assertEquals(first.isSingleValueField(),
definition.isSingleValueField());
+ }
+ }
+
+ @Test
+ public void testAddToSchemaIsIdempotentAndNeverOverwrites() {
+ Schema schema = new Schema.SchemaBuilder().setSchemaName("test")
+ .addSingleValueDimension("dim", FieldSpec.DataType.STRING)
+ .build();
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+ Set<String> afterFirst = new HashSet<>(schema.getColumnNames());
+ assertTrue(afterFirst.containsAll(BuiltInVirtualColumnDefinitions.NAMES));
+
+ // A second call must not duplicate or replace anything
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+ assertEquals(new HashSet<>(schema.getColumnNames()), afterFirst);
+
+ for (BuiltInVirtualColumnDefinitions.Definition definition :
BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(definition.getName());
+ assertNotNull(fieldSpec);
+ assertEquals(fieldSpec.getDataType(), definition.getDataType());
+ }
+ }
+
+ /// A user-defined column of the same name must win, so that adding a
built-in virtual column never silently
+ /// changes the type of an existing user column in the broker's schema.
+ @Test
+ public void testAddToSchemaDoesNotOverrideUserColumn() {
+ Schema schema = new Schema.SchemaBuilder().setSchemaName("test")
+ .addSingleValueDimension(BuiltInVirtualColumn.TOTALDOCS,
FieldSpec.DataType.STRING)
+ .build();
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+
assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.TOTALDOCS).getDataType(),
FieldSpec.DataType.STRING);
+ }
+}
diff --git
a/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java
b/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java
index d3ee0e31aeb..4266bbc2e5a 100644
--- a/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java
+++ b/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java
@@ -34,6 +34,30 @@ import static org.testng.Assert.assertEquals;
public class SchemaInfoTest {
+ /// The schema `SchemaInfo` is built from has usually been through the
broker's `addBuiltInVirtualColumns`, which adds
+ /// every built-in virtual column as a plain dimension field. None of them
may be reported as a user dimension,
+ /// whatever their number.
+ @Test
+ public void testBuiltInVirtualColumnsAreNotCounted() {
+ Schema schema = new Schema.SchemaBuilder().setSchemaName("TestSchema")
+ .addDimensionField("dim1", FieldSpec.DataType.STRING)
+ .addDimensionField("dim2", FieldSpec.DataType.INT)
+ .addDateTimeField("dt1", FieldSpec.DataType.LONG, "1:HOURS:EPOCH",
"1:HOURS")
+ .addMetricField("metric", INT)
+ .build();
+ assertEquals(new SchemaInfo(schema).getNumDimensionFields(), 2);
+
+ // Adding the built-in virtual columns must not change the reported
dimension count
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+ assertEquals(schema.getDimensionFieldSpecs().size(),
+ 2 +
CommonConstants.Segment.BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS.size());
+ SchemaInfo schemaInfo = new SchemaInfo(schema);
+ assertEquals(schemaInfo.getNumDimensionFields(), 2);
+ assertEquals(schemaInfo.getNumDateTimeFields(), 1);
+ assertEquals(schemaInfo.getNumMetricFields(), 1);
+ assertEquals(schemaInfo.getNumComplexFields(), 0);
+ }
+
@Test
public void testSchemaInfoSerDeserWithVirtualColumns()
throws IOException {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
