This is an automated email from the ASF dual-hosted git repository.
ColinLeeo pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/tsfile.git
The following commit(s) were added to refs/heads/develop by this push:
new e5e4cccfd [Python][C++] Fix tree-model TsFileDataFrame returning empty
data on reused readers and uppercase names (#862)
e5e4cccfd is described below
commit e5e4cccfd192d35207747e0458767dcf5a3908d9
Author: Le Yang <[email protected]>
AuthorDate: Thu Jul 16 14:43:07 2026 +0800
[Python][C++] Fix tree-model TsFileDataFrame returning empty data on reused
readers and uppercase names (#862)
* fix: correct tree-model TsFileDataFrame reads on reused readers and
uppercase names
Two related fixes for reading tree-model TsFiles through TsFileDataFrame,
which together caused 'read one series OK, then reading another series returns
empty' on a reused reader.
C++: make StringArrayDeviceID::split_table_name()/init_prefix_segments()
idempotent. Device IDs are cached and reused across queries; the previous code
appended to prefix_segments_ on every call, so prefixes accumulated (and
leaked) across successive queries and corrupted per-device path columns, making
a second series read match nothing. Now it clears/frees existing prefixes
before rebuilding. Adds DeviceIdTest.SplitTableNameIsIdempotent.
Python: read series with uppercase measurement/path names.
query_table_on_tree synthesizes a case-insensitive table schema, so the native
layer lower-cases column names and path segments, while the dataset catalog
keeps the original casing. The tree read paths matched field columns and device
paths case-sensitively and returned empty for uppercase names.
_read_series_by_row_tree and _read_arrow_tree now match case-insensitively.
Adds a regression test.
* style(python): apply spotless (black) formatting to tree-model regression
test
* revert(python): drop case-insensitive tree-model matching; add
case-collision regression test
Revert the .lower() case-insensitive column/path matching in the tree read
paths (from d13cd236). It let single-variant uppercase measurements read, but
conflated case-distinct measurements (e.g. temperature vs Temperature) into one
series. Restore case-sensitive matching (native still lower-cases names, so
uppercase reads are empty for now) and add a regression test asserting
case-distinct measurements do not collide. Two tree-model tests are
intentionally red pending the C++ root fi [...]
* fix(cpp): preserve case for tree-derived (virtual) table schemas
Tree-model device paths and measurement names are case-sensitive, but the
virtual TableSchema synthesized in query_on_tree lower-cased them via the
TableSchema constructor. This collapsed case-distinct series (e.g.
"temperature" vs "Temperature") into one column, so reads returned empty
or mixed data.
- TableSchema: add virtual_table flag; skip lower-casing table/measurement
names and keep column_pos_index_ in original case for virtual tables.
- find_column_index/find_id_column_order: match case-sensitively for
virtual tables, unchanged (lower-cased) for real tables.
- query_on_tree: build the synthesized schema with virtual_table=true.
* docs(cpp): trim case-sensitivity comments in TableSchema
* spotless
---
cpp/src/common/device_id.cc | 8 ++
cpp/src/common/schema.h | 35 ++++---
cpp/src/reader/table_query_executor.cc | 8 +-
cpp/test/common/device_id_test.cc | 20 ++++
python/tests/test_tsfile_dataset.py | 164 +++++++++++++++++++++++++++++++++
5 files changed, 221 insertions(+), 14 deletions(-)
diff --git a/cpp/src/common/device_id.cc b/cpp/src/common/device_id.cc
index e88cdac8a..0b5e6c107 100644
--- a/cpp/src/common/device_id.cc
+++ b/cpp/src/common/device_id.cc
@@ -103,6 +103,13 @@ std::string StringArrayDeviceID::get_device_name() const {
}
void StringArrayDeviceID::init_prefix_segments() {
+ // Idempotent: device IDs are cached and reused across queries, so clear
+ // previous prefixes before rebuilding to avoid accumulation and leaks.
+ for (const auto& prefix_segment : prefix_segments_) {
+ delete prefix_segment;
+ }
+ prefix_segments_.clear();
+
#ifdef ENABLE_ANTLR4
auto splits = storage::PathNodesGenerator::invokeParser(*segments_[0]);
#else
@@ -130,6 +137,7 @@ int StringArrayDeviceID::serialize(common::ByteStream&
write_stream) {
int StringArrayDeviceID::deserialize(common::ByteStream& read_stream) {
int ret = common::E_OK;
+
uint32_t num_segments;
if (RET_FAIL(common::SerializationUtil::read_var_uint(num_segments,
read_stream))) {
diff --git a/cpp/src/common/schema.h b/cpp/src/common/schema.h
index 81008b715..827ae6d83 100644
--- a/cpp/src/common/schema.h
+++ b/cpp/src/common/schema.h
@@ -198,9 +198,16 @@ class TableSchema {
* in the table.
*/
TableSchema(const std::string& table_name,
- const std::vector<common::ColumnSchema>& column_schemas)
- : table_name_(table_name), updatable_(false) {
- to_lowercase_inplace(table_name_);
+ const std::vector<common::ColumnSchema>& column_schemas,
+ bool virtual_table = false)
+ : table_name_(table_name),
+ is_virtual_table_(virtual_table),
+ updatable_(false) {
+ // Virtual (tree-derived) schemas are case-sensitive; only real tables
+ // are normalized to lower case.
+ if (!is_virtual_table_) {
+ to_lowercase_inplace(table_name_);
+ }
for (const common::ColumnSchema& column_schema : column_schemas) {
column_schemas_.emplace_back(std::make_shared<MeasurementSchema>(
column_schema.get_column_name(),
@@ -210,7 +217,9 @@ class TableSchema {
}
int idx = 0;
for (const auto& measurement_schema : column_schemas_) {
- to_lowercase_inplace(measurement_schema->measurement_name_);
+ if (!is_virtual_table_) {
+ to_lowercase_inplace(measurement_schema->measurement_name_);
+ }
column_pos_index_.insert(
std::make_pair(measurement_schema->measurement_name_, idx++));
}
@@ -325,21 +334,23 @@ class TableSchema {
int32_t get_columns_num() const { return column_schemas_.size(); }
int find_column_index(const std::string& column_name) {
- std::string lower_case_column_name = to_lower(column_name);
- auto it = column_pos_index_.find(lower_case_column_name);
+ // Virtual tables match case-sensitively; real tables are lower-cased.
+ std::string lookup_name =
+
+ is_virtual_table_ ? column_name : to_lower(column_name);
+ auto it = column_pos_index_.find(lookup_name);
if (it != column_pos_index_.end()) {
return it->second;
} else {
int index = -1;
for (size_t i = 0; i < column_schemas_.size(); ++i) {
- if (column_schemas_[i]->measurement_name_ ==
- lower_case_column_name) {
+ if (column_schemas_[i]->measurement_name_ == lookup_name) {
index = static_cast<int>(i);
break;
}
}
if (index != -1) {
- column_pos_index_[lower_case_column_name] = index;
+ column_pos_index_[lookup_name] = index;
}
return index;
}
@@ -440,12 +451,12 @@ class TableSchema {
}
int32_t find_id_column_order(const std::string& column_name) {
- std::string lower_case_column_name = to_lower(column_name);
+ std::string lookup_name =
+ is_virtual_table_ ? column_name : to_lower(column_name);
int column_order = 0;
for (size_t i = 0; i < column_schemas_.size(); ++i) {
- if (column_schemas_[i]->measurement_name_ ==
- lower_case_column_name &&
+ if (column_schemas_[i]->measurement_name_ == lookup_name &&
column_categories_[i] == common::ColumnCategory::TAG) {
return column_order;
} else if (column_categories_[i] == common::ColumnCategory::TAG) {
diff --git a/cpp/src/reader/table_query_executor.cc
b/cpp/src/reader/table_query_executor.cc
index d5145104d..3983f8540 100644
--- a/cpp/src/reader/table_query_executor.cc
+++ b/cpp/src/reader/table_query_executor.cc
@@ -233,8 +233,12 @@ int TableQueryExecutor::query_on_tree(
}
}
- auto schema = std::make_shared<TableSchema>("default", col_schema);
- schema->set_virtual_table();
+ // Tree-derived (virtual) tables are case-sensitive: keep the original
+ // casing of device path segments and measurement names so that
+ // case-distinct series (e.g. "temperature" vs "Temperature") map to
+ // separate columns instead of colliding.
+ auto schema = std::make_shared<TableSchema>("default", col_schema,
+ /*virtual_table=*/true);
std::shared_ptr<ColumnMapping> column_mapping =
std::make_shared<ColumnMapping>();
for (size_t i = 0; i < col_schema.size(); ++i) {
diff --git a/cpp/test/common/device_id_test.cc
b/cpp/test/common/device_id_test.cc
index 9d97607ab..839258f67 100644
--- a/cpp/test/common/device_id_test.cc
+++ b/cpp/test/common/device_id_test.cc
@@ -99,4 +99,24 @@ TEST(DeviceIdTest, NullTagVsLiteralNullAreDistinct) {
ASSERT_FALSE(null_first == literal_null);
ASSERT_TRUE(null_first != literal_null);
}
+
+// Regression: cached device IDs are reused across queries, so
+// split_table_name() must be idempotent and not accumulate prefix segments.
+TEST(DeviceIdTest, SplitTableNameIsIdempotent) {
+ StringArrayDeviceID device_id("root.ln.wf01.wt01");
+
+ const std::vector<std::string> expected = {"root", "ln", "wf01", "wt01"};
+
+ for (int round = 0; round < 3; ++round) {
+ device_id.split_table_name();
+
+ ASSERT_EQ(static_cast<int>(expected.size()),
+ device_id.get_split_seg_num());
+ for (int i = 0; i < device_id.get_split_seg_num(); ++i) {
+ std::string* seg = device_id.get_split_segname_at(i);
+ ASSERT_NE(nullptr, seg);
+ ASSERT_EQ(expected[static_cast<size_t>(i)], *seg);
+ }
+ }
+}
} // namespace storage
diff --git a/python/tests/test_tsfile_dataset.py
b/python/tests/test_tsfile_dataset.py
index 12b847e9e..ee016d087 100644
--- a/python/tests/test_tsfile_dataset.py
+++ b/python/tests/test_tsfile_dataset.py
@@ -1507,12 +1507,176 @@ def test_dataset_tree_model_series_access(tmp_path):
np.testing.assert_array_equal(aligned.timestamps, np.arange(5,
dtype=np.int64))
+def test_dataset_tree_model_reads_uppercase_measurement_names(tmp_path):
+ """Tree-model series with uppercase measurement names must read data.
+
+ Regression: a measurement like ``Temperature``/``STATUS`` must return its
+ values, not an empty array, when read back through TsFileDataFrame.
+ """
+ from tsfile import Field, RowRecord, TimeseriesSchema, TsFileWriter
+
+ path = tmp_path / "tree_upper.tsfile"
+ writer = TsFileWriter(str(path))
+ writer.register_timeseries(
+ "root.ln.wf01.wt01", TimeseriesSchema("Temperature", TSDataType.DOUBLE)
+ )
+ writer.register_timeseries(
+ "root.ln.wf01.wt01", TimeseriesSchema("STATUS", TSDataType.INT32)
+ )
+ for t in range(5):
+ writer.write_row_record(
+ RowRecord(
+ "root.ln.wf01.wt01",
+ t,
+ [
+ Field("Temperature", float(t) + 0.5, TSDataType.DOUBLE),
+ Field("STATUS", t * 2, TSDataType.INT32),
+ ],
+ )
+ )
+ writer.close()
+
+ with TsFileDataFrame(str(path), show_progress=False) as tsdf:
+ assert sorted(tsdf.list_timeseries()) == [
+ "root.ln.wf01.wt01.STATUS",
+ "root.ln.wf01.wt01.Temperature",
+ ]
+ np.testing.assert_array_equal(
+ tsdf["root.ln.wf01.wt01.Temperature"][:],
+ np.array([0.5, 1.5, 2.5, 3.5, 4.5]),
+ )
+ np.testing.assert_array_equal(
+ tsdf["root.ln.wf01.wt01.STATUS"][:],
+ np.array([0.0, 2.0, 4.0, 6.0, 8.0]),
+ )
+
+
+def
test_dataset_tree_model_case_distinct_measurements_do_not_collide(tmp_path,
capsys):
+ """Case-distinct measurements on one device must not be conflated.
+
+ ``temperature`` and ``Temperature`` are two independent series; each must
+ return its own values instead of an empty array or the other's data.
+ """
+ from tsfile import Field, RowRecord, TimeseriesSchema, TsFileWriter
+
+ path = tmp_path / "tree_case.tsfile"
+
+ writer = TsFileWriter(str(path))
+ writer.register_timeseries(
+ "root.case.d1", TimeseriesSchema("temperature", TSDataType.DOUBLE)
+ )
+ writer.register_timeseries(
+ "root.case.d1", TimeseriesSchema("Temperature", TSDataType.DOUBLE)
+ )
+ for t in range(3):
+ writer.write_row_record(
+ RowRecord(
+ "root.case.d1",
+ t,
+ [
+ Field("temperature", 30.0 + t, TSDataType.DOUBLE),
+ Field("Temperature", 40.0 + t, TSDataType.DOUBLE),
+ ],
+ )
+ )
+ writer.close()
+
+ with TsFileDataFrame(str(path), show_progress=False) as tsdf:
+ # Two independent tree-model series are discovered (case preserved).
+ assert len(tsdf) == 2
+ assert sorted(tsdf.list_timeseries()) == [
+ "root.case.d1.Temperature",
+ "root.case.d1.temperature",
+ ]
+
+ # Metadata layer keeps both rows distinct with their original casing.
+ meta = tsdf.list_timeseries_metadata()
+ assert sorted(meta.index.tolist()) == [
+ "root.case.d1.Temperature",
+ "root.case.d1.temperature",
+ ]
+ for name in ("root.case.d1.temperature", "root.case.d1.Temperature"):
+ assert meta.loc[name, "_col_1"] == "case"
+ assert meta.loc[name, "_col_2"] == "d1"
+ assert meta.loc[name, "count"] == 3
+ assert set(meta["field"]) == {"temperature", "Temperature"}
+
+ # print(tsdf) renders both case-distinct series verbatim. The timestamp
+ # text is derived from format_timestamp so the whole-string match stays
+ # stable across timezones.
+ t0, t2 = format_timestamp(0), format_timestamp(2)
+ expected_repr = "\n".join(
+ [
+ "TsFileDataFrame(tree model, 2 time series, 1 files)",
+ " _col_1 _col_2 field start_time
end_time count",
+ f"0 case d1 Temperature {t0} {t2} 3",
+ f"1 case d1 temperature {t0} {t2} 3",
+ ]
+ )
+ assert repr(tsdf) == expected_repr
+ print(tsdf)
+ assert capsys.readouterr().out == expected_repr + "\n"
+
+ # Each series must read back its OWN values -- not empty, not the
+ # other's data. Lower-casing anywhere in the read path breaks this.
+ np.testing.assert_array_equal(
+ tsdf["root.case.d1.temperature"][:],
+ np.array([30.0, 31.0, 32.0]),
+ )
+ np.testing.assert_array_equal(
+ tsdf["root.case.d1.Temperature"][:],
+ np.array([40.0, 41.0, 42.0]),
+ )
+
+ # Aligned read of both case-distinct series keeps them separated.
+ aligned = tsdf.loc[
+ 0:2,
+ ["root.case.d1.temperature", "root.case.d1.Temperature"],
+ ]
+ assert aligned.series_names == [
+ "root.case.d1.temperature",
+ "root.case.d1.Temperature",
+ ]
+ np.testing.assert_array_equal(
+ aligned.values,
+ np.array([[30.0, 40.0], [31.0, 41.0], [32.0, 42.0]]),
+ )
+
+
+def test_tree_reader_handles_stale_path_columns_after_reused_queries(tmp_path):
+ """Reusing a reader must not leak prefix path state across queries.
+
+ Reading one device series then another reuses the cached device id; stale
+ prefix segments used to mismatch the device and return empty data.
+ """
+ path = tmp_path / "tree.tsfile"
+ _write_tree_file(path)
+
+ with TsFileDataFrame(str(path), show_progress=False) as tsdf:
+ # First read establishes query state on the reader.
+ np.testing.assert_array_equal(
+ tsdf["root.ln.wf01.wt01.temperature"][:],
+ np.array([0.5, 1.5, 2.5, 3.5, 4.5]),
+ )
+ # Second read reuses the same reader for another device.
+ np.testing.assert_array_equal(
+ tsdf["root.ln.wf02.wt02.status"][:],
+ np.array([0.0, 2.0, 4.0, 6.0, 8.0]),
+ )
+ # Read back the first series to confirm alternating reads stay stable.
+ np.testing.assert_array_equal(
+ tsdf["root.ln.wf01.wt01.temperature"][:],
+ np.array([0.5, 1.5, 2.5, 3.5, 4.5]),
+ )
+
+
def test_dataset_tree_model_list_timeseries_metadata(tmp_path):
path = tmp_path / "tree.tsfile"
_write_tree_file(path)
with TsFileDataFrame(str(path), show_progress=False) as tsdf:
meta = tsdf.list_timeseries_metadata()
+
assert isinstance(meta, pd.DataFrame)
assert list(meta.columns) == [
"field",
