ColinLeeo opened a new issue, #1038:
URL: https://github.com/apache/tsfile/issues/1038
<!--
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
-->
## Problem
A C++ aligned table query using `TimeIn` can silently omit matching rows
after decoding a page with no matching timestamps. The page decoder reports
`E_NO_MORE_DATA` for an empty filtered result even when the same chunk still
contains unread pages. The series iterator then skips the remainder of that
chunk or ends the series.
This was independently reproduced on the current default branch, `develop`,
at `eb733133b9eee0b1eb74f7671d809ecd730cedc3`. It is a pre-existing reader bug
found while reviewing #1026; please track its fix separately from that PR.
## Reproduction
1. Write an aligned table with TAG `id` and INT64 fields `v`, `w`, and
`ghost`, using PLAIN encoding and UNCOMPRESSED compression.
2. Use three devices, each with timestamps 0 through 20, written in three
groups of seven rows with an explicit flush after each group. Set
`page_writer_max_point_num_ = 5`.
3. Leave `ghost` unwritten, omit `v` entirely for one device, and include
nulls in `v`/`w`. Requesting the missing measurement makes the query use the
per-column aligned read path.
4. Use serial reading and a 4096-byte internal TsBlock budget, which
accommodates the decoder's 129-row batch for these single-column blocks.
5. Query `{"v", "ghost", "w", "v"}` with `TimeIn({0, 2, 3, 6, 7, 10, 13, 19,
20}, false)`, no field predicate, offset 0, and no limit.
Expected: 9 matching timestamps per device, 27 rows total. Null field values
must not remove rows that match the time predicate.
Actual: 7 timestamps per device, 21 rows total. Each device loses timestamps
19 and 20. The API returns `E_OK` and reports normal end of results; it does
not report an error.
```text
Expected per device: 0 2 3 6 7 10 13 19 20
Actual per device: 0 2 3 6 7 10 13
```
<details>
<summary>Standalone C++ reproducer</summary>
Save this as `timein_empty_page_repro.cc`. From the repository root, compile
it against the built C++ library, for example:
```sh
c++ -std=c++11 -I cpp/src timein_empty_page_repro.cc \
-L /path/to/build/lib -ltsfile -Wl,-rpath,/path/to/build/lib \
-o timein_empty_page_repro
./timein_empty_page_repro /tmp/timein-empty-page-repro.tsfile
```
The program prints `after scan ret=0 count=21 expected=27` and exits with
code 4 on the affected implementation. Correct behavior is 27 rows and exit
code 0.
```cpp
#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <string>
#include <vector>
#include "common/config/config.h"
#include "common/tablet.h"
#include "file/local_random_access_read_file.h"
#include "file/write_file.h"
#include "reader/filter/time_operator.h"
#include "reader/table_query_executor.h"
#include "writer/tsfile_table_writer.h"
using namespace common;
using namespace storage;
void check(int ret) { if (ret != E_OK) { std::cerr << "error=" << ret <<
'\n'; std::exit(2); } }
int main(int argc, char** argv) {
libtsfile_init();
const auto saved = g_config_value_;
g_config_value_.page_writer_max_point_num_ = 5;
g_config_value_.tsblock_max_memory_ = argc > 2 ? std::atoi(argv[2]) : 4096;
g_config_value_.parallel_read_enabled_ = false;
const std::string path = argc > 1 ? argv[1] :
"/tmp/timein-empty-page-repro.tsfile";
{
WriteFile file;
check(file.create(path, O_WRONLY | O_CREAT | O_TRUNC, 0666));
TableSchema schema("t", {
{"id", STRING, UNCOMPRESSED, PLAIN, ColumnCategory::TAG},
{"v", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
{"w", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
{"ghost", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD}});
TsFileTableWriter writer(&file, &schema);
for (int chunk = 0; chunk < 3; ++chunk) for (int d = 0; d < 3; ++d) {
std::vector<std::string> names{"id", "w"};
std::vector<TSDataType> types{STRING, INT64};
std::vector<ColumnCategory> categories{ColumnCategory::TAG,
ColumnCategory::FIELD};
if (d != 1) { names.push_back("v"); types.push_back(INT64);
categories.push_back(ColumnCategory::FIELD); }
Tablet tablet("t", names, types, categories, 7);
for (int r = 0; r < 7; ++r) {
int t = chunk * 7 + r;
check(tablet.add_timestamp(r, t));
const std::string id = "d" + std::to_string(d);
check(tablet.add_value(r, "id", id.c_str()));
if (t % 3 != 0) check(tablet.add_value(r, "w", int64_t(d * 1000 +
t)));
if (d != 1 && t % 4 != 0) check(tablet.add_value(r, "v", int64_t(d *
100 + t)));
}
check(writer.write_table(tablet));
check(writer.flush());
}
check(writer.close());
}
std::cerr << "before query memory=" << g_config_value_.tsblock_max_memory_
<< '\n';
LocalRandomAccessReadFile file;
check(file.open(path));
TableQueryExecutor executor(&file, -1);
ResultSet* result = nullptr;
check(executor.query("t", {"v", "ghost", "w", "v"}, new TimeIn({0, 2, 3,
6, 7, 10, 13, 19, 20}, false), nullptr, nullptr, 0, -1, result));
std::cerr << "after query before first next\n";
bool next = false;
int ret = E_OK, count = 0;
while ((ret = result->next(next)) == E_OK && next) {
++count;
std::cout << result->get_value<int64_t>(1) << " ";
if (count > 200) return 3;
}
std::cerr << "after scan ret=" << ret << " count=" << count << "
expected=27\n";
executor.destroy_query_data_set(result);
std::remove(path.c_str());
g_config_value_ = saved;
libtsfile_destroy();
return ret == E_OK && count == 27 ? 0 : 4;
}
```
</details>
## Root cause
The final chunk for each device has two time pages:
| Page | Timestamps | Required behavior |
| --- | --- | --- |
| First | 14 through 18 | No timestamps match; continue scanning the same
chunk |
| Second | 19 and 20 | Both timestamps match and must be returned |
[`AlignedChunkReader::decode_tv_buf_into_tsblock_by_datatype`](https://github.com/apache/tsfile/blob/eb733133b9eee0b1eb74f7671d809ecd730cedc3/cpp/src/reader/aligned_chunk_reader.cc#L1057-L1059)
changes a successful decode into `E_NO_MORE_DATA` whenever the output TsBlock
is empty:
```cpp
if (ret_tsblock->get_row_count() == 0 && ret == E_OK) {
ret = E_NO_MORE_DATA;
}
```
That conflates a page with no matching rows with exhaustion of the data
stream. The trace after decoding the 14-through-18 page shows:
```text
ret=E_NO_MORE_DATA rows=0 has_more_data=true has_next_chunk=false
```
[`TsFileSeriesScanIterator::get_next`](https://github.com/apache/tsfile/blob/eb733133b9eee0b1eb74f7671d809ecd730cedc3/cpp/src/reader/tsfile_series_scan_iterator.cc#L376-L399)
only retries an empty block when `ret` indicates success. With
`E_NO_MORE_DATA`, it advances directly to the next chunk if one exists;
otherwise it returns the end-of-data status. In this reproduction the final
chunk has no successor, so the column is treated as finished and its page
containing timestamps 19 and 20 is never read.
`TimeIn::satisfy(Statistic*)` conservatively returns true, so this page
reaches row-level filtering rather than being skipped by page statistics. The
missing-column projection exposes the affected single-column aligned path. The
underlying defect is the empty-page/end-of-stream status handling, not the set
membership test itself.
## Diagnostic verification
Using a macOS arm64 Debug build with SYSTEM LZ4 and other optional
compressors disabled:
- Unmodified `develop` at `eb733133b9eee0b1eb74f7671d809ecd730cedc3`: 21 of
27 rows returned.
- In an isolated temporary source copy, removing only the
empty-output-to-`E_NO_MORE_DATA` conversion above restores all 27 expected
timestamps. The no-time-filter control still returns all 63 rows. This is a
targeted diagnostic experiment, not a fully validated production fix.
- Restoring the original source and rebuilding reproduces 21 of 27 rows
again.
- The same failure was previously reproduced on #1026's base
`1180b13675e11a0230708af2cc70d037e0e12623` and standalone head
`c01ee105d397b1ed16a4daf335db9d8176036bac`. No fix has been added to that PR. A
released-version range has not been established.
This is separate from the small-capacity hang in #1035: this reproduction
has enough internal block capacity, terminates normally, and returns incomplete
results.
Suggested regression coverage should include fully filtered pages followed
by matching pages within the same chunk, both with and without subsequent
chunks, and check the sparse/missing-column read path as well as all-matching
and no-matching controls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]