ColinLeeo opened a new issue, #1038:
URL: https://github.com/apache/tsfile/issues/1038

   <!--
    * Licensed to the Apache Software Foundation (ASF) under one
    * or more contributor license agreements.  See the NOTICE file
    * distributed with this work for additional information
    * regarding copyright ownership.  The ASF licenses this file
    * to you under the Apache License, Version 2.0 (the
    * "License"); you may not use this file except in compliance
    * with the License.  You may obtain a copy of the License at
    *
    *     http://www.apache.org/licenses/LICENSE-2.0
    *
    * Unless required by applicable law or agreed to in writing,
    * software distributed under the License is distributed on an
    * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    * KIND, either express or implied.  See the License for the
    * specific language governing permissions and limitations
    * under the License.
   
   -->
   
   ## Problem
   
   A C++ aligned table query using `TimeIn` can silently omit matching rows 
after decoding a page with no matching timestamps. The page decoder reports 
`E_NO_MORE_DATA` for an empty filtered result even when the same chunk still 
contains unread pages. The series iterator then skips the remainder of that 
chunk or ends the series.
   
   This was independently reproduced on the current default branch, `develop`, 
at `eb733133b9eee0b1eb74f7671d809ecd730cedc3`. It is a pre-existing reader bug 
found while reviewing #1026; please track its fix separately from that PR.
   
   ## Reproduction
   
   1. Write an aligned table with TAG `id` and INT64 fields `v`, `w`, and 
`ghost`, using PLAIN encoding and UNCOMPRESSED compression.
   2. Use three devices, each with timestamps 0 through 20, written in three 
groups of seven rows with an explicit flush after each group. Set 
`page_writer_max_point_num_ = 5`.
   3. Leave `ghost` unwritten, omit `v` entirely for one device, and include 
nulls in `v`/`w`. Requesting the missing measurement makes the query use the 
per-column aligned read path.
   4. Use serial reading and a 4096-byte internal TsBlock budget, which 
accommodates the decoder's 129-row batch for these single-column blocks.
   5. Query `{"v", "ghost", "w", "v"}` with `TimeIn({0, 2, 3, 6, 7, 10, 13, 19, 
20}, false)`, no field predicate, offset 0, and no limit.
   
   Expected: 9 matching timestamps per device, 27 rows total. Null field values 
must not remove rows that match the time predicate.
   
   Actual: 7 timestamps per device, 21 rows total. Each device loses timestamps 
19 and 20. The API returns `E_OK` and reports normal end of results; it does 
not report an error.
   
   ```text
   Expected per device: 0 2 3 6 7 10 13 19 20
   Actual per device:   0 2 3 6 7 10 13
   ```
   
   <details>
   <summary>Standalone C++ reproducer</summary>
   
   Save this as `timein_empty_page_repro.cc`. From the repository root, compile 
it against the built C++ library, for example:
   
   ```sh
   c++ -std=c++11 -I cpp/src timein_empty_page_repro.cc \
     -L /path/to/build/lib -ltsfile -Wl,-rpath,/path/to/build/lib \
     -o timein_empty_page_repro
   ./timein_empty_page_repro /tmp/timein-empty-page-repro.tsfile
   ```
   
   The program prints `after scan ret=0 count=21 expected=27` and exits with 
code 4 on the affected implementation. Correct behavior is 27 rows and exit 
code 0.
   
   ```cpp
   #include <cstdio>
   #include <cstdlib>
   #include <iostream>
   #include <string>
   #include <vector>
   #include "common/config/config.h"
   #include "common/tablet.h"
   #include "file/local_random_access_read_file.h"
   #include "file/write_file.h"
   #include "reader/filter/time_operator.h"
   #include "reader/table_query_executor.h"
   #include "writer/tsfile_table_writer.h"
   using namespace common;
   using namespace storage;
   void check(int ret) { if (ret != E_OK) { std::cerr << "error=" << ret << 
'\n'; std::exit(2); } }
   int main(int argc, char** argv) {
     libtsfile_init();
     const auto saved = g_config_value_;
     g_config_value_.page_writer_max_point_num_ = 5;
     g_config_value_.tsblock_max_memory_ = argc > 2 ? std::atoi(argv[2]) : 4096;
     g_config_value_.parallel_read_enabled_ = false;
     const std::string path = argc > 1 ? argv[1] : 
"/tmp/timein-empty-page-repro.tsfile";
     {
       WriteFile file;
       check(file.create(path, O_WRONLY | O_CREAT | O_TRUNC, 0666));
       TableSchema schema("t", {
         {"id", STRING, UNCOMPRESSED, PLAIN, ColumnCategory::TAG},
         {"v", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
         {"w", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
         {"ghost", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD}});
       TsFileTableWriter writer(&file, &schema);
       for (int chunk = 0; chunk < 3; ++chunk) for (int d = 0; d < 3; ++d) {
         std::vector<std::string> names{"id", "w"};
         std::vector<TSDataType> types{STRING, INT64};
         std::vector<ColumnCategory> categories{ColumnCategory::TAG, 
ColumnCategory::FIELD};
         if (d != 1) { names.push_back("v"); types.push_back(INT64); 
categories.push_back(ColumnCategory::FIELD); }
         Tablet tablet("t", names, types, categories, 7);
         for (int r = 0; r < 7; ++r) {
           int t = chunk * 7 + r;
           check(tablet.add_timestamp(r, t));
           const std::string id = "d" + std::to_string(d);
           check(tablet.add_value(r, "id", id.c_str()));
           if (t % 3 != 0) check(tablet.add_value(r, "w", int64_t(d * 1000 + 
t)));
           if (d != 1 && t % 4 != 0) check(tablet.add_value(r, "v", int64_t(d * 
100 + t)));
         }
         check(writer.write_table(tablet));
         check(writer.flush());
       }
       check(writer.close());
     }
     std::cerr << "before query memory=" << g_config_value_.tsblock_max_memory_ 
<< '\n';
     LocalRandomAccessReadFile file;
     check(file.open(path));
     TableQueryExecutor executor(&file, -1);
     ResultSet* result = nullptr;
     check(executor.query("t", {"v", "ghost", "w", "v"}, new TimeIn({0, 2, 3, 
6, 7, 10, 13, 19, 20}, false), nullptr, nullptr, 0, -1, result));
     std::cerr << "after query before first next\n";
     bool next = false;
     int ret = E_OK, count = 0;
     while ((ret = result->next(next)) == E_OK && next) {
       ++count;
       std::cout << result->get_value<int64_t>(1) << " ";
       if (count > 200) return 3;
     }
     std::cerr << "after scan ret=" << ret << " count=" << count << " 
expected=27\n";
     executor.destroy_query_data_set(result);
     std::remove(path.c_str());
     g_config_value_ = saved;
     libtsfile_destroy();
     return ret == E_OK && count == 27 ? 0 : 4;
   }
   ```
   
   </details>
   
   ## Root cause
   
   The final chunk for each device has two time pages:
   
   | Page | Timestamps | Required behavior |
   | --- | --- | --- |
   | First | 14 through 18 | No timestamps match; continue scanning the same 
chunk |
   | Second | 19 and 20 | Both timestamps match and must be returned |
   
   
[`AlignedChunkReader::decode_tv_buf_into_tsblock_by_datatype`](https://github.com/apache/tsfile/blob/eb733133b9eee0b1eb74f7671d809ecd730cedc3/cpp/src/reader/aligned_chunk_reader.cc#L1057-L1059)
 changes a successful decode into `E_NO_MORE_DATA` whenever the output TsBlock 
is empty:
   
   ```cpp
   if (ret_tsblock->get_row_count() == 0 && ret == E_OK) {
       ret = E_NO_MORE_DATA;
   }
   ```
   
   That conflates a page with no matching rows with exhaustion of the data 
stream. The trace after decoding the 14-through-18 page shows:
   
   ```text
   ret=E_NO_MORE_DATA rows=0 has_more_data=true has_next_chunk=false
   ```
   
   
[`TsFileSeriesScanIterator::get_next`](https://github.com/apache/tsfile/blob/eb733133b9eee0b1eb74f7671d809ecd730cedc3/cpp/src/reader/tsfile_series_scan_iterator.cc#L376-L399)
 only retries an empty block when `ret` indicates success. With 
`E_NO_MORE_DATA`, it advances directly to the next chunk if one exists; 
otherwise it returns the end-of-data status. In this reproduction the final 
chunk has no successor, so the column is treated as finished and its page 
containing timestamps 19 and 20 is never read.
   
   `TimeIn::satisfy(Statistic*)` conservatively returns true, so this page 
reaches row-level filtering rather than being skipped by page statistics. The 
missing-column projection exposes the affected single-column aligned path. The 
underlying defect is the empty-page/end-of-stream status handling, not the set 
membership test itself.
   
   ## Diagnostic verification
   
   Using a macOS arm64 Debug build with SYSTEM LZ4 and other optional 
compressors disabled:
   
   - Unmodified `develop` at `eb733133b9eee0b1eb74f7671d809ecd730cedc3`: 21 of 
27 rows returned.
   - In an isolated temporary source copy, removing only the 
empty-output-to-`E_NO_MORE_DATA` conversion above restores all 27 expected 
timestamps. The no-time-filter control still returns all 63 rows. This is a 
targeted diagnostic experiment, not a fully validated production fix.
   - Restoring the original source and rebuilding reproduces 21 of 27 rows 
again.
   - The same failure was previously reproduced on #1026's base 
`1180b13675e11a0230708af2cc70d037e0e12623` and standalone head 
`c01ee105d397b1ed16a4daf335db9d8176036bac`. No fix has been added to that PR. A 
released-version range has not been established.
   
   This is separate from the small-capacity hang in #1035: this reproduction 
has enough internal block capacity, terminates normally, and returns incomplete 
results.
   
   Suggested regression coverage should include fully filtered pages followed 
by matching pages within the same chunk, both with and without subsequent 
chunks, and check the sparse/missing-column read path as well as all-matching 
and no-matching controls.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to