ColinLeeo opened a new issue, #1035:
URL: https://github.com/apache/tsfile/issues/1035

   <!--
    * Licensed to the Apache Software Foundation (ASF) under one
    * or more contributor license agreements.  See the NOTICE file
    * distributed with this work for additional information
    * regarding copyright ownership.  The ASF licenses this file
    * to you under the Apache License, Version 2.0 (the
    * "License"); you may not use this file except in compliance
    * with the License.  You may obtain a copy of the License at
    *
    *     http://www.apache.org/licenses/LICENSE-2.0
    *
    * Unless required by applicable law or agreed to in writing,
    * software distributed under the License is distributed on an
    * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    * KIND, either express or implied.  See the License for the
    * specific language governing permissions and limitations
    * under the License.
   
   -->
   
   ## Problem
   
   An aligned C++ table query can spin indefinitely inside the first 
`ResultSet::next()` when the internal TsBlock cannot hold the decoder's fixed 
batch of 129 rows. The query itself returns `E_OK`; reading its first result 
never returns.
   
   A reproducer using `g_config_value_.tsblock_max_memory_ = 128` hangs with no 
time or field predicates. The same file/query completes and returns all 63 rows 
with the default 2 MiB setting (`2097152`). This is the **internal TsBlock 
memory capacity**, not the result API's requested output batch size.
   
   ## Reproduction conditions
   
   - Aligned table with TAG `id` and INT64 fields `v`, `w`, and `ghost`; PLAIN 
encoding and UNCOMPRESSED compression.
   - `ghost` is declared in the schema but is never written. `v` is absent for 
one device, and the written value columns also contain nulls.
   - Three devices, each with 21 timestamps written across three flushes; the 
page writer limit is 5 points.
   - Query `{"v", "ghost", "w", "v"}` with no predicates, offset 0, no limit, 
and serial reading. Missing requested measurements force the per-column aligned 
read path.
   - Set the internal TsBlock memory budget to 128 bytes before creating the 
reader, then call `ResultSet::next()`.
   
   Expected: return all 63 rows using appropriately sized internal batches. If 
a capacity is unsupported, return an explicit error rather than retrying 
without progress.
   
   Actual: the first `next()` does not return; a bounded reproduction was 
terminated after 5 seconds.
   
   <details>
   <summary>Standalone C++ reproducer</summary>
   
   Save this as `small_tsblock_repro.cc` and compile it against the built C++ 
library, for example from the repository root:
   
   ```sh
   c++ -std=c++11 -I cpp/src small_tsblock_repro.cc \
     -L /path/to/build/lib -ltsfile -Wl,-rpath,/path/to/build/lib \
     -o small_tsblock_repro
   ```
   
   ```cpp
   #include <cstdio>
   #include <cstdlib>
   #include <iostream>
   #include <string>
   #include <vector>
   #include "common/config/config.h"
   #include "common/tablet.h"
   #include "file/local_random_access_read_file.h"
   #include "file/write_file.h"
   #include "reader/table_query_executor.h"
   #include "writer/tsfile_table_writer.h"
   using namespace common;
   using namespace storage;
   void check(int ret) { if (ret != E_OK) { std::cerr << "error=" << ret << 
'\n'; std::exit(2); } }
   int main(int argc, char** argv) {
     libtsfile_init();
     const auto saved = g_config_value_;
     g_config_value_.page_writer_max_point_num_ = 5;
     g_config_value_.tsblock_max_memory_ = argc > 2 ? std::atoi(argv[2]) : 128;
     g_config_value_.parallel_read_enabled_ = false;
     const std::string path = argv[1];
     {
       WriteFile file;
       check(file.create(path, O_WRONLY | O_CREAT | O_TRUNC, 0666));
       TableSchema schema("t", {
         {"id", STRING, UNCOMPRESSED, PLAIN, ColumnCategory::TAG},
         {"v", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
         {"w", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
         {"ghost", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD}});
       TsFileTableWriter writer(&file, &schema);
       for (int chunk = 0; chunk < 3; ++chunk) for (int d = 0; d < 3; ++d) {
         std::vector<std::string> names{"id", "w"};
         std::vector<TSDataType> types{STRING, INT64};
         std::vector<ColumnCategory> categories{ColumnCategory::TAG, 
ColumnCategory::FIELD};
         if (d != 1) { names.push_back("v"); types.push_back(INT64); 
categories.push_back(ColumnCategory::FIELD); }
         Tablet tablet("t", names, types, categories, 7);
         for (int r = 0; r < 7; ++r) {
           int t = chunk * 7 + r;
           check(tablet.add_timestamp(r, t));
           const std::string id = "d" + std::to_string(d);
           check(tablet.add_value(r, "id", id.c_str()));
           if (t % 3 != 0) check(tablet.add_value(r, "w", int64_t(d * 1000 + 
t)));
           if (d != 1 && t % 4 != 0) check(tablet.add_value(r, "v", int64_t(d * 
100 + t)));
         }
         check(writer.write_table(tablet));
         check(writer.flush());
       }
       check(writer.close());
     }
     std::cerr << "before query memory=" << g_config_value_.tsblock_max_memory_ 
<< '\n';
     LocalRandomAccessReadFile file;
     check(file.open(path));
     TableQueryExecutor executor(&file, -1);
     ResultSet* result = nullptr;
     check(executor.query("t", {"v", "ghost", "w", "v"}, nullptr, nullptr, 
nullptr, 0, -1, result));
     std::cerr << "after query before first next\n";
     bool next = false;
     int ret = E_OK, count = 0;
     while ((ret = result->next(next)) == E_OK && next) {
       ++count;
       if (count > 200) return 3;
     }
     std::cerr << "after scan ret=" << ret << " count=" << count << " 
expected=63\n";
     executor.destroy_query_data_set(result);
     std::remove(path.c_str());
     g_config_value_ = saved;
     libtsfile_destroy();
     return ret == E_OK && count == 63 ? 0 : 4;
   }
   ```
   
   Run the two cases with an external 5-second timeout (e.g. `timeout` on 
Linux):
   
   ```sh
   timeout 5s ./small_tsblock_repro /tmp/small-tsblock-128.tsfile 128
   ./small_tsblock_repro /tmp/small-tsblock-default.tsfile 2097152
   ```
   
   The small-capacity case stops after `after query before first next`. The 
default-capacity case prints `after scan ret=0 count=63 expected=63` and exits 
with code 0. Terminate the small-capacity process externally; an iteration 
counter around `next()` cannot interrupt this hang.
   
   </details>
   
   ## Cause
   
   
[`AlignedChunkReader::decode_tv_batch`](https://github.com/apache/tsfile/blob/eb733133b9eee0b1eb74f7671d809ecd730cedc3/cpp/src/reader/aligned_chunk_reader.cc#L858-L872)
 checks `row_appender.remaining() < BATCH` with `BATCH = 129` and returns 
`E_OVERFLOW` **before consuming any input**. For the time-plus-INT64 internal 
block in this reproducer, 128 bytes only allow 8 rows; the condition remains 
true even on an empty block and even though a page has fewer than 129 points.
   
   `decode_time_value_buf_into_tsblock` preserves the unfinished page and 
converts that overflow to `E_OK`. The series iterator receives an empty 
successful block, sees that the chunk still has data, and retries. Since 
neither the input cursor nor available block capacity changes, it repeats 
without progress.
   
   The multi-column `multi_DECODE_TV_BATCH` path already caps its effective 
batch with `min(BATCH, row_appender.remaining())`. The single-column typed path 
still has the unconditional 129-row requirement.
   
   ## Verification and scope
   
   - Reproduced on the base of #1026, 
`1180b13675e11a0230708af2cc70d037e0e12623`: timeout on the first `next()`.
   - Reproduced again on standalone PR head 
`c01ee105d397b1ed16a4daf335db9d8176036bac` with a macOS arm64 Debug library: 
128 bytes times out after 5 seconds; 2097152 bytes exits successfully with all 
63 rows. No predicates are supplied.
   - Source inspection of current `develop`, 
`eb733133b9eee0b1eb74f7671d809ecd730cedc3`, confirms the same fixed-capacity 
guard remains. The runtime runs above used the PR head/base, not a separately 
rebuilt `develop` library.
   - This is a pre-existing reader issue found during review of #1026. Track 
its fix separately; no fix is included in that PR. A released-version range has 
not been established.
   
   Suggested regression coverage should run the sparse/missing-column query 
with internal capacities below, at, and above 129 rows, check forward progress 
with a bounded timeout, and compare complete results with the default-capacity 
query.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to