This is an automated email from the ASF dual-hosted git repository.
ColinLeeo pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/tsfile.git
The following commit(s) were added to refs/heads/develop by this push:
new b4b40c0b4 fix(skills): update tsfile skills to follow current version
(#916)
b4b40c0b4 is described below
commit b4b40c0b43ae7e9434e03fc12df9d24465c3b569
Author: Zeyu Zhang <[email protected]>
AuthorDate: Tue Aug 25 09:56:45 2026 +0800
fix(skills): update tsfile skills to follow current version (#916)
---
cpp/tools/skills/tsfile-cli/SKILL.md | 57 +++++++-----
skills/tsfile-cli/SKILL.md | 23 +++--
skills/tsfile/SKILL.md | 63 +++++++++++--
skills/tsfile/references/advanced-apis.md | 99 +++++++++++++++++++++
skills/tsfile/references/cpp-build-write.md | 83 ++++++++++++++++++
skills/tsfile/references/dataframe-arrow.md | 100 +++++++++++++++++++++
skills/tsfile/references/docs-map.yaml | 107 ++++++++++++++++++++---
skills/tsfile/references/encoding-compression.md | 77 ++++++++++++++++
skills/tsfile/references/java-tools.md | 89 +++++++++++++++++++
skills/tsfile/references/performance.md | 2 +
skills/tsfile/references/properties-filters.md | 80 +++++++++++++++++
11 files changed, 735 insertions(+), 45 deletions(-)
diff --git a/cpp/tools/skills/tsfile-cli/SKILL.md
b/cpp/tools/skills/tsfile-cli/SKILL.md
index 3aa94fd75..8261cc70a 100644
--- a/cpp/tools/skills/tsfile-cli/SKILL.md
+++ b/cpp/tools/skills/tsfile-cli/SKILL.md
@@ -1,32 +1,43 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
---
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
name: tsfile-cli
-description: Use when you need to inspect, preview, export, OR import an
Apache TsFile (.tsfile) from the command line — list devices/tables, dump
schema, read file/series metadata, count rows, sample/preview rows, or write
CSV/TSV into a new .tsfile — via the project's C++ `tsfile-cli` in cpp/tools.
+description: >-
+ Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect,
+ preview, export, or sample an Apache TsFile; report metadata or per-series
+ counts; or use its explicit single-table CSV/TSV write command.
---
# tsfile-cli
-Single pipe-friendly C++ binary to inspect **and** import `.tsfile` (TsFile's
analogue of
-`parquet-cli`/`pqrs`). Source `cpp/tools/`. Read data → stdout, diagnostics →
stderr;
-`write` imports CSV/TSV → a new file.
+Single pipe-friendly C++ binary to inspect `.tsfile` files and create a new
+table-model TsFile from CSV/TSV (TsFile's analogue of `parquet-cli`/`pqrs`).
+Source `cpp/tools/`. Read data → stdout, diagnostics → stderr.
+
+## Scope
+
+Use this skill only for the C++ `tsfile-cli` binary. Route Java
+`csv2tsfile`/`parquet2tsfile`/`arrow2tsfile` and table point-count metadata
+checks or backfill to the top-level `tsfile` skill when it is installed.
+
+The names overlap but the semantics do not: `tsfile-cli count` is a read-only
+per-series report, not the Java table point-count property tool. `tsfile-cli
+write` is the C++ binary's narrow one-file/stream, one-table CSV/TSV import; it
+does not replace the Java batch and format-aware import tools.
## Binary
diff --git a/skills/tsfile-cli/SKILL.md b/skills/tsfile-cli/SKILL.md
index 37c59a207..4cbb9cc57 100644
--- a/skills/tsfile-cli/SKILL.md
+++ b/skills/tsfile-cli/SKILL.md
@@ -16,20 +16,29 @@
# specific language governing permissions and limitations
# under the License.
name: tsfile-cli
-description: Use when you need to inspect, preview, export, OR import an
Apache TsFile (.tsfile) from the command line — list devices/tables, dump
schema, read file/series metadata, count rows, sample/preview rows, or write
CSV/TSV into a new .tsfile — via the project's C++ `tsfile-cli` in cpp/tools.
+description: >-
+ Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect,
+ preview, export, or sample an Apache TsFile; report metadata or per-series
+ counts; or use its explicit single-table CSV/TSV write command.
---
# tsfile-cli
-Single pipe-friendly C++ binary to inspect **and** import `.tsfile` (TsFile's
analogue of
-`parquet-cli`/`pqrs`). Source `cpp/tools/`. Read data → stdout, diagnostics →
stderr;
-`write` imports CSV/TSV → a new file.
+Single pipe-friendly C++ binary to inspect `.tsfile` files and create a new
+table-model TsFile from CSV/TSV (TsFile's analogue of `parquet-cli`/`pqrs`).
+Source `cpp/tools/`. Read data → stdout, diagnostics → stderr.
## Scope
-Use this skill for command-line builds and operations. For Java, Python, C++,
-or C SDK integration, schema design, and programmatic tree-model writes, load
-the sibling `tsfile` skill at `../tsfile/SKILL.md`.
+Use this skill only for the C++ `tsfile-cli` binary. For Java, Python, C++, or
C
+SDK integration, schema design, Java CSV/Parquet/Arrow import, Java table
+point-count metadata checks or backfill, and programmatic tree-model writes,
+load the sibling `tsfile` skill at `../tsfile/SKILL.md`.
+
+The names overlap but the semantics do not: `tsfile-cli count` is a read-only
+per-series report, not the Java table point-count property tool. `tsfile-cli
+write` is the C++ binary's narrow one-file/stream, one-table CSV/TSV import; it
+does not replace the Java batch and format-aware import tools.
## Binary
diff --git a/skills/tsfile/SKILL.md b/skills/tsfile/SKILL.md
index 360b8df26..85e5f06f6 100644
--- a/skills/tsfile/SKILL.md
+++ b/skills/tsfile/SKILL.md
@@ -16,17 +16,25 @@
# specific language governing permissions and limitations
# under the License.
name: tsfile
-description: Work with Apache TsFile programmatic SDKs and file-format
concepts in Java, Python, C++, or C. Use for reading, writing, querying, schema
or data-model design, encoding/compression decisions, performance analysis, API
compatibility, and cross-language TsFile integration. Route shell inspection,
preview, export, sampling, and CSV/TSV conversion to the sibling tsfile-cli
skill.
+description: >-
+ Work with the current Apache TsFile SDKs, tools, and file format. Use for
+ Java, Python, C++, or C reads and writes; Python DataFrame and Arrow batches;
+ Java CSV/Parquet/Arrow import; file properties; TAG filters; query-by-row and
+ metadata APIs; C++ incomplete-writer recovery and append; static/shared C++
+ builds; parallel and TsBlock writes; encoding/compression factories; schema
+ design; performance; and cross-language integration. Route operations
+ specific to the C++ tsfile-cli binary to the sibling tsfile-cli skill.
---
# TsFile
## Scope
-Use this skill for SDK code, Tree/Table model decisions, schema design,
-compatibility, and cross-language integration. Use the sibling
-`../tsfile-cli/SKILL.md` for shell-oriented inspection, preview, export,
-sampling, and CSV/TSV-to-TsFile conversion.
+Use this skill for SDK code, Java import and point-count tools, Tree/Table
model
+decisions, schema design, compatibility, and cross-language integration. Use
+the sibling `../tsfile-cli/SKILL.md` for the C++ `tsfile-cli` binary: file
+inspection, preview, export, sampling, and its explicit single-table CSV/TSV
+`write` operation.
## Operating Rules
@@ -56,6 +64,19 @@ Read only the files required by the current task:
- Model selection, schema, data types, and generic read/write workflow:
`references/core-concepts.md`
- Java SDK code and API guardrails: `references/java.md`
+- Java CSV/Parquet/Arrow import and table point-count tools:
+ `references/java-tools.md`
+- Python DataFrame datasets, pandas conversion, and Arrow batch I/O:
+ `references/dataframe-arrow.md`
+- File-level properties and Table-model TAG filters across bindings:
+ `references/properties-filters.md`
+- C++ static/shared linking and parallel write configuration:
+ `references/cpp-build-write.md`
+- Current Java and C++ encoding/compression factory matrices:
+ `references/encoding-compression.md`
+- Query-by-row, timeseries metadata, C++ incomplete-file recovery and append,
+ Java TsBlock writes, custom/buffered input, and Tablet object/size APIs:
+ `references/advanced-apis.md`
- Python SDK code and binding-specific behavior: `references/python.md`
- C++ SDK code and resource management: `references/cpp.md`
- C wrapper entry points and lifecycle: `references/c.md`
@@ -80,6 +101,38 @@ For compatibility questions, report the local source
version and the requested
release separately. Never silently combine signatures from different versions
or language bindings.
+## Current Baseline Routing
+
+- Use `references/dataframe-arrow.md` for `TsFileDataFrame`, pandas conversion,
+ PyArrow `Table`/`RecordBatch`, or Arrow C Data batch paths.
+- Use `references/properties-filters.md` for binary file properties or
+ Table-model TAG predicate construction and composition.
+- Use `references/cpp-build-write.md` for C++ library linkage or write
+ concurrency. Configure the current source APIs; do not emit legacy aliases.
+- Use `references/encoding-compression.md` before naming a factory or claiming
+ that an encoding/type or compressor combination is constructible.
+- Use `references/advanced-apis.md` for C++ crash recovery and append through
+ `RestorableTsFileIOWriter`, plus the remaining current advanced read and
write
+ surfaces listed in Offline Reference Routing.
+
+Treat these as baseline capabilities. Do not add alternate calls, degraded
+results, or compatibility branches for earlier TsFile source lines unless the
+user explicitly targets a different checkout or release.
+
+## Java Tools
+
+- Route `csv2tsfile`, `parquet2tsfile`, `arrow2tsfile`, schema-driven import,
+ import auto-inference, directory import, and failed-file handling to the Java
+ tools in `java/tools/`. Read `references/java-tools.md` before constructing
or
+ running a command.
+- Route requests to inspect or backfill the persisted table-level point-count
+ property to the Java `tsfile-table-point-count` tool. This operation can
+ replace the input file when metadata is missing, so follow the mutation
+ checks in `references/java-tools.md`.
+- Do not substitute C++ `tsfile-cli count` for the Java point-count tool. The
+ former reports per-series counts without changing the file; the latter
+ validates and may backfill table-level metadata.
+
## Bundled Resources
- Run `scripts/resolve-version.sh [--root <checkout>]` to obtain Maven, C++,
diff --git a/skills/tsfile/references/advanced-apis.md
b/skills/tsfile/references/advanced-apis.md
new file mode 100644
index 000000000..c8672657f
--- /dev/null
+++ b/skills/tsfile/references/advanced-apis.md
@@ -0,0 +1,99 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Current Advanced APIs
+
+Load only the section matching the task and verify exact constructors against
+the listed current source.
+
+## Row-window Queries
+
+- C++ exposes `TsFileReader::queryByRow` for Tree paths and for Table columns,
+ with `offset`, `limit`, optional Table TAG filter, and optional batch size.
+ `TsFileTreeReader::queryByRow` accepts device and measurement lists.
+- Python exposes `query_tree_by_row` and `query_table_by_row`; Table queries
+ accept `tag_filter` and `batch_size`.
+- Treat `limit < 0` as unlimited and reject negative offsets. Close or destroy
+ every result set through the binding that created it.
+
+## Timeseries Metadata
+
+- C++ `TsFileReader::get_timeseries_metadata()` returns metadata for all
+ devices; its device-list overload restricts the request.
+- Python `get_timeseries_metadata(device_ids=None)` returns a mapping keyed by
+ the full device-segment tuple. `None` selects all devices, while `[]` returns
+ an empty map; null TAG segments remain `None`.
+- C callers must use the all-devices or selected-devices declarations and the
+ matching free routine in `tsfile_cwrapper.h`.
+
+## Java TsBlock Write
+
+Use `TableTsBlock2TsFileWriter` when the input is already a Table-model
+`TsBlock`. Its constructor maps time, TAG, and FIELD column indexes and can
+generate a monotonically increasing time column per device. Feed complete
+blocks through `write(TsBlock)`, then close the writer. Build the mapping from
+the current `TableSchema`; do not infer constructor positions from memory.
+
+## Java Tablet Object and Size APIs
+
+- Use `Tablet.addObjectPathValue` only for an `OBJECT` column; overloads accept
+ a measurement name or column index and a `String` or `byte[]` path.
+- Use `Tablet.serializedSize()` to allocate or validate the serialized Tablet
+ payload. Do not replace it with a RAM-size estimate.
+
+## Custom and Buffered Java Input
+
+`TsFileSequenceReader` accepts a `TsFileInput`, enabling custom storage/input
+implementations. For local buffered reads, construct `BufferedTsFileInput(Path,
+bufferSize)` and pass it to the reader. Buffer size must be positive, and the
+reader owns and closes the supplied input if initialization fails or when the
+reader is closed.
+
+## C++ Recovery and Append
+
+Use `storage::RestorableTsFileIOWriter` to inspect and optionally truncate an
+incomplete file before continuing a write. After `open(path, true)`, require
+`has_crashed()` and `can_write()` before constructing `TsFileTreeWriter` or
+`TsFileTableWriter` with the restorable writer. The recovered schema is used;
+do not register a replacement schema. A complete file reports
+`can_write() == false`. Keep the restorable writer alive longer than the facade
+writer and preserve increasing timestamps beyond the recovered last values.
+
+## Java BitMap Ranges
+
+Use `markRange(start, length)` and `unmarkRange(start, length)` for mutations,
+and `isRangeAnyMarked`, `isRangeAllMarked`, or `isRangeNoneMarked` for range
+tests. Ranges are `(start, length)`, not `(start, end)`; empty ranges return
+`false`, `true`, and `true`, respectively. Out-of-bounds ranges throw.
+
+## Source Anchors
+
+- Row queries: `cpp/src/reader/tsfile_reader.h`,
+ `cpp/src/reader/tsfile_tree_reader.h`, and `python/tsfile/tsfile_reader.pyx`
+- Metadata: those readers, `cpp/src/cwrapper/tsfile_cwrapper.h`,
+ `python/tsfile/schema.py`, and `python/tests/test_reader_metadata.py`
+- TsBlock:
`java/tsfile/src/main/java/org/apache/tsfile/write/v4/TableTsBlock2TsFileWriter.java`
+- Tablet:
`java/tsfile/src/main/java/org/apache/tsfile/write/record/Tablet.java`
+- Input: `TsFileSequenceReader.java`, `TsFileInput.java`, and
+ `BufferedTsFileInput.java` under
+ `java/tsfile/src/main/java/org/apache/tsfile/read/reader/`
+- Recovery: `cpp/src/file/restorable_tsfile_io_writer.h` and the current C++
+ Tree/Table writer headers and recovery tests
+- BitMap: `java/common/src/main/java/org/apache/tsfile/utils/BitMap.java` and
+ `java/tsfile/src/test/java/org/apache/tsfile/utils/BitMapTest.java`
diff --git a/skills/tsfile/references/cpp-build-write.md
b/skills/tsfile/references/cpp-build-write.md
new file mode 100644
index 000000000..87d8d7307
--- /dev/null
+++ b/skills/tsfile/references/cpp-build-write.md
@@ -0,0 +1,83 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# C++ Build and Parallel Write
+
+## Shared or Static libtsfile
+
+`TSFILE_BUILD_SHARED` selects the library kind and defaults to `ON`.
+
+```sh
+# Maven profile: shared default or static override
+./mvnw clean verify -P with-cpp
+./mvnw clean verify -P with-cpp -Dtsfile.build.shared=OFF
+
+# Direct CMake static build
+cmake -S cpp -B cpp/build/static \
+ -DTSFILE_BUILD_SHARED=OFF -DBUILD_TEST=OFF
+cmake --build cpp/build/static --target tsfile
+
+# Repository wrapper
+cd cpp && bash build.sh --build-static
+```
+
+Use a separate build directory when switching library kind. A direct MSVC
+consumer of the static archive must define `TSFILE_STATIC`; consumers of the
+exported CMake target inherit the required definition. Verify the artifact in
+`cpp/target/build/lib` for Maven or the selected CMake build tree.
+
+`ENABLE_SIMD` also defaults to `ON` in the current CMake and Maven builds. Set
+it explicitly only for portability diagnosis or controlled benchmarking; it
+is independent of static/shared selection and thread support.
+
+## Parallel Write
+
+Compile thread support with `ENABLE_THREADS=ON` (the default). `OFF` strips the
+threaded paths. Configure the current global worker pool through
+`common/global.h`:
+
+```cpp
+#include "common/global.h"
+
+common::set_parallel_write_enabled(true);
+int status = common::set_thread_count(4); // valid range: 1..64
+```
+
+The current baseline uses `common::set_parallel_write_enabled` and
+`common::set_thread_count`; use those exact names and namespace. The pool is
+shared by parallel read and write paths. Configure it before creating active
+readers/writers. Changing the count after initialization rebuilds the pool and
+must never race with an operation using it.
+
+Current source initializes parallel writing as enabled and the pool size as 6.
+Do not substitute a hardware-concurrency default from older prose. Parallel
+work is used only when the compiled path and runtime conditions qualify; keep
+the tablet/batch write contract correct when it falls back to serial work.
+
+Always check write return codes. If a multi-column batch fails partway, stop
+using that writer rather than attempting to flush a potentially misaligned
+partial batch.
+
+## Source Anchors
+
+- Link selection: `cpp/CMakeLists.txt`, `cpp/src/CMakeLists.txt`,
+ `cpp/build.sh`, `cpp/pom.xml`, and `cpp/README.md`
+- Runtime API and defaults: `cpp/src/common/global.h`,
+ `cpp/src/common/global.cc`, and `cpp/src/common/config/config.h`
+- Write behavior: `cpp/src/writer/tsfile_writer.cc` and current writer tests
diff --git a/skills/tsfile/references/dataframe-arrow.md
b/skills/tsfile/references/dataframe-arrow.md
new file mode 100644
index 000000000..843c00def
--- /dev/null
+++ b/skills/tsfile/references/dataframe-arrow.md
@@ -0,0 +1,100 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Python DataFrame and Arrow
+
+Use the current Python binding and its tests as the API authority.
+
+## Select the Surface
+
+- Use `to_dataframe` for an eager pandas result, or batched pandas iterator,
+ from one Tree- or Table-model file.
+- Use `dataframe_to_tsfile` for an inferred Table-model schema and a complete
+ pandas-to-TsFile conversion.
+- Use `TsFileDataFrame` for a lazy unified numeric view across one or more
+ TsFile shards, metadata selection, or timestamp-aligned series access.
+- Use Arrow batches for columnar interoperability without Python row loops.
+- Use `references/java-tools.md` instead for Arrow file import through the Java
+ `arrow2tsfile` command.
+
+## pandas Conversion
+
+```python
+from tsfile import dataframe_to_tsfile, to_dataframe
+
+frame = to_dataframe(
+ "input.tsfile", table_name="sensors", column_names=["device", "temp"],
+ start_time=0, end_time=10_000, max_row_num=100_000,
+)
+dataframe_to_tsfile(
+ frame, "output.tsfile", table_name="sensors",
+ time_column="time", tag_column=["device"],
+)
+```
+
+Set `as_iterator=True` on `to_dataframe` to consume pandas batches. The writer
+lowercases column names, requires an integer time column when one is supplied,
+and otherwise uses a `time` column or the DataFrame index. Validate inferred
+object-column types before writing production data.
+
+## Lazy Multi-file Dataset
+
+```python
+from tsfile import TsFileDataFrame
+
+with TsFileDataFrame(["part-1.tsfile", "part-2.tsfile"], show_progress=False)
as ds:
+ metadata = ds.list_timeseries_metadata()
+ selected = ds[ds["count"] > 0]
+ aligned = selected.loc[0:10_000, [0, 1]]
+```
+
+A single dataset must not mix Tree- and Table-model files. The root
+`TsFileDataFrame` owns readers; use its context manager and do not expect a
+subset view to close shared readers. Boolean dataset selection is positional:
+build the mask from `ds[...]`, as above; use `list_timeseries_metadata()` for
+named inspection rather than as the mask source.
+
+## Arrow Batch I/O
+
+```python
+with reader.query_table(
+ "sensors", ["device", "temp"], batch_size=8192
+) as result:
+ while True:
+ table = result.read_arrow_batch()
+ if table is None:
+ break
+ consume(table) # pyarrow.Table
+
+with TsFileTableWriter("output.tsfile", schema) as writer:
+ writer.write_arrow_batch(record_batch_or_table)
+```
+
+Use `batch_size > 0` for Arrow reads. Arrow writes accept a
+`pyarrow.RecordBatch` or `pyarrow.Table`; include the registered time column
and
+match all remaining columns to the Table schema. Use the wrapper writer when
+possible so it resolves the time-column index from the schema.
+
+## Source Anchors
+
+- `python/tsfile/utils.py`, `python/tsfile/dataset/`, and the current Python
+ reader/writer bindings
+- `python/tsfile/tsfile_table_writer.py`
+- `python/tests/test_dataframe.py`, `python/tests/test_tsfile_dataset.py`,
+ `python/tests/test_batch_arrow.py`, and `python/tests/test_write_arrow.py`
diff --git a/skills/tsfile/references/docs-map.yaml
b/skills/tsfile/references/docs-map.yaml
index 4fb4f342f..ed6e21305 100644
--- a/skills/tsfile/references/docs-map.yaml
+++ b/skills/tsfile/references/docs-map.yaml
@@ -113,8 +113,73 @@ dataframe:
guide_zh:
https://tsfile.apache.org/zh/UserGuide/latest/DataFrame/TsFileDataFrame.html
version_scope: v2.x_latest_alias
local_authorities:
- - python/tsfile/
- - python/tests/
+ - python/tsfile/utils.py
+ - python/tsfile/dataset/
+ - python/tsfile/tsfile_reader.pyx
+ - python/tsfile/tsfile_writer.pyx
+ - python/tsfile/tsfile_table_writer.py
+ - python/tests/test_dataframe.py
+ - python/tests/test_tsfile_dataset.py
+ - python/tests/test_batch_arrow.py
+ - python/tests/test_write_arrow.py
+ - skills/tsfile/references/dataframe-arrow.md
+
+current_capabilities:
+ file_properties_and_tag_filters:
+ local_authorities:
+ -
java/tsfile/src/main/java/org/apache/tsfile/read/filter/factory/TagFilterBuilder.java
+ - java/tsfile/src/main/java/org/apache/tsfile/read/v4/ITsFileReader.java
+ - java/tsfile/src/main/java/org/apache/tsfile/write/v4/ITsFileWriter.java
+ - cpp/src/reader/filter/tag_filter.h
+ - cpp/src/reader/tsfile_reader.h
+ - cpp/src/writer/tsfile_writer.h
+ - cpp/src/writer/tsfile_table_writer.h
+ - cpp/src/cwrapper/tsfile_cwrapper.h
+ - python/tsfile/tag_filter.py
+ - python/tsfile/tsfile_reader.pyx
+ - python/tsfile/tsfile_writer.pyx
+ - python/tsfile/tsfile_table_writer.py
+ - skills/tsfile/references/properties-filters.md
+ cpp_build_and_parallel_write:
+ local_authorities:
+ - cpp/CMakeLists.txt
+ - cpp/src/CMakeLists.txt
+ - cpp/build.sh
+ - cpp/src/common/global.h
+ - cpp/src/common/global.cc
+ - cpp/src/common/config/config.h
+ - cpp/src/writer/tsfile_writer.cc
+ - skills/tsfile/references/cpp-build-write.md
+ encoding_and_compression_factories:
+ local_authorities:
+ -
java/tsfile/src/main/java/org/apache/tsfile/file/metadata/enums/TSEncoding.java
+ -
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/TSEncodingBuilder.java
+ -
java/tsfile/src/main/java/org/apache/tsfile/encoding/decoder/Decoder.java
+ -
java/tsfile/src/main/java/org/apache/tsfile/file/metadata/enums/CompressionType.java
+ - java/tsfile/src/main/java/org/apache/tsfile/compress/ICompressor.java
+ - java/tsfile/src/main/java/org/apache/tsfile/compress/IUnCompressor.java
+ - cpp/src/encoding/encoder_factory.h
+ - cpp/src/encoding/decoder_factory.h
+ - cpp/src/compress/compressor_factory.h
+ - cpp/src/common/db_common.h
+ - skills/tsfile/references/encoding-compression.md
+ advanced_apis:
+ local_authorities:
+ - cpp/src/reader/tsfile_reader.h
+ - cpp/src/reader/tsfile_tree_reader.h
+ - cpp/src/cwrapper/tsfile_cwrapper.h
+ - python/tsfile/tsfile_reader.pyx
+ - python/tsfile/schema.py
+ -
java/tsfile/src/main/java/org/apache/tsfile/write/v4/TableTsBlock2TsFileWriter.java
+ - java/tsfile/src/main/java/org/apache/tsfile/write/record/Tablet.java
+ -
java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java
+ -
java/tsfile/src/main/java/org/apache/tsfile/read/reader/TsFileInput.java
+ -
java/tsfile/src/main/java/org/apache/tsfile/read/reader/BufferedTsFileInput.java
+ - cpp/src/file/restorable_tsfile_io_writer.h
+ - cpp/src/writer/tsfile_tree_writer.h
+ - cpp/src/writer/tsfile_table_writer.h
+ - java/common/src/main/java/org/apache/tsfile/utils/BitMap.java
+ - skills/tsfile/references/advanced-apis.md
ecosystem:
version_policy: verify_connector_dependencies_before_use
@@ -123,11 +188,33 @@ ecosystem:
spark:
https://tsfile.apache.org/UserGuide/latest/Ecosystem-Integration/Spark-TsFile.html
tools:
- owner_skill:
- path: ../tsfile-cli/SKILL.md
- base: skill_root
- cli: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-CLI.html
- viewer: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-Viewer.html
- local_authorities:
- - cpp/tools/README.md
- - cpp/tools/
+ java:
+ owner_skill:
+ path: SKILL.md
+ base: skill_root
+ import:
+ formats:
+ - csv
+ - parquet
+ - arrow
+ local_authorities:
+ - java/tools/README.md
+ - java/tools/src/main/java/org/apache/tsfile/tools/
+ - java/tools/src/assembly/resources/tools/
+ - skills/tsfile/references/java-tools.md
+ table_point_count:
+ local_authorities:
+ - java/tools/README.md
+ -
java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileTablePointCountTool.java
+ - java/tools/src/assembly/resources/tools/tsfile-table-point-count.sh
+ - java/tools/src/assembly/resources/tools/tsfile-table-point-count.bat
+ - skills/tsfile/references/java-tools.md
+ cpp_cli:
+ owner_skill:
+ path: ../tsfile-cli/SKILL.md
+ base: skill_root
+ cli: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-CLI.html
+ viewer: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-Viewer.html
+ local_authorities:
+ - cpp/tools/README.md
+ - cpp/tools/
diff --git a/skills/tsfile/references/encoding-compression.md
b/skills/tsfile/references/encoding-compression.md
new file mode 100644
index 000000000..b745687ca
--- /dev/null
+++ b/skills/tsfile/references/encoding-compression.md
@@ -0,0 +1,77 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Current Encoding and Compression Factories
+
+Prefer schema constructors for normal file writes. Use these low-level
+factories only for codec, page, or format work. An enum member does not prove
+that a factory can construct the requested codec.
+
+## Java
+
+Validate with `TSEncoding.isSupported(dataType, encoding)`, create encoders
with
+`TSEncodingBuilder.getEncodingBuilder(encoding).getEncoder(dataType)`, and
+create decoders with `Decoder.getDecoderByType(encoding, dataType)`.
+
+| data type | current supported encodings |
+|---|---|
+| `BOOLEAN` | `PLAIN`, `RLE` |
+| `INT32`, `INT64`, `DATE`, `TIMESTAMP` | `PLAIN`, `RLE`, `TS_2DIFF`,
`GORILLA`, `ZIGZAG`, `CHIMP`, `SPRINTZ`, `RLBE` |
+| `FLOAT` | `PLAIN`, `RLE`, `TS_2DIFF`, `GORILLA_V1`, `GORILLA`, `CHIMP`,
`SPRINTZ`, `RLBE` |
+| `DOUBLE` | the FLOAT set plus `CAMEL` |
+| `TEXT`, `STRING` | `PLAIN`, `DICTIONARY` |
+| `BLOB`, `OBJECT` | `PLAIN` |
+
+Create compressors with `ICompressor.getCompressor(CompressionType)` and
+decompressors with `IUnCompressor.getUnCompressor(CompressionType)`. The
+current Java factory set is `UNCOMPRESSED`, `SNAPPY`, `GZIP`, `LZ4`, `ZSTD`,
+and `LZMA2`.
+
+## C++
+
+Use `EncoderFactory::alloc_time_encoder`,
+`EncoderFactory::alloc_value_encoder`, `DecoderFactory::alloc_time_decoder`,
+`DecoderFactory::alloc_value_decoder`, and each factory's `free` function.
+Current time factories accept `PLAIN` or `TS_2DIFF`.
+
+| value encoding | current accepted data types |
+|---|---|
+| `PLAIN` | all types handled by `PlainEncoder`/`PlainDecoder` |
+| `DICTIONARY` | `STRING`, `TEXT` |
+| `RLE` | `INT32`, `DATE`, `INT64`, `TIMESTAMP` |
+| `TS_2DIFF`, `GORILLA` | `INT32`, `DATE`, `INT64`, `TIMESTAMP`, `FLOAT`,
`DOUBLE` |
+| `ZIGZAG` | `INT32`, `INT64` |
+| `SPRINTZ` | `INT32`, `INT64`, `FLOAT`, `DOUBLE` |
+
+`CompressorFactory::alloc_compressor` always supports `UNCOMPRESSED`; `SNAPPY`,
+`GZIP`, `LZO`, and `LZ4` require their corresponding compile definitions.
+`SDT`, `PAA`, and `PLA` are enum values but the current compressor factory
+returns no implementation for them. Check for `nullptr` and release successful
+allocations with `CompressorFactory::free`.
+
+Python exposes schema encoding/compression enums through the native binding but
+does not expose these Java/C++ low-level factories as an equivalent Python API.
+
+## Source Anchors
+
+- Java: `TSEncoding.java`, `TSEncodingBuilder.java`, `Decoder.java`,
+ `CompressionType.java`, `ICompressor.java`, and `IUnCompressor.java`
+- C++: `cpp/src/encoding/encoder_factory.h`,
+ `cpp/src/encoding/decoder_factory.h`,
+ `cpp/src/compress/compressor_factory.h`, and `cpp/src/common/db_common.h`
diff --git a/skills/tsfile/references/java-tools.md
b/skills/tsfile/references/java-tools.md
new file mode 100644
index 000000000..e344226c6
--- /dev/null
+++ b/skills/tsfile/references/java-tools.md
@@ -0,0 +1,89 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Java Tools
+
+Use this reference for the packaged Java command-line tools. Use
+`java/tools/README.md`, its tests, and the current source as the exact command
+authority. Do not infer these commands from the C++ `tsfile-cli` interface.
+
+## Locate or Build
+
+- In a packaged distribution, run scripts from `$TSFILE_HOME/tools/`; they use
+ the jars under `$TSFILE_HOME/lib/`.
+- In a source checkout, build the Java distribution with the checkout's
+ documented `./mvnw clean package -P with-java -DskipTests` flow before using
+ the packaged scripts. Treat `java/tools/src/assembly/resources/tools/` as
+ packaging source, not as an independently installed runtime.
+- Require Java and `JAVA_HOME` as documented by the current checkout.
+
+## Import CSV, Parquet, or Arrow
+
+Prefer the Java importer for format-aware or batch conversion: a source file or
+directory, CSV/Parquet/Arrow input, explicit schema mapping, schema inference,
+failed-file collection, threading, or CSV chunking.
+
+```sh
+$TSFILE_HOME/tools/csv2tsfile.sh --source input.csv --target output
+$TSFILE_HOME/tools/parquet2tsfile.sh --source parquet-dir --target output
+$TSFILE_HOME/tools/arrow2tsfile.sh --source input.arrow --target output
+$TSFILE_HOME/tools/csv2tsfile.sh --source input-dir --target output \
+ --fail_dir failed --schema import.schema
+```
+
+Use the corresponding `.bat` scripts on Windows. Omit `--schema` only when the
+current auto-inference rules are acceptable. Read `java/tools/README.md` before
+generating a schema or relying on time-column, tag-column, null, delimiter,
+type-inference, file-naming, or failure-handling behavior.
+
+Use the C++ `tsfile-cli write` command only when the requested operation is its
+narrow pipe-friendly case: one CSV/TSV stream or file, one new table-model
+TsFile, and an explicit inline `--columns` schema.
+
+## Check or Backfill Table Point Count
+
+Use the Java point-count tool for the persisted table-level property:
+
+```sh
+$TSFILE_HOME/tools/tsfile-table-point-count.sh /data/example.tsfile
+```
+
+Use the `.bat` wrapper on Windows. Pass exactly one complete TsFile. The count
+is the total number of non-null FIELD values for each table; TAG and time
+columns are excluded.
+
+Interpret `UPDATED` as an in-place metadata backfill, `ALREADY_PRESENT` as no
+change, and `NO_TABLE` as a tree-only/no-table result. Before running it,
+resolve the exact input path, ensure the containing directory is writable and
+has room for a complete temporary copy, and preserve a backup when the data is
+not otherwise recoverable. Missing and incomplete files must remain errors.
+
+Do not use C++ `tsfile-cli count` as a substitute. That command reads
+per-series statistics and prints counts; it neither validates nor writes the
+table-level point-count property.
+
+## Source Anchors
+
+- Import behavior: `java/tools/README.md`,
+ `java/tools/src/main/java/org/apache/tsfile/tools/`, and its tests.
+- Import wrappers: `java/tools/src/assembly/resources/tools/`.
+- Point-count implementation:
+
`java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileTablePointCountTool.java`.
+- Point-count wrapper and user contract: `java/tools/README.md` and
+ `java/tools/src/assembly/resources/tools/tsfile-table-point-count.*`.
diff --git a/skills/tsfile/references/performance.md
b/skills/tsfile/references/performance.md
index 6972f0aad..6a745844e 100644
--- a/skills/tsfile/references/performance.md
+++ b/skills/tsfile/references/performance.md
@@ -44,6 +44,8 @@ than a universal setting.
## Encoding and Compression
+- Read `references/encoding-compression.md` for the current constructible
+ factory/type matrix. Do not infer factory support from enum membership.
- Start from the implementation defaults for the target version and language.
- Match encodings to observed data distribution only after measuring: monotonic
integers, floating-point continuity, boolean/cardinality patterns, and text
diff --git a/skills/tsfile/references/properties-filters.md
b/skills/tsfile/references/properties-filters.md
new file mode 100644
index 000000000..6fd461d24
--- /dev/null
+++ b/skills/tsfile/references/properties-filters.md
@@ -0,0 +1,80 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# File Properties and TAG Filters
+
+## File-level Properties
+
+Treat property values as untyped binary application metadata. Add or replace
+them only while a writer is open, close the writer to persist them, and define
+an explicit portable encoding for numbers or structures.
+
+- Java: call `ITsFileWriter.addTsFileProperty(String, byte[])` and
+ `ITsFileReader.getTsFileProperties()`.
+- Python: call `add_tsfile_property(str, bytes)` and
+ `get_tsfile_properties()`. The setter accepts exactly `bytes`; the reader
+ preserves null as `None` and a non-null empty value as `b""`.
+- C++: call `add_tsfile_property(...)` on `TsFileWriter` or
+ `TsFileTableWriter`, and `TsFileReader::get_tsfile_properties()`. The
+ `TsFileProperties` value distinguishes null from an empty byte vector.
+- C: use the declarations and matching free function in
+ `cpp/src/cwrapper/tsfile_cwrapper.h`; do not guess buffer ownership.
+
+The Java table point-count metadata is a reserved use of file properties. Use
+`references/java-tools.md` for inspection or backfill instead of editing those
+keys directly.
+
+## Table TAG Filters
+
+Build TAG predicates against a `TableSchema`, apply them only to Table-model
+queries, and pass field/time restrictions through their separate query
+arguments. TAG values are strings; do not apply a TAG builder to FIELD columns.
+
+- Java `TagFilterBuilder`: `eq`, `neq`, `lt`, `lteq`, `gt`, `gteq`,
+ `betweenAnd`, `notBetweenAnd`, `regExp`, `notRegExp`, `like`, `notLike`, plus
+ `and`, `or`, and `not`. Pass the result to the five-argument
+ `ITsFileReader.query(..., Filter tagFilter)` overload.
+- Python factories: `tag_eq`, `tag_neq`, `tag_lt`, `tag_lteq`, `tag_gt`,
+ `tag_gteq`, `tag_regexp`, `tag_not_regexp`, `tag_is_null`,
+ `tag_is_not_null`, `tag_between`, and `tag_not_between`. Compose with `&`,
+ `|`, and `~`, then pass `tag_filter=` to `query_table` or
+ `query_table_by_row`.
+- C++ `TagFilterBuilder`: use `eq`/`neq`/comparison, regex, null, range, and
+ static logical builders from `cpp/src/reader/filter/tag_filter.h`; pass the
+ resulting `Filter*` to the matching `TsFileReader::query` or `queryByRow`
+ overload and follow the current ownership implementation.
+- C wrapper: construct `TagFilterHandle` values with the declared factory and
+ composition functions, pass them to the tag-filter query entry point, and
+ release them with `tsfile_tag_filter_free`.
+
+Do not translate operator names mechanically across bindings: Java additionally
+exposes LIKE, while current C++/Python expose explicit null predicates.
+
+## Source Anchors
+
+- Java:
`java/tsfile/src/main/java/org/apache/tsfile/read/filter/factory/TagFilterBuilder.java`,
+ `java/tsfile/src/main/java/org/apache/tsfile/read/v4/ITsFileReader.java`, and
+ `java/examples/src/main/java/org/apache/tsfile/v4/TagFilterExample.java`
+- Python: `python/tsfile/tag_filter.py`, `python/tsfile/tsfile_reader.pyx`, and
+ `python/tests/test_tag_filter*.py`
+- C/C++: `cpp/src/reader/filter/tag_filter.h`,
+ `cpp/src/reader/tsfile_reader.h`, and `cpp/src/cwrapper/tsfile_cwrapper.h`
+- Properties: the current writer/reader interfaces plus
+ `python/tests/test_tsfile_properties.py` and
+ `cpp/test/writer/tsfile_properties_test.cc`
