XiaoHongbo-Hope commented on code in PR #8731:
URL: https://github.com/apache/paimon/pull/8731#discussion_r3683214226
##########
docs/docs/pypaimon/pytorch.md:
##########
@@ -59,6 +59,24 @@ when it is false, it will read the full amount of data into
memory.
**`prefetch_concurrency`** (default: 1): When streaming is true, number of
threads used for parallel prefetch within each DataLoader worker. Set to a
value greater than 1 to partition splits across threads and increase read
throughput. Has no effect when streaming is false.
+## File Format Metadata Cache
+
+Reusable PyArrow Dataset metadata is cached across reads. Configure its
estimated
+size limit with:
+
+```python
+table = table.copy({
+ "file-format.metadata-cache.max-size": "50 mb",
Review Comment:
> Does this look like a catalog-level parameter?
Thanks, agreed. This should be catalog-level，updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
