lucasfang commented on code in PR #272: URL: https://github.com/apache/paimon-cpp/pull/272#discussion_r3966701132
########## src/paimon/common/utils/file_block_cache.h: ########## @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#pragma once + +#include <atomic> +#include <cstdint> +#include <future> +#include <memory> +#include <mutex> +#include <unordered_map> + +#include "paimon/common/utils/read_ahead_cache.h" +#include "paimon/fs/file_system.h" +#include "paimon/memory/bytes.h" +#include "paimon/memory/memory_pool.h" +#include "paimon/status.h" +#include "paimon/visibility.h" + +namespace paimon { + +/// A cache of fixed-size blocks of one file, serving the reads that no +/// prefetched range covers: a parquet reader reads the footer and the page index +/// before any range can be registered, and every reader of the file reads the +/// same bytes. +/// +/// Blocks are aligned to the END of the file: block 0 is +/// [file_size - block_size, file_size). The metadata of a parquet/orc file lives +/// in its tail and arrow reads exactly the last 64 KiB as the footer, so an Review Comment: ORC does not go through arrow, but it does go through this cache: the prefetch reader is format-agnostic (`AbstractSplitRead::CreateFileBatchReader` wraps every format except `blob`, `avro` and `mosaic`), and the ORC file is read by the ORC C++ library through `OrcInputStreamImpl`. Its tail reads are: - The postscript read: the last `DIRECTORY_SIZE_GUESS = 16 KiB` of the file, in `orc/c++/src/Reader.cc`. That falls inside the last 64 KiB block, so all the readers of the file share one fetch for it, which is the same benefit parquet gets. - The footer read, issued only when the postscript plus the footer do not fit into those 16 KiB: `footerSize` bytes at `fileLength - tailSize`, an offset and a size of its own. It can straddle two blocks or exceed one, and `CanServe()` declines both, so that read goes to the stream uncached, like any read one block cannot serve. So the bad case for ORC is a lost caching opportunity for one read, never a wrong result or a failed read: `Read()` returns false with `dest` untouched and ORC reads the bytes itself. A format that issues no metadata read at all simply never asks the block cache for anything. To answer the question behind it directly: yes, a read larger than `block_size`, or straddling two blocks, is not cached at all — `CanServe()` declines it by design, because one block serving one read is what keeps a served read to a single fetch and a single copy, and keeps a declined read free of partial-hit assembly. The block cache is the fallback for the uncovered reads of a file, whose granularity is set by `block_size` and whose total is bounded by `block_cache_limit`; it is not a general-purpose file cache, and a caller whose read is bigger than a block keeps reading that read from the stream, exactly as before this PR. Serving large uncovered reads would mean splitting a request across blocks, fetching each with its own single flight and assembling the copies, which I would rather keep out of this PR; I can file a follow-up issue if you want it. Since none of this is specific to one format, the header comment no longer quotes the parquet or the ORC read sizes. It now states the mechanism — the fallback below the prefetched ranges, blocks aligned to the end of the file, one block per read and what is declined — and `CacheConfig::block_size_`'s comment states the granularity trade-off instead of naming a reader. The format-specific numbers are answered here, in the thread that asked for them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
