github-actions[bot] commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4231053556
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -46,8 +57,30 @@
* but their graphs are catalog-scoped executable services rather than
entry-owned metadata. Walking
* those graphs both double-counts shared state and reaches strongly
encapsulated JDK objects. The
* estimator therefore expands only immutable metadata owned by the entry.
+ *
+ * <p>A cached value is weighed once, at admission, but a {@link
FileStoreTable} keeps growing afterwards: the
+ * first {@code latestSnapshot()} or scan builds its transient store (a {@code
FileStore} holding its own
+ * {@code RowType} copies of the schema and a copy of the options) and field
lookups build the four lazy index
+ * maps of every {@code RowType}. That growth is reserved from the schema's
shape. On Paimon 1.4.2 it measures
+ * about 1-2 KB per table, 200-270 bytes per field, another 200 bytes per
field nested in a row and 40 bytes per
+ * option; the constants below keep a margin above that.
*/
final class PaimonCacheSizeEstimator {
+ private static final long STORE_GROWTH_BASE_BYTES = 2048L;
+ private static final long STORE_GROWTH_FIELD_BYTES = 320L;
+ private static final long STORE_GROWTH_NESTED_FIELD_BYTES = 256L;
+ // A child type of an array, map, multiset or vector: the store's copy of
it.
+ private static final long STORE_GROWTH_TYPE_NODE_BYTES = 128L;
+ private static final long STORE_GROWTH_OPTION_BYTES = 64L;
+ // A partition, primary or bucket key: its slot in the derived key types;
a primary key is also copied as a
+ // renamed key field.
+ private static final long STORE_GROWTH_KEY_BYTES = 192L;
+ // RESTCatalog with data tokens gives each table its own RESTTokenFileIO.
The FileIO it delegates to lives in
+ // Paimon's process-wide cache, but the vended token it fetches on first
data access (an STS key id, secret and
+ // security token, about 2 KB) belongs to the table and arrives after
admission.
+ private static final long REST_TOKEN_FILE_IO_BYTES =
+
JvmSizeUtils.saturatedAdd(JvmSizeUtils.instanceSize(RESTTokenFileIO.class),
4096L);
Review Comment:
[P2] Account for REST tokens larger than the fixed reserve. With
`paimon.data-token.enabled=true` and a metadata weight limit, the REST table is
admitted before scan planning calls `validToken()` on its retained
`RESTTokenFileIO`. Paimon 1.4.2 stores the server's unrestricted token map
there, and this 4 KB allowance is never reweighed. A credential larger than the
allowance leaves the table cached below its actual retained weight; across many
tables, `meta.cache.max-weight` can be exceeded. Please reweigh token
replacement or decline weighted retention for token-bearing tables, and cover a
long first-fetch or refresh token in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
