liaoxin01 opened a new pull request, #68089:
URL: https://github.com/apache/doris/pull/68089
### What problem does this PR solve?
Related PR: #57922
Problem Summary:
#57922 moved file cache TTL management from block granularity to tablet
granularity. As part of that change every path stopped computing an
expiration
timestamp and started passing the raw ttl_seconds instead:
expiration_time = tablet_meta->ttl_seconds();
The receiving side was never updated. KeyMeta::expiration_time still
documents
itself as an absolute time, and that value is what gets persisted into the
block
meta store and compared by the cache consistency check. Those fields now
hold a
duration such as 3600 rather than a timestamp, so the only information left
in
them is "non-zero means this is a TTL block".
Meanwhile the expiration decision itself moved into BlockFileCacheTtlMgr,
which
sweeps by tablet_ctime + ttl < now, anchored at the tablet creation time.
Two consequences:
1. Blocks are created as TTL blocks regardless of whether the tablet is still
within its TTL window. Once a tablet is past creation_time + ttl_seconds,
the
load, compaction and query paths keep putting blocks into the TTL queue
and
the background sweep keeps pulling them straight back out. Each conversion
writes the block meta store and takes the cache lock to move the block
between LRU queues, and the loop never terminates for the rest of the
tablet's life.
2. The two writer paths disagreed with each other. The packed file path in
RowsetWriterContext still computed an absolute newest_write_timestamp +
ttl,
while the regular file writer path passed the relative ttl_seconds, so
segment files and packed small files of the same rowset were stamped with
different kinds of value. check_file_cache_consistency reports that as
EXPIRATION_TIME_INCONSISTENT.
This PR keeps the tablet creation time as the anchor and gives that deadline
a
single definition, TabletMeta::file_cache_ttl_expiration_time(), which
returns
the absolute creation_time + ttl_seconds, or 0 when the tablet has no TTL or
is
already past the deadline. The load, compaction, schema change, query and
warm
up paths all stamp the blocks they create with that value, so a block's
recorded
expiration time now agrees with the sweep that acts on it, and a tablet past
its
deadline has its blocks created as NORMAL directly instead of cycling through
the TTL queue.
BlockFileCacheTtlMgr additionally no longer promotes the blocks of an already
expired tablet on the edge where it first sees that tablet, which previously
caused one full promote/demote pass per TTL tablet after every BE restart.
The existing regression tests could not catch this: they all create a table
and
load into it immediately, so the tablet creation time and the data write time
coincide and the bug is invisible.
### Release note
Fixed the file cache TTL expiration time being computed inconsistently across
the load, compaction, query and warm up paths. Data belonging to a tablet
that
is past its TTL deadline is now written directly into the normal cache queue
instead of repeatedly entering and leaving the TTL queue.
### Check List (For Author)
- Test: Unit Test, Regression test. Both are added in this PR but have NOT
been
executed locally yet, so they still need to pass in CI.
- Added TabletMetaTest.FileCacheTtlExpirationTime, covering no TTL,
unknown
creation time, a live tablet, an expired tablet, the exact deadline,
and a
ttl large enough to overflow.
- Added
regression-test/suites/cloud_p0/cache/ttl/test_ttl_expired_tablet.groovy,
which lets a tablet pass its deadline before loading anything, then
asserts
every block lands in the normal queue and that ttl_cache_size stays at 0
across repeated samples, i.e. no promote/demote churn.
- Adjusted alter_ttl_seconds.groovy. It uses ttl=5 and the load itself
outlives the deadline, so its intermediate "wait until all blocks are
ttl"
assertion no longer holds by design. Its remaining assertions are
unchanged
and are now stronger.
- All modified translation units were verified to compile, and
clang-format
reports no changes for the modified files.
- Behavior changed: Yes. Blocks of a tablet past its TTL deadline are now
created
as NORMAL directly, rather than created as TTL and later demoted by the
background sweep. The deadline itself is unchanged: it remains the tablet
creation time plus file_cache_ttl_seconds.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
