[
https://issues.apache.org/jira/browse/CASSANALYTICS-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111341#comment-18111341
]
Yifan Cai commented on CASSANALYTICS-104:
-----------------------------------------
+1
> Eliminate redundant filesystem lookups in SSTable direct streaming
> ------------------------------------------------------------------
>
> Key: CASSANALYTICS-104
> URL: https://issues.apache.org/jira/browse/CASSANALYTICS-104
> Project: Apache Cassandra Analytics
> Issue Type: Improvement
> Components: Writer
> Reporter: Yifan Cai
> Assignee: Lukasz Antoniak
> Priority: Normal
> Fix For: 0.5
>
> Attachments: image-2026-09-03-15-33-26-025.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Each SSTable currently triggers 2-3 redundant filesystem directory scans when
> running in the DIRECT mode to bulk write:
> # SortedSSTableWriter.prepareSStablesToSend() - scans directory, calculates
> digests,returns flat Map<Path, Digest>
> # DirectStreamSession.sendSSTableToOneReplica() - scans directory again with
> globpattern to find components
> # SortedSSTableWriter.close() → calculateFileDigestMap() - scans directory
> again foreach data file
> Additionally, prepareSStablesToSend() returns a flat map that discards
> SSTable componentgrouping, which must be reconstructed via additional
> filesystem lookups.
> The solution proposed is grouping the SSTable components by its base name to
> avoid the extra file system directory scans. Basically, revising the data
> structure from Map<Path, Digest> to Map<String, Map<Path, Digest>> (string
> key is the SSTable base name).
> It should reduce the filesystem scans, as well as eliminate glob pattern
> matching overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
