KKcorps opened a new pull request, #19490:
URL: https://github.com/apache/pinot/pull/19490
Stacked on #19488 (the first commit here is that PR; review the second
commit).
## TL;DR
Every minion METADATA push uploads the segment tar to the output filesystem
and then downloads the
same tar back to extract `metadata.properties` and `creation.meta`. This PR
builds that two-file
metadata tar from the local converted segment before the segment directory
is deleted, and adds
`SegmentPushUtils` overloads that push caller-supplied metadata tars. Both
minion base executors
use them. Wire behavior is unchanged: same headers, same URIs, same
controller handling. Minion S3
traffic per pushed segment halves and the full-segment copy in
`java.io.tmpdir` goes away.
## The problem
`SegmentPushUtils.sendSegmentUriAndMetadata(spec, fileSystem, uriToTarPath,
...)` and its batch
twin take a map of staged tar paths and call
`generateSegmentMetadataFile(fileSystem, tarUri)`,
which copies the whole tar into `java.io.tmpdir`, scans the gzip stream
twice with `untarOneFile`,
tars the two entries and deletes the copy. The `preferMetadataTarGz`
shortcut only helps when a
`<segment>.metadata.tar.gz` sidecar already exists next to the tar, and the
minion executors never
write one. So for a task that pushes N segments of size S in METADATA mode,
the minion moves
2·N·S bytes: N·S up, N·S back down. For hundreds-of-MB segments that is the
dominant cost of the
push step, and the temporary full copy lands on the minion's
`java.io.tmpdir`.
The two files are already on local disk at that point. The executor tars the
converted segment
directory and only then deletes it.
## The approach
1. `SegmentPushUtils.generateSegmentMetadataFile(File segmentDir, File
outputDir, String segmentName)`
copies `metadata.properties` and `creation.meta` out of a local segment
directory (v1/v3 layouts
via `SegmentDirectoryPaths`) and tars them as
`<segmentName>.metadata.tar.gz`.
2. `sendSegmentUriAndMetadata(spec, Map<String, File>
segmentUriToMetadataFile, headers, params)`
and the batch `sendSegmentsUriAndMetadata(...)` twin push caller-supplied
metadata tars. The
segment name comes from the metadata file name, the caller keeps
ownership of the files. The
existing `PinotFS` overloads are unchanged in behavior and now share one
private push loop with
the new ones, so the retry, header and batch-tar logic exists once.
3. `BaseSingleSegmentConversionExecutor` builds the metadata tar right after
the conversion for
every METADATA push (the controller-copy path from #19488 already did),
and the default path
pushes it with the new overload instead of the `PinotFS` one.
4. `BaseMultipleSegmentsConversionExecutor` builds one metadata tar per
output segment in the tar
loop, keeps them alongside the tars, pushes them with the new overloads
in both the per-segment
and the batch mode, and deletes them with the tars.
```mermaid
flowchart LR
subgraph Before["Before"]
A[tar converted segment] --> B[upload tar to output FS]
B --> C[download tar back to java.io.tmpdir]
C --> D[extract 2 files, tar them]
D --> E[POST metadata]
end
subgraph After["After"]
F[tar converted segment] --> G[copy 2 files from local dir, tar them]
G --> H[upload segment tar to output FS]
H --> I[POST metadata]
end
```
## Key components
| Class / file | Role |
|---|---|
| `SegmentPushUtils` | `generateSegmentMetadataFile(File, File, String)`,
the two local-metadata push overloads, shared private `pushSegmentMetadata` /
`pushSegmentsMetadata` loops, `getSegmentMetadataFile` (sidecar-or-download)
used by both `PinotFS` paths. |
| `BaseSingleSegmentConversionExecutor` | Builds the metadata tar for every
METADATA push and hands it to the default path. |
| `BaseMultipleSegmentsConversionExecutor` | Builds one metadata tar per
output segment, pushes with the new overloads. `updateSegmentUriToTarPathMap`
becomes `getSegmentUris`. |
| `BaseTaskExecutor` | `createSegmentMetadataTarFile` from #19488 moves into
`SegmentPushUtils`. |
## Compatibility
- Wire behavior is identical. The controller receives the same metadata tar
contents, the same
`DOWNLOAD_URI`, `UPLOAD_TYPE` and copy flag, and the same batch tar layout.
- The existing `PinotFS` overloads keep their signatures and behavior for
external callers (the
ingestion job runners, StarTree's ingestion tasks).
- No new config keys. `preferMetadataTarGz` still applies to the `PinotFS`
overloads.
## Testing
- `SegmentPushUtilsTest`: local metadata tar contents from a v3 layout (data
file excluded); single
and batch local-metadata pushes against the TLS test server, checking
`UPLOAD_TYPE`,
`DOWNLOAD_URI`, the copy flag and that the caller's files survive;
rejection of a metadata file
not named `<segment>.metadata.tar.gz`.
- `BaseSingleSegmentConversionExecutorTest`: the default METADATA path
pushes a locally built
metadata tar describing the converted segment through the new overload and
never calls the
`PinotFS` overload.
- `BaseMultipleSegmentsConversionExecutorTest`: end-to-end `executeTask` on
METADATA push, per-segment
and batch, with two output segments: one metadata tar per segment named
after it, containing
exactly the two files and describing that segment; the `PinotFS` overloads
are never called; the
staged tars stay in the output dir as the download URLs.
- Integration: `PurgeMinionClusterIntegrationTest` (single executor,
METADATA table),
`RealtimeToOfflineSegmentsMinionClusterIntegrationTest#testRealtimeToOfflineSegmentsMetadataPushTask`
and
`MergeRollupMinionClusterIntegrationTest#testOfflineTableSingleLevelConcatWithMetadataPush`
(multi executor, per-segment and batch) on a local-FS cluster.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Y375AgHYsh1YqNSsfvF1a8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
