xiangfu0 commented on code in PR #19898:
URL: https://github.com/apache/pinot/pull/19898#discussion_r4289440203
##########
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/java/org/apache/pinot/plugin/ingestion/batch/standalone/SegmentGenerationJobRunner.java:
##########
@@ -230,9 +232,17 @@ private void submitSegmentGenTask(File localTempDir, URI
inputFileURI, int seqId
File localOutputTempDir = new File(localTempDir, "output");
FileUtils.forceMkdir(localOutputTempDir);
- //copy input path to local
- File localInputDataFile = createLocalInputDateFile(inputFileURI,
localInputTempDir);
- _inputDirFS.copyToLocalFile(inputFileURI, localInputDataFile);
+ // Gzip readers can unpack beside the input, so keep those writes inside
an owned staging directory.
+ File inputFile = new File(URIUtils.decode(inputFileURI.getRawPath()));
+ boolean borrowedInputFile =
PinotFSFactory.isFileSystemInstanceOf(_inputDirFS, LocalPinotFS.class)
+ && !RecordReaderUtils.isGZippedFile(inputFile);
Review Comment:
Still the one remaining optional nit from last round: `isGZippedFile` opens
the file here, so an input that vanishes between listing and this call surfaces
as a raw `FileNotFoundException` rather than the clearer per-file "Failed to
generate" failure. Fine to leave as-is — just flagging it's unaddressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
