xiangfu0 commented on code in PR #19898:
URL: https://github.com/apache/pinot/pull/19898#discussion_r4289523224
##########
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/java/org/apache/pinot/plugin/ingestion/batch/standalone/SegmentGenerationJobRunner.java:
##########
@@ -230,9 +232,17 @@ private void submitSegmentGenTask(File localTempDir, URI
inputFileURI, int seqId
File localOutputTempDir = new File(localTempDir, "output");
FileUtils.forceMkdir(localOutputTempDir);
- //copy input path to local
- File localInputDataFile = createLocalInputDateFile(inputFileURI,
localInputTempDir);
- _inputDirFS.copyToLocalFile(inputFileURI, localInputDataFile);
+ // Gzip readers can unpack beside the input, so keep those writes inside
an owned staging directory.
+ File inputFile = new File(URIUtils.decode(inputFileURI.getRawPath()));
+ boolean borrowedInputFile =
PinotFSFactory.isFileSystemInstanceOf(_inputDirFS, LocalPinotFS.class)
+ && !RecordReaderUtils.isGZippedFile(inputFile);
Review Comment:
As explained in [the earlier
reply](https://github.com/apache/pinot/pull/19898#discussion_r4289317967), the
old copy and the gzip probe both run before task submission, outside the
per-file generation wrapper. The existing missing-path diagnostic is retained,
so I am leaving this repeated optional suggestion unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
