kkrugler commented on issue #7791:
URL: https://github.com/apache/pinot/issues/7791#issuecomment-978403625


   @richardstartin - yes, re aborting the download once the two files of 
interest (`metadata.properties` and `creation.meta`) have been extracted. This 
will still work even if these files aren't near the beginning of the file, or 
before the bigger data files. It will just not be as much of an optimization, 
but even in the worst case it will be faster than the current code (no creation 
of temp files on local disk).
   
   As to what happens with copying of files - it's not really the same as 
messing with transaction logs. Ops people (like the team for one of our 
clients) copy around segment files all the time, as they try out our different 
optimizations for how data is indexed, or manually roll back after there's an 
issue with the data being used to build new segments.
   
   Re where to put the metadata files - If the two metadata files are packaged 
as as currently expected by the controller, then they'll be `<blah>.tar.gz` 
files. In that case they really shouldn't be in the same directory as the 
actual segment tarballs, as segment pushes will try to push everything with 
that suffix to the controller. If you do a metadata push then it could rely on 
a naming convention to only process the metadata files.
   
   You could leave the files untarred, but now you have a bunch of 
`<name>-metadata.properties` and `<name>-creation.meta` files littering the 
segment directory. We have HDFS directories with 1200+ segments, so adding 
another 2400 of these tiny files doesn't feel clean.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to