lksvenoy-r7 commented on issue #8889: URL: https://github.com/apache/pinot/issues/8889#issuecomment-1160308891
Some additional information regarding segment creation: I was speaking with @mayankshriv regarding extending the segment index creation driver to allow in-memory segment creation, but unfortunately the interface implements File getOutputDirectory(), meaning we are bound to using a file. One approach which could work is to abstract away File, and use the Pinot filesystem to allow the output to be elsewhere. A general improvement to the architecture would be to concretely return types which have to pass through PinotFS. As it stands, since PinotFS takes a generic java URI, there's no way of knowing if a specific URI has to pass through this pattern. A good inspiration for this would be to look at the way flink handles abstracting away the file system. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
