shou123 commented on issue #36962: URL: https://github.com/apache/arrow/issues/36962#issuecomment-1660983216
> Hi @shou123, > > The `SkyhookFileFormat` was implemented by the authors as an extension to pyarrow. It is not part of the pyarrow codebase. We can consider this ticket as an enhancement request instead of as a bug. > > From the research paper: `"We extend the FileFormat API in Arrow to create a SkyhookFileFormat API that leverages the DOA API to enable offloading fragment scan operations to Ceph OSDs."`[1] > > [1] https://arxiv.org/pdf/2204.06074.pdf Thank you @danepitkin. Do you know how to find the pyarrow skyhook source code? As I understand, they implement skyhook as C++, but no python version. But there are some skyhook implement by using pyarrow. Such as: [](https://gist.github.com/JayjeetAtGithub/0798e1d5aad666f20f5c9e3385c6f774) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
