jja725 commented on PR #1541: URL: https://github.com/apache/datafusion-ballista/pull/1541#issuecomment-4187534330
> Thanks for contribution @jja725, as I mentioned in the original task i think support for shuffle server makes sense. > > I would like to suggest few changes to this PR. > > I would suggest to move newly added crate `riffle` to standalone project and implement it as extension to ballista. Once we have it running we could move it to https://github.com/datafusion-contrib . > > There are two reasons for this, > > * Firstly there are multiple shuffle server implementations which could be supported and we should not pick winner. > * Secondly, ballista struggles with maintenance capacity and this would add burden to active maintainers. > > The changes which support this use case and are generic should be part of this PR, others, use case specific should be moved to external project. > > there are few examples which we can follow for ballista extensions > > 1. https://github.com/milenkovicm/ballista_delta > 2. https://github.com/milenkovicm/ballista_extensions > > We could > > * move specific riffle configuration changes to extension configuration > * replace actual shuffle read/write implementation in executor using `ExecutionEngine` or similar > > I can help you and support you to make this ballista extension > > Let me know what you think Yeah it make sense. Appreciate the help to work through this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
