GitHub user zhztheplayer edited a comment on the discussion: Flink support
Thank everyone putting efforts on Flink support. Glad to see Flink experts willing to join. For the implementation, my 2 cents so far is to extract a Java accelerator library out for our next framework support. I have been working on one https://github.com/velox4j/velox4j at which you may want to take a look. @shuai-xu Substrait haven't been helping much on the extensibility of Gluten. We have to do a bunch of customizations on it that makes Gluten difficult to integrate with other Susbtrait consumer libraries. Moreover, Velox and CH both have incompatible usages on same Substrait features, for example the `optimization` / `enhancement` and so, which make it difficult to integrate a new framework with both VL / CH backends at the same time. My feeling is a layered design will be the better arch for such accelerators. We build a Java library that tightly binds to the native libraries through JNI, then translate framework's query plan into library's query plan in Java in one shot. Things will be made much clearer and maintenance will be much simpler with such architecture. GitHub link: https://github.com/apache/incubator-gluten/discussions/8849#discussioncomment-12346751 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
