Hi guys
I'm opening this thread to discuss supporting the Apache Arrow format
in Flink.
Arrow is a language-independent columnar memory format that has become
widely used in different systems, and It can also serve as an
inter-exchange format between other systems.
So, using it directly in the Flink system will be nice. We also received
some requests from slack[1][2] and jira[3].
In our company's internal usage, we have used flink-python moudle's
ArrowReader and ArrowWriter to support this work. But it still can not
integrate with the current flink-formats framework closely.
So, I'd like to introduce the flink-arrow formats module to support the
arrow format naturally.
Looking forward to some suggestions.
Best,
Aitozi
[1]: https://apache-flink.slack.com/archives/C03GV7L3G2C/p1677915016551629
[2]: https://apache-flink.slack.com/archives/C03GV7L3G2C/p1666326443152789
[3]: https://issues.apache.org/jira/browse/FLINK-10929
