Thanks Aldrin for the pointers. Did I understand the effort correctly in that it deals with accessing arrow enabled data via Spark. What I have is a Java based Spark DataFrame and I need to go the other direction, convert this DataFrame to an arrow format so that I can server it via Arrow Flight…do you think this could be achieved with the arrow-spark module you have pointed to…Thanks for your suggestions. Sitaraman
From: Aldrin <[email protected]> Date: Thursday, February 9, 2023 at 10:31 AM To: [email protected] <[email protected]> Subject: Re: Transferring a spark data frame from Java to Python using Arrow, ArrowFlight. ***** EXTERNAL EMAIL ***** Hello! This repo [1] and this paper [2] may be relevant. [1]: https://github.com/Sebastiaan-Alvarez-Rodriguez/arrow-spark-publication<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSebastiaan-Alvarez-Rodriguez%2Farrow-spark-publication&data=05%7C01%7Cvilayannur.sitaraman%40hitachivantara.com%7C06b7529de02e437369ec08db0acbd268%7C18791e1761594f52a8d4de814ca8284a%7C0%7C0%7C638115642904637214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MibjINiAdsKOVDjRyDaAENFvoqPULZnOjBYfEehBOR0%3D&reserved=0> [2]: https://arxiv.org/pdf/2106.13020.pdf<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Farxiv.org%2Fpdf%2F2106.13020.pdf&data=05%7C01%7Cvilayannur.sitaraman%40hitachivantara.com%7C06b7529de02e437369ec08db0acbd268%7C18791e1761594f52a8d4de814ca8284a%7C0%7C0%7C638115642904637214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XF4EtdA5pToVbWL2Al%2BXGqnvWKGo42W0lsGcGRkX8Bg%3D&reserved=0> Aldrin Montana Computer Science PhD Student UC Santa Cruz On Wed, Feb 8, 2023 at 7:11 PM Vilayannur Sitaraman <[email protected]<mailto:[email protected]>> wrote: Hi, I just successfully wrote my first flight server and client that transfers data read from and arrow file from Java Server to Python Client. I would like to be able to transfer a Spark DataFrame created in Java to Python using Arrow and ArrowFlight. If I can convert a Spark Dataframe created in java to an Arrow file format then I can use the above created flight server and python client to do the transfer. But I am not sure how to convert a Spark dataframe created in Java to Arrow format in a Java module. Any help/pointers appreciated. Sitaraman
