Re: Java - Spark dataframe to Arrow format

2019-12-06 Thread GaoXiang Wang
a/org/apache/spark/sql/vectorized/ArrowColumnVector.java > > * > > > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala > > * > > > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scal

Re: Java - Spark dataframe to Arrow format

2019-12-05 Thread GaoXiang Wang
Hi Jeetendra and Liya, I am actually having a similar use case. We have some data stored as *parquet format in HDFS* and would like to make use of Apache Arrow to improve compute performance if possible. Right now, I didn't see there is a direct way to do in Java with Spark. I have search the Spa