[
https://issues.apache.org/jira/browse/SPARK-20960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Song Jun updated SPARK-20960:
-----------------------------
Description:
_emphasized text_ColumnVector is an internal interface in Spark SQL, which is
only used for vectorized parquet reader to represent the in-memory columnar
format.
In Spark 2.3 we want to make ColumnVector public, so that we can provide a more
efficient way for data exchanges between Spark and external systems. For
example, we can use ColumnVector to build the columnar read API in data source
framework, we can use ColumnVector to build a more efficient UDF API, etc.
We also want to introduce a new ColumnVector implementation based on Apache
Arrow(basically just a wrapper over Arrow), so that external systems(like
Python Pandas DataFrame) can build ColumnVector very easily.
was:
ColumnVector is an internal interface in Spark SQL, which is only used for
vectorized parquet reader to represent the in-memory columnar format.
In Spark 2.3 we want to make ColumnVector public, so that we can provide a more
efficient way for data exchanges between Spark and external systems. For
example, we can use ColumnVector to build the columnar read API in data source
framework, we can use ColumnVector to build a more efficient UDF API, etc.
We also want to introduce a new ColumnVector implementation based on Apache
Arrow(basically just a wrapper over Arrow), so that external systems(like
Python Pandas DataFrame) can build ColumnVector very easily.
> make ColumnVector public
> ------------------------
>
> Key: SPARK-20960
> URL: https://issues.apache.org/jira/browse/SPARK-20960
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Wenchen Fan
>
> _emphasized text_ColumnVector is an internal interface in Spark SQL, which is
> only used for vectorized parquet reader to represent the in-memory columnar
> format.
> In Spark 2.3 we want to make ColumnVector public, so that we can provide a
> more efficient way for data exchanges between Spark and external systems. For
> example, we can use ColumnVector to build the columnar read API in data
> source framework, we can use ColumnVector to build a more efficient UDF API,
> etc.
> We also want to introduce a new ColumnVector implementation based on Apache
> Arrow(basically just a wrapper over Arrow), so that external systems(like
> Python Pandas DataFrame) can build ColumnVector very easily.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
