[
https://issues.apache.org/jira/browse/ARROW-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajeshwar Agrawal updated ARROW-3949:
-------------------------------------
Description:
It would be a great to have examples of using parquet arrow high-level API for
the following 2 cases
* Storing nested data types (storing nested data types is touted as major
merit of parquet, so I think this case should be included as an example).
Ideally, an example of how to use {{arrow::StructArray}} nested with several
primities types, list types and other nested types would cover every case of
nested hierarchy of complex data representations
* Buffered or Batched writes to parquet file. Parquet is meant to be used for
large amounts of data. The current examples stores all of the data as in arrow
data structures, before writing to parquet file, which has a huge memory
footprint, proportional to the amount of data being stored. An example of
writing directly to row groups and columns, can nicely demonstrate how to store
data with smaller memory footprint. The current example creates a
{{arrow::Table}}, which needs to be filled with {{arrow::Array}}(s) of entire
data, size of which is bounded by the amount of RAM. Ideally, an example which
generates some data in several {{arrow::Array}}(s), and then stores (appends)
them as a new Row Group (or Column Chunk) in a {{parquet::arrow::FileWriter}},
using {{NewRowGroup}} and {{WriteColumnChunk}} functions, thus demonstrating a
lower memory footprint for writing a parquet file with huge amounts of data
was:
It would be a great help to have examples of using parquet arrow high-level API
for the following 2 cases
* Storing nested data types (storing nested data types is touted as major
merit of parquet, so I think this case should be included as an example).
Ideally, an example of how to use StructArray nested with several primities
types, list types and other struct type would cover every case of nested
hierarchy of complex data representations
* Buffered or Batched writes to parquet file. Parquet is meant to be used for
large amounts of data. The current examples store all of the data as in arrow
data structures, before writing to parquet file. Would be great to include an
example of batched writes, which is helpful in most use cases of parquet. The
current example creates a {{arrow::Table}}, which needs to be filled with
{{arrow::Array}}(s) of entire data. Ideally, an example which generates some
data in several {{arrow::Array}}(s), and then stores (appends) them as a new
Row Group (or Column Chunk) in an existing (new) parquet file (writer), using
{{NewRowGroup}} and {{WriteColumnChunk}} functions, thus demonstrating a lower
memory footprint for writing a parquet file
> parquet cpp - improve examples
> ------------------------------
>
> Key: ARROW-3949
> URL: https://issues.apache.org/jira/browse/ARROW-3949
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Rajeshwar Agrawal
> Priority: Minor
>
> It would be a great to have examples of using parquet arrow high-level API
> for the following 2 cases
> * Storing nested data types (storing nested data types is touted as major
> merit of parquet, so I think this case should be included as an example).
> Ideally, an example of how to use {{arrow::StructArray}} nested with several
> primities types, list types and other nested types would cover every case of
> nested hierarchy of complex data representations
> * Buffered or Batched writes to parquet file. Parquet is meant to be used
> for large amounts of data. The current examples stores all of the data as in
> arrow data structures, before writing to parquet file, which has a huge
> memory footprint, proportional to the amount of data being stored. An example
> of writing directly to row groups and columns, can nicely demonstrate how to
> store data with smaller memory footprint. The current example creates a
> {{arrow::Table}}, which needs to be filled with {{arrow::Array}}(s) of entire
> data, size of which is bounded by the amount of RAM. Ideally, an example
> which generates some data in several {{arrow::Array}}(s), and then stores
> (appends) them as a new Row Group (or Column Chunk) in a
> {{parquet::arrow::FileWriter}}, using {{NewRowGroup}} and
> {{WriteColumnChunk}} functions, thus demonstrating a lower memory footprint
> for writing a parquet file with huge amounts of data
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
