[
https://issues.apache.org/jira/browse/TAJO-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948765#comment-13948765
]
Hyunsik Choi commented on TAJO-30:
----------------------------------
Big +1 for the latest patch.
I have a trivial question. As you can see the comments in TajoParquetWriter,
the default block is stated as 50 MB. I'm expecting that you may intend 128MB.
Would you mind checking it?
If we need to modify the comment, I'll commit your latest patch + the comment
fix without letting you resubmit a updated patch. Because this is is very
trivial, this way would be more productive.
{noformat}
/**
* Creates a new TajoParquetWriter. The default block size is 50 MB.
* The default page size is 1 MB. Default compression is no compression.
*
* @param file The Path of the file to write to.
* @param schema The Tajo schema of the table.
* @throws IOException
*/
public TajoParquetWriter(Path file, Schema schema) throws IOException {
{noformat}
FYI, I share how to create parquet tables in tajo shell. You may already know
the way.
{code}
tpch> create table orders_parquet5 using parquet as select * from orders;
Progress: 0%, response time: 0.405 sec
Progress: 0%, response time: 1.207 sec
Progress: 33%, response time: 2.209 sec
Progress: 83%, response time: 3.211 sec
Progress: 100%, response time: 3.471 sec
final state: QUERY_SUCCEEDED, response time: 3.471 sec
OK
tpch> select count(*) from orders_parquet5;
Progress: 8%, response time: 0.396 sec
Progress: 100%, response time: 0.923 sec
2014-03-27 11:02:49,276 WARN storage.AbstractStorageManager
(AbstractStorageManager.java:<init>(86)) - does not support block metadata.
('dfs.datanode.hdfs-blocks-metadata.enabled')
final state: QUERY_SUCCEEDED, response time: 0.923 sec
result:
hdfs://127.0.0.1:8020/tmp/tajo-hyunsik/staging/q_1395884147706_0005/RESULT, 1
rows (8 B)
?count
-------------------------------
1500000
tpch>
{code}
> Parquet Integration
> -------------------
>
> Key: TAJO-30
> URL: https://issues.apache.org/jira/browse/TAJO-30
> Project: Tajo
> Issue Type: New Feature
> Reporter: Hyunsik Choi
> Assignee: David Chen
> Labels: Parquet
> Attachments: TAJO-30.patch, TAJO-30_20140326_05:06:57.patch,
> TAJO-30_20140326_05:34:17.patch, null_handling.patch
>
>
> Parquet is a columnar storage format developed by Twitter. Implement Parquet
> (http://parquet.io/) support for Tajo.
> The implementation consists of the following:
> * {{ParquetScanner}} and {{ParquetAppender}} - FileScanner and FileAppenders
> for reading and writing Parquet.
> * {{TajoParquetReader}} and {{TajoParquetWriter}} - Top-level reader and
> writer for serializing/deserializing to Tajo Tuples.
> * {{TajoReadSupport}} and {{TajoWriteSupport}} - Abstractions to perform
> conversion between Parquet and Tajo records.
> * {{TajoRecordMaterializer}} - Materializes Tajo Tuples from Parquet's
> internal representation.
> * {{TajoRecordConverter}} - Used by {{TajoRecordMateriailzer}} to
> materialize a Tajo Tuple.
> * {{TajoSchemaConverter}} - Converts between Tajo and Parquet schemas.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
