[
https://issues.apache.org/jira/browse/TAJO-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258644#comment-14258644
]
ASF GitHub Bot commented on TAJO-1266:
--------------------------------------
GitHub user charsyam opened a pull request:
https://github.com/apache/tajo/pull/320
TAJO-1266: Too many logs when writing a parquet relation
In parquet, They use parquet.Log class
and It's log level is determined in Compile Time In parquet source.
So we can't change it.
Fortunately, the logs that @jihoonson mentioned, They are in tajo.
so I just changed its code to use log.debug.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/charsyam/tajo feature/TAJO-1266
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/320.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #320
----
commit b9a7ff36010eb0a6642133538055b661a8ea6181
Author: clark.kang <[email protected]>
Date: 2014-12-25T04:51:51Z
TAJO-1266
----
> Too many logs when writing a parquet relation
> ---------------------------------------------
>
> Key: TAJO-1266
> URL: https://issues.apache.org/jira/browse/TAJO-1266
> Project: Tajo
> Issue Type: Improvement
> Reporter: Jihoon Son
> Assignee: DaeMyung Kang
>
> See the title.
> The parquet writer emits too many logs by default.
> Here is a part of the whole log.
> {noformat}
> Dec 22, 2014 3:28:59 PM parquet.Log info
> INFO: written 337B for [p_retailprice] DOUBLE: 737,002 values, 79B raw, 79B
> comp, 6 pages, encodings: [PLAIN_DICTIONARY, BIT_PACKED, RLE], dic { 2
> entries, 16B raw, 2B comp}
> Dec 22, 2014 3:28:59 PM parquet.Log info
> INFO: written 790B for [p_comment] BINARY: 737,002 values, 143B raw, 143B
> comp, 11 pages, encodings: [PLAIN_DICTIONARY, BIT_PACKED, RLE], dic { 2
> entries, 34B raw, 2B comp}
> Dec 22, 2014 3:28:59 PM parquet.Log info
> INFO: written 337B for [s_suppkey] INT64: 737,002 values, 79B raw, 79B comp,
> 6 pages, encodings: [PLAIN_DICTIONARY, BIT_PACKED, RLE], dic { 2 entries, 16B
> raw, 2B comp}
> {noformat}
> As you can see, a new log is emitted even when only 337B is additionally
> written.
> This will significantly degrade the write performance, so we need to turn off
> the parquet log by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
