Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/11057 )

Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and 
TIMESTAMP_MICROS from Parquet
......................................................................


Patch Set 4:

(6 comments)

This is still working progress, the min max stats have similar issues as 
IMPALA-7559. The way millisec and microsec columns are distinguished should be 
also improved.

http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG@19
PS4, Line 19: TODOs:
> Maybe test dictionary filtering and runtime filtering for the new types too
Dictionary filtering is disabled for types that need conversion or validation 
(see HdfsParquetScanner::IsDictFilterable()). All timestamps need validation 
(probably INT64_NANO won't need it in the future), so dict filtering is always 
disabled for them.

It shouldn't be very hard to implement correct dictionary filtering, but it is 
probably not in the scope of this change.
Doing the timezone conversion during dictionary scanning would be also a good 
way to speed up low NDV queries. + Checking that all timestamps in the 
dictionary are valid could enable the column readers to skip validation when 
reading dictionary encoded data pages.

I didn't look at runtime filtering yet, but I guess that it does not need 
special handling. I will check it before the next patch.


http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG@22
PS4, Line 22: - CREATE TABLE LIKE PARQUET should be also tested with the new
> Oh i guess you already plan to address my one comment.
I will deal with this in the next commit. Changing the tests to use the 
functions in https://gerrit.cloudera.org/#/c/11127/ will exercise CREATE TABLE 
LIKE.


http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG@24
PS4, Line 24:
> We should also do some kind of perf sanity test. E.g. a large table with th
I did tests locally and mentioned the results in the Jira. If the changes in 
https://gerrit.cloudera.org/#/c/11183/ are merged, then the result is the same 
as with INT96.


http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc@408
PS4, Line 408:   if (slot_desc_->type().type != TYPE_TIMESTAMP) return;
> To me it feels a bit weird to use an early return statement here. They are
Done


http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc@1686
PS4, Line 1686:   return parent->obj_pool_.Add(reader);
> I feel like it's error-prone that we're allocating the Reader in many place
I tried to rewrite it to use unique_ptr, but the result was not very nice. I 
will think a bit more on how to do this better.


http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc@1696
PS4, Line 1696:   if (node.element->type != parquet::Type::INT64) {
> Do we guarantee this invariant somewhere else? Would be good to note where.
Parquet format only allows INT64 and INT96 timestamps at the moment. This is 
also enforced by ParquetMetadataUtils::ValidateColumn



--
To view, visit http://gerrit.cloudera.org:8080/11057
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3
Gerrit-Change-Number: 11057
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
Gerrit-Comment-Date: Thu, 13 Sep 2018 19:24:09 +0000
Gerrit-HasComments: Yes

Reply via email to