Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 )
Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet ...................................................................... Patch Set 4: (6 comments) This is still working progress, the min max stats have similar issues as IMPALA-7559. The way millisec and microsec columns are distinguished should be also improved. http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG@19 PS4, Line 19: TODOs: > Maybe test dictionary filtering and runtime filtering for the new types too Dictionary filtering is disabled for types that need conversion or validation (see HdfsParquetScanner::IsDictFilterable()). All timestamps need validation (probably INT64_NANO won't need it in the future), so dict filtering is always disabled for them. It shouldn't be very hard to implement correct dictionary filtering, but it is probably not in the scope of this change. Doing the timezone conversion during dictionary scanning would be also a good way to speed up low NDV queries. + Checking that all timestamps in the dictionary are valid could enable the column readers to skip validation when reading dictionary encoded data pages. I didn't look at runtime filtering yet, but I guess that it does not need special handling. I will check it before the next patch. http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG@22 PS4, Line 22: - CREATE TABLE LIKE PARQUET should be also tested with the new > Oh i guess you already plan to address my one comment. I will deal with this in the next commit. Changing the tests to use the functions in https://gerrit.cloudera.org/#/c/11127/ will exercise CREATE TABLE LIKE. http://gerrit.cloudera.org:8080/#/c/11057/4//COMMIT_MSG@24 PS4, Line 24: > We should also do some kind of perf sanity test. E.g. a large table with th I did tests locally and mentioned the results in the Jira. If the changes in https://gerrit.cloudera.org/#/c/11183/ are merged, then the result is the same as with INT96. http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc File be/src/exec/parquet-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc@408 PS4, Line 408: if (slot_desc_->type().type != TYPE_TIMESTAMP) return; > To me it feels a bit weird to use an early return statement here. They are Done http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc@1686 PS4, Line 1686: return parent->obj_pool_.Add(reader); > I feel like it's error-prone that we're allocating the Reader in many place I tried to rewrite it to use unique_ptr, but the result was not very nice. I will think a bit more on how to do this better. http://gerrit.cloudera.org:8080/#/c/11057/4/be/src/exec/parquet-column-readers.cc@1696 PS4, Line 1696: if (node.element->type != parquet::Type::INT64) { > Do we guarantee this invariant somewhere else? Would be good to note where. Parquet format only allows INT64 and INT96 timestamps at the moment. This is also enforced by ParquetMetadataUtils::ValidateColumn -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Thu, 13 Sep 2018 19:24:09 +0000 Gerrit-HasComments: Yes
