Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17771 )
Change subject: WiP: IMPALA-10798 : Prototype for JSON reader ...................................................................... Patch Set 24: (49 comments) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc File be/src/exec/hdfs-json-scanner.cc: http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@206 PS24, Line 206: // VLOG_QUERY << "decimal128" << arrow::decimal128(ct.precision, ct.scale)->ToString(); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@237 PS24, Line 237: // VLOG_QUERY << "decimal128" << arrow::decimal128(ct.precision, ct.scale)->ToString(); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@279 PS24, Line 279: VLOG_QUERY << " PrintTuple:::" << PrintTuple(template_tuple_, *scan_node_->tuple_desc()); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@303 PS24, Line 303: std::shared_ptr<arrow::Field> field_a = arrow::field(cvf.name(), ColumnType2ArrowType(cvf.type())); line too long (103 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@321 PS24, Line 321: VLOG_QUERY<< "Filename:::"<< stream_->filename()<< " "<< "num_rows_openfunction():::" << num_rows_; line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@348 PS24, Line 348: VLOG_QUERY << " PrintTemplateTuple:::" << PrintTuple(template_tuple, *tuple_desc) << std::endl; line too long (103 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@350 PS24, Line 350: VLOG_QUERY << " PrintTemplateTuple:::" << PrintTuple(template_tuple, *tuple_desc) << std::endl; line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@371 PS24, Line 371: //if (!EvalRuntimeFilters(reinterpret_cast<TupleRow*>(row)) || !ExecNode::EvalConjuncts(evals.data(), evals.size(), row)) { line too long (127 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@384 PS24, Line 384: VLOG_QUERY<< "Column in GetNextInternal" << column->type()->ToString() << column->ToString(); line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-json-scanner.cc@499 PS24, Line 499: //if (EvalRuntimeFilters(reinterpret_cast<TupleRow*>(row)) && ExecNode::EvalConjuncts(evals.data(), evals.size(), row)) { line too long (125 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc File be/src/exec/hdfs-scan-node-base.cc: http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@70 PS24, Line 70: "for all reads, regardless of whether the read is local or remote. By default, the " line too long (126 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@71 PS24, Line 71: "IO data cache is only used if the data is expected to be remote. Used by tests."); line too long (125 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@79 PS24, Line 79: " across all Disk I/O threads in HDFS read operations."); line too long (112 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@81 PS24, Line 81: " spent across all Disk I/O threads in HDFS open operations."); line too long (122 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@84 PS24, Line 84: " while it is executing I/O operations on behalf of a scan."); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@91 PS24, Line 91: "disks accessed by HDFS scan. Each local disk is counted as a disk and each type of" line too long (150 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@92 PS24, Line 92: " remote filesystem (e.g. HDFS remote reads, S3) is counted as a distinct disk."); line too long (148 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@94 PS24, Line 94: " average number of HDFS read threads executing read operations on behalf of this " line too long (161 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@95 PS24, Line 95: "scan. Higher values (i.e. close to the aggregate number of I/O threads across " line too long (158 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@96 PS24, Line 96: "all disks accessed) show that this scan is using a larger proportion of the I/O " line too long (160 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@97 PS24, Line 97: "capacity of the system. Lower values show that either this scan is not I/O bound" line too long (160 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@98 PS24, Line 98: " or that it is getting a small share of the I/O capacity of the system."); line too long (153 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@106 PS24, Line 106: "Use this to determine if the scan got all of the reservation it wanted. Does not " line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@107 PS24, Line 107: "include subsequent reservation increases done by scanner implementation " line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@129 PS24, Line 129: "threads spent waiting for I/O. This value can be compared to the value of " line too long (128 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@130 PS24, Line 130: "ScannerThreadsTotalWallClockTime of MT_DOP = 0 scan nodes or otherwise compared " line too long (134 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@131 PS24, Line 131: "to the total time reported for MT_DOP > 0 scan nodes. High values show that " line too long (130 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@132 PS24, Line 132: "scanner threads are spending significant time waiting for I/O instead of " line too long (127 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@133 PS24, Line 133: "processing data. Note that this includes the time when the thread is runnable " line too long (132 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@137 PS24, Line 137: "Each sample in the counter is the size of a single column that is scanned by the " line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@141 PS24, Line 141: "Each sample in the counter is the size of a single column that is scanned by the " line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@353 PS24, Line 353: VLOG_QUERY<< hdfs_table_->partition_descriptors().size() << " " << hdfs_table_->DebugString() << " " << shared_state_.use_mt_scan_node_ << " " << instance_ctx_pbs.size() << " " << tnode_->hdfs_scan_node << GetStackTrace(); line too long (233 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@751 PS24, Line 751: metadata->partition_id, FilterStats::FILES_KEY, filter_ctxs, file, state)) { line too long (93 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@1236 PS24, Line 1236: "Read $0 of data across network that was expected to be local. Block locality " line too long (144 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@1237 PS24, Line 1237: "metadata for table '$1.$2' may be stale. This only affects query performance " line too long (144 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@1238 PS24, Line 1238: "and not result correctness. One of the common causes for this warning is HDFS " line too long (145 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@1239 PS24, Line 1239: "rebalancer moving some of the file's blocks. If the issue persists, consider " line too long (144 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@1240 PS24, Line 1240: "running \"INVALIDATE METADATA `$1`.`$2`\".", line too long (110 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@1241 PS24, Line 1241: PrettyPrinter::Print(unexpected_remote_bytes_->value(), TUnit::BYTES), line too long (135 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/be/src/exec/hdfs-scan-node-base.cc@1242 PS24, Line 1242: hdfs_table_->database(), hdfs_table_->name()))); line too long (113 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/17771/24/bin/bootstrap_toolchain.py@489 PS24, Line 489: " flake8: E501 line too long (98 > 90 characters) http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@458 PS24, Line 458: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1420 PS24, Line 1420: 0, fileDesc.getFileLength(), partition.getId(), fileDesc.getFileLength(), line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1421 PS24, Line 1421: fileDesc.getFileCompression().toThrift(), fileDesc.getModificationTime(), line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1425 PS24, Line 1425: currentOffset, currentLength, partition.getId(), fileDesc.getFileLength(), line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1426 PS24, Line 1426: fileDesc.getFileCompression().toThrift(), fileDesc.getModificationTime(), line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1443 PS24, Line 1443: LOG.info(String.format("scanRangeLocations: %s, LargestScanRangeBytes: %d, FileMaxScanRangeBytes = %s, " + line too long (114 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1444 PS24, Line 1444: "RemainingLength = %d , CurrentOffset = %d, currentLength = %d", scanRangeLocations, largestScanRangeBytes_, line too long (132 > 90) http://gerrit.cloudera.org:8080/#/c/17771/24/tests/query_test/test_tpch_queries.py File tests/query_test/test_tpch_queries.py: http://gerrit.cloudera.org:8080/#/c/17771/24/tests/query_test/test_tpch_queries.py@39 PS24, Line 39: s flake8: E501 line too long (96 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/17771 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If79364a421d862d0d837f9be694911e388d4d629 Gerrit-Change-Number: 17771 Gerrit-PatchSet: 24 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Comment-Date: Tue, 21 Mar 2023 09:04:09 +0000 Gerrit-HasComments: Yes
