Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14635 )
Change subject: IMPALA-8525: preads should use hdfsPreadFully rather than hdfsPread ...................................................................... Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/14635/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14635/2//COMMIT_MSG@9 PS2, Line 9: Modifies HdfsFileReader so that it calls hdfsPreadFully instead of > Is hdfsPreadFully supported for all filesystems? It's supported by all filesystems supported by Impala - HDFS, S3A, ADLS, ABFS as well as Google Cloud Store (GCS). http://gerrit.cloudera.org:8080/#/c/14635/2/be/src/runtime/io/hdfs-file-reader.cc File be/src/runtime/io/hdfs-file-reader.cc: http://gerrit.cloudera.org:8080/#/c/14635/2/be/src/runtime/io/hdfs-file-reader.cc@137 PS2, Line 137: int chunk_size = bytes_to_read - *bytes_read; > Maybe rename to bytes_remaining or similar, given that we're not trying to Done http://gerrit.cloudera.org:8080/#/c/14635/2/be/src/runtime/io/hdfs-file-reader.cc@224 PS2, Line 224: if (FLAGS_use_hdfs_pread || IsS3APath(scan_range_->file_string()->c_str())) { > Should we switch ABFS too? Since we disabled the chunking fix for that as w Oddly enough, none of this makes a significant difference for ABFS. I plan to investigate why in a separate JIRA. I don't think the chunk-size optimization was added for ABFS with much experimentation, so I don't think we lose much, especially with the fix in HDFS-14285. -- To view, visit http://gerrit.cloudera.org:8080/14635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29ea34897096bc790abdeb98073a47f1c4c10feb Gerrit-Change-Number: 14635 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Fri, 08 Nov 2019 16:36:44 +0000 Gerrit-HasComments: Yes
