[
https://issues.apache.org/jira/browse/IMPALA-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236324#comment-17236324
]
Tim Armstrong commented on IMPALA-9437:
---------------------------------------
[~adoroszlai][~arp] here's the relevant code:
https://github.com/apache/impala/blob/981ef104654e187ae18b25f31df2fd324e687643/be/src/exec/hdfs-table-sink.cc#L402
Here's the logic for parquet default_block_size:
https://github.com/apache/impala/blob/1bd27a3ea87622edc869d3ff72ce9b0881451c95/be/src/exec/parquet/hdfs-parquet-table-writer.cc#L1211
And for text:
https://github.com/apache/impala/blob/8b8a49e617818e9bcf99b784b63587c95cebd622/be/src/exec/hdfs-text-table-writer.cc#L63
And HDFS_BLOCK_SIZE:
https://github.com/apache/impala/blob/981ef104654e187ae18b25f31df2fd324e687643/be/src/exec/parquet/hdfs-parquet-table-writer.h#L94,
which is 256MB, which is the default ozone block size.
The block size is used for sizing parquet files, so that row groups line up to
block boundaries. I think it's a little weird if we get the wrong size, e.g.
the ozone block size is 128MB and the parquet writer thinks it's 256mb - you
get a row group split across blocks, which will make parallelism suboptimal.
I think if we replaced the reference to HDFS_BLOCK_SIZE with a call to a
function that returned the actual configured default for the path being written
to, that would be a generally good improvement.
It does depend a bit on whether people use non-default Ozone block sizes with
any frequency.
> Cannot get Ozone file block size after opening the file
> -------------------------------------------------------
>
> Key: IMPALA-9437
> URL: https://issues.apache.org/jira/browse/IMPALA-9437
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Sahil Takiar
> Priority: Major
>
> When creating a tmp file on HDFS, {{HdfsTableSink::CreateNewTmpFile}} first
> opens the file, and then stats the file ({{hdfsGetPathInfo}}) before actually
> writing any data to the file or closing the file. HDFS seems to allow this
> behavior. However, Ozone, S3A, and ABFS do not. Impala does this for HDFS in
> order to get the block size of the opened file. According to
> {{HdfsTableSink}} it is possible for HDFS to create a block size with a
> different one than requested by Impala. So in order to track the correct
> block size for a file, the file needs to be stat'ed after opening it. For S3A
> and ABFS this isn't a big deal, because they aren't block based filesystem,
> but Ozone is. So we should investigate the impact of not having this
> capability and consider adding it to the Ozone client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
