Sahil Takiar created IMPALA-9437:
------------------------------------
Summary: Cannot get Ozone file block size after opening the file
Key: IMPALA-9437
URL: https://issues.apache.org/jira/browse/IMPALA-9437
Project: IMPALA
Issue Type: Sub-task
Reporter: Sahil Takiar
When creating a tmp file on HDFS, {{HdfsTableSink::CreateNewTmpFile}} first
opens the file, and then stats the file ({{hdfsGetPathInfo}}) before actually
writing any data to the file or closing the file. HDFS seems to allow this
behavior. However, Ozone, S3A, and ABFS do not. Impala does this for HDFS in
order to get the block size of the opened file. According to {{HdfsTableSink}}
it is possible for HDFS to create a block size with a different one than
requested by Impala. So in order to track the correct block size for a file,
the file needs to be stat'ed after opening it. For S3A and ABFS this isn't a
big deal, because they aren't block based filesystem, but Ozone is. So we
should investigate the impact of not having this capability and consider adding
it to the Ozone client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
