Oliver Caballero Alvarez created SPARK-48571:
------------------------------------------------
Summary: Reduce the number of accesses to S3 object storage
Key: SPARK-48571
URL: https://issues.apache.org/jira/browse/SPARK-48571
Project: Spark
Issue Type: Task
Components: Spark Core
Affects Versions: 3.5.0
Reporter: Oliver Caballero Alvarez
Attachments: Spark 3.2 Hadoop-aws 3.1.PNG, Spark 3.2 Hadoop-aws
3.4.PNG, Spark 3.5 Hadoop-aws 3.1.PNG
If we access a Spark table on an object storage file system with parquet files,
the object storage suffers many requests that seem to be unnecessary. To
explain this I will do it with an example:
I have created a simple table, with 3 files:
*business/t_filter/country=ES/data_date_part=2023-09-27/part-00000-0f52aae9-2db8-415e-93f3-8331539c0ead.c000*
*business/t_filter/country=ES/data_date_part=2023-06-01/part-00000-0f52aae9-2db8-415e-93f3-8331539c0ead.c000*
*business/t_filter/country=ES/data_date_part=2023-09-27/part-00000-f10096c1-53bc-4e2f-bc56-eba65acfa44a.c000*
and I have put a table that represents business/t_filter with country and
data_date_part partitions, you have the following requests.
If you use versions prior to Spark 3.5 or Hadoop 3.4, in my case it is exactly
Spark 3.2.3 and Hadoop 3.1, the number of requests you have are the following
-> IMAGE 1 Spark 3.2.3 Hadoop 3.1
In this image we can see all the requests where we can find the following
errors:
* Two HEAD and two LIST are made with the implementation of S3, of the folders
where the files are located, which could only be resolved with a single list.
This bug has already been resolved in ->
https://issues.apache.org/jira/browse/HADOOP-18073 -> Result : IMAGE 2 Spark
3.2.3 Hadoop 3.4
* For each file, the parquet footing is listed twice. This bug is resolved in
-> https://issues.apache.org/jira/browse/SPARK-42388 -> Result : IMAGESpark
3.5.0 Hadoop 3.1
* A Head Object is launched twice each time a file is read, this could be
reduced by implementing the FileSystem interface so that it could receive the
FileStatus that has already been calculated above.
** https://issues.apache.org/jira/browse/HADOOP-19199
** https://issues.apache.org/jira/browse/PARQUET-2493
** https://issues.apache.org/jira/browse/HADOOP-19200
* The requests could be reduced when reading the parquet footer, since first
you have to read the size of the schema and then the schema, which implies two
HTTP/HTTPS requests to S3. It would be nice if there was a minimum threshold,
for example 100KB, in which, if the file is smaller than that, it would not
have to make two requests, and the entire file would be brought, since bringing
100 KB will take less time in one request to bring 8 B in a request and then
another request for x KB. Even so, I don't know if this task makes sense.
It would be to change this implementation, with an environment variable, that
if it is set to -1 it does the same, but if it has a threshold set, up to that
threshold you do not have to call the seek function twice, which repeats a GET
Object.
**
[https://github.com/apache/parquet-java/blob/apache-parquet-1.14.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java]
With all these improvements, updating to the latest version of Spark and Hadoop
would go from more than 30 requests to 11 in the proposed example.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
