[
https://issues.apache.org/jira/browse/IMPALA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253428#comment-17253428
]
Zoltán Borók-Nagy commented on IMPALA-2753:
-------------------------------------------
Yeah, it does have relevance to Iceberg. Currently we name the files like
Iceberg's default location provider does:
[https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/LocationProviders.java#L71-L89]
So currently the paths are similar to the paths we use for regular filesystem
tables, except we have the additional 'data' directory under the table
directory. But other than that partition directories and file names are the
same.
Iceberg also has an object store location provider:
[https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/LocationProviders.java#L91-L131]
It basically adds a random prefix (which is the hash of the partition path +
filename) to the final file path:
[https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/LocationProviders.java#L110-L111]
We could do the same if table property "write.object-storage.path" is set.
Created IMPALA-10403 for tracking this task.
> Investigate performance gains for adding random prefix to file name
> -------------------------------------------------------------------
>
> Key: IMPALA-2753
> URL: https://issues.apache.org/jira/browse/IMPALA-2753
> Project: IMPALA
> Issue Type: Sub-task
> Components: Perf Investigation
> Affects Versions: Impala 2.5.0
> Reporter: Mostafa Mokhtar
> Priority: Minor
> Labels: s3
>
> I noticed which is not directly related to Impala is that the file naming
> convention HDFS produces is the anti pattern of what S3 recommends.
> If we do a trick with the naming convention we can one up Hive when running
> on S3.
> {code}
> examplebucket/2013-26-05-15-00-00/cust1234234/photo1.jpg
> examplebucket/2013-26-05-15-00-00/cust3857422/photo2.jpg
> examplebucket/2013-26-05-15-00-00/cust8474937/photo2.jpg
> examplebucket/2013-26-05-15-00-00/cust1248473/photo3.jpg
> ...
> examplebucket/2013-26-05-15-00-01/cust1248473/photo4.jpg
> examplebucket/2013-26-05-15-00-01/cust1248473/photo5.jpg
> examplebucket/2013-26-05-15-00-01/cust1248473/photo6.jpg
> examplebucket/2013-26-05-15-00-01/cust1248473/photo7.jpg
> ...
> {code}
> The sequence pattern in the key names introduces a performance problem. To
> understand the issue, let’s look at how Amazon S3 stores key names.
> Amazon S3 maintains an index of object key names in each AWS region. Object
> keys are stored lexicographically across multiple partitions in the index.
> That is, Amazon S3 stores key names in alphabetical order. The key name
> dictates which partition the key is stored in. Using a sequential prefix,
> such as timestamp or an alphabetical sequence, increases the likelihood that
> Amazon S3 will target a specific partition for a large number of your keys,
> overwhelming the I/O capacity of the partition. If you introduce some
> randomness in your key name prefixes, the key names, and therefore the I/O
> load, will be distributed across more than one partition.
> If you anticipate that your workload will consistently exceed 100 requests
> per second, you should avoid sequential key names. If you must use sequential
> numbers or date and time patterns in key names, add a random prefix to the
> key name. The randomness of the prefix more evenly distributes key names
> across multiple index partitions. Examples of introducing randomness are
> provided later in this topic.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
