[
https://issues.apache.org/jira/browse/SPARK-34339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278452#comment-17278452
]
Apache Spark commented on SPARK-34339:
--------------------------------------
User 'HeartSaVioR' has created a pull request for this issue:
https://github.com/apache/spark/pull/31464
> Expose the number of truncated paths in Utils.buildLocationMetadata()
> ---------------------------------------------------------------------
>
> Key: SPARK-34339
> URL: https://issues.apache.org/jira/browse/SPARK-34339
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> SPARK-31793 introduces Utils.buildLocationMetadata() to reduce the length of
> location metadata. It effectively reduced the memory usage as only a few
> paths will be included (and it's controlled by threshold value), but there's
> no indication of the fact for truncation.
> If the first 2 of 5 paths are only fit to the threshold,
> Utils.buildLocationMetadata() shows the first 2 paths, but there's no mention
> that 3 paths are truncated. Even no mention that some paths have been
> truncated; it shows the same with the output just first 2 paths were
> presented. This could bring confusion.
> We should allow more space (like 10+ chars) to represent the fact N paths are
> truncated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
