[
https://issues.apache.org/jira/browse/SPARK-34339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-34339:
------------------------------------
Assignee: (was: Apache Spark)
> Expose the number of truncated paths in Utils.buildLocationMetadata()
> ---------------------------------------------------------------------
>
> Key: SPARK-34339
> URL: https://issues.apache.org/jira/browse/SPARK-34339
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> SPARK-31793 introduces Utils.buildLocationMetadata() to reduce the length of
> location metadata. It effectively reduced the memory usage as only a few
> paths will be included (and it's controlled by threshold value), but there's
> no indication of the fact for truncation.
> If the first 2 of 5 paths are only fit to the threshold,
> Utils.buildLocationMetadata() shows the first 2 paths, but there's no mention
> that 3 paths are truncated. Even no mention that some paths have been
> truncated; it shows the same with the output just first 2 paths were
> presented. This could bring confusion.
> We should allow more space (like 10+ chars) to represent the fact N paths are
> truncated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
