Jungtaek Lim created SPARK-34339:
------------------------------------
Summary: Expose the number of truncated paths in
Utils.buildLocationMetadata()
Key: SPARK-34339
URL: https://issues.apache.org/jira/browse/SPARK-34339
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0
Reporter: Jungtaek Lim
SPARK-31793 introduces Utils.buildLocationMetadata() to reduce the length of
location metadata. It effectively reduced the memory usage as only a few paths
will be included (and it's controlled by threshold value), but there's no
indication of the fact for truncation.
If the first 2 of 5 paths are only fit to the threshold,
Utils.buildLocationMetadata() shows the first 2 paths, but there's no mention
that 3 paths are truncated. Even no mention that some paths have been
truncated; it shows the same with the output just first 2 paths were presented.
This could bring confusion.
We should allow more space (like 10+ chars) to represent the fact N paths are
truncated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
