Christian Homberg created SPARK-27966:
-----------------------------------------
Summary: input_file_name empty when listing files in parallel
Key: SPARK-27966
URL: https://issues.apache.org/jira/browse/SPARK-27966
Project: Spark
Issue Type: Bug
Components: Input/Output
Affects Versions: 2.4.0
Environment: Databricks 5.3 (includes Apache Spark 2.4.0, Scala 2.11)
Worker Type: 14.0 GB Memory, 4 Cores, 0.75 DBU Standard_DS3_v2
Workers: 3
Driver Type: 14.0 GB Memory, 4 Cores, 0.75 DBU Standard_DS3_v2
Reporter: Christian Homberg
I ran into an issue similar and probably related to SPARK-26128. The
`org.apache.spark.sql.functions.input_file_name` is sometime empty.
My environment is databricks and debugging Log4j output showed me that the
issue occurred when the files are being listed in parallel, e.g. when
19/06/06 11:50:47 INFO InMemoryFileIndex: Start listing leaf files and
directories. Size of Paths: 127; threshold: 32
This is not an issue when listing less than 32 files. Alternatively setting
spark.sql.sources.parallelPartitionDiscovery.threshold to 9999 resolves the
issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
