[
https://issues.apache.org/jira/browse/SPARK-47957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christos Karras updated SPARK-47957:
------------------------------------
Priority: Minor (was: Major)
> pyspark.pandas.read_excel can't get _metadata because it causes a "max
> iterations reached for batch Resolution" error
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-47957
> URL: https://issues.apache.org/jira/browse/SPARK-47957
> Project: Spark
> Issue Type: Bug
> Components: Pandas API on Spark
> Affects Versions: 3.5.1
> Reporter: Christos Karras
> Priority: Minor
>
> I'm trying to add _metadata.file_path to a Spark DataFrame that was read from
> Excel files using ps.read_excel, but it causes this error: "Max iterations
> (100) reached for batch Resolution, please set
> 'spark.sql.analyzer.maxIterations' to a larger value". Increasing
> spark.sql.analyzer.maxIterations to larger values does not resolve the error,
> it just increases the execution time more and more as I try larger values.
>
> Excel files are fairly simple (1 sheet, 8 columns, 2000 rows) and there are
> only a few files (5)
>
> Sample code to reproduce:
> ```python
> import pyspark.pandas as ps
> from pyspark.sql import DataFrame
> from pyspark.sql.functions import col, lit
>
> adls_full_path: str =
> f"abfss://[email protected]/path/2024-04-01/filenamewithwildcards*.xlsx"
>
> input_df: DataFrame = (
> ps
> .read_excel(adls_full_path)
> .to_spark()
> .withColumn("metadata_file_path", col("_metadata.file_path"))
> )
> ```
>
> This code will raise the following error on .withColumn("metadata_file_path",
> col("_metadata.file_path")):
> ``
> Py4JJavaError: An error occurred while calling o1835.withColumn. :
> java.lang.RuntimeException: Max iterations (100) reached for batch
> Resolution, please set 'spark.sql.analyzer.maxIterations' to a larger value.
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:352)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at
> com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94) at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.executeBatch$1(RuleExecutor.scala:289)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$9(RuleExecutor.scala:382)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$9$adapted(RuleExecutor.scala:382)
> at scala.collection.immutable.List.foreach(List.scala:431) at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:382)
> at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:256)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeSameContext(Analyzer.scala:415)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:408)
> at
> org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:322)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:408)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:341)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:248)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:166)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:248)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:393)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:407)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:392)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:244)
> at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:394)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$4(QueryExecution.scala:573)
> at
> org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:1079)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:573)
> at
> com.databricks.util.LexicalThreadLocal$Handle.runWith(LexicalThreadLocal.scala:63)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:569)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1175) at
> org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:569)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:238)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:237)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:219)
> at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:102) at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1175) at
> org.apache.spark.sql.SparkSession.$anonfun$withActiveAndFrameProfiler$1(SparkSession.scala:1182)
> at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
> at
> org.apache.spark.sql.SparkSession.withActiveAndFrameProfiler(SparkSession.scala:1182)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:100) at
> org.apache.spark.sql.Dataset.$anonfun$org$apache$spark$sql$Dataset$$withPlan$1(Dataset.scala:4739)
> at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
> at
> org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:4739)
> at org.apache.spark.sql.Dataset.select(Dataset.scala:1647) at
> org.apache.spark.sql.Dataset.$anonfun$withColumns$1(Dataset.scala:2922) at
> com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94) at
> org.apache.spark.sql.Dataset.withColumns(Dataset.scala:2896) at
> org.apache.spark.sql.Dataset.withColumn(Dataset.scala:2860) at
> sun.reflect.GeneratedMethodAccessor635.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at
> py4j.Gateway.invoke(Gateway.java:306) at
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at
> py4j.commands.CallCommand.execute(CallCommand.java:79) at
> py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199)
> at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at
> java.lang.Thread.run(Thread.java:750)
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
