Takuya Ueshin created SPARK-42458: ------------------------------------- Summary: createDataFrame should support DDL string as schema Key: SPARK-42458 URL: https://issues.apache.org/jira/browse/SPARK-42458 Project: Spark Issue Type: Sub-task Components: Connect Affects Versions: 3.4.0 Reporter: Takuya Ueshin
{code:python} File "/.../python/pyspark/sql/connect/readwriter.py", line 393, in pyspark.sql.connect.readwriter.DataFrameWriter.option Failed example: with tempfile.TemporaryDirectory() as d: # Write a DataFrame into a CSV file with 'nullValue' option set to 'Hyukjin Kwon'. df = spark.createDataFrame([(100, None)], "age INT, name STRING") df.write.option("nullValue", "Hyukjin Kwon").mode("overwrite").format("csv").save(d) # Read the CSV file as a DataFrame. spark.read.schema(df.schema).format('csv').load(d).show() Exception raised: Traceback (most recent call last): File "/.../lib/python3.9/doctest.py", line 1334, in __run exec(compile(example.source, filename, "single", File "<doctest pyspark.sql.connect.readwriter.DataFrameWriter.option[2]>", line 3, in <module> df = spark.createDataFrame([(100, None)], "age INT, name STRING") File "/.../python/pyspark/sql/connect/session.py", line 312, in createDataFrame raise ValueError( ValueError: Some of types cannot be determined after inferring, a StructType Schema is required in this case {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org