Kevin Mader created SPARK-18178: ----------------------------------- Summary: Importing Pandas Tables with Missing Values Key: SPARK-18178 URL: https://issues.apache.org/jira/browse/SPARK-18178 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 2.0.0 Reporter: Kevin Mader
If you import a table with missing values (like below) and create a dataframe from it, everything works fine until the command is actually execute (.first(), or .toPandas(), etc). The problem came up with a much larger table with values that were not NAN, just empty. ``` import pandas as pd from io import StringIO test_df = pd.read_csv(StringIO(',Scan Options\n15,SAT2\n16,\n')) sqlContext.createDataFrame(test_df).registerTempTable('Test') o_qry = sqlContext.sql("SELECT * FROM Test LIMIT 1") o_qry.first() ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org