Re: Spark 1.6.2 can read hive tables created with sqoop, but Spark 2.0.0 cannot

The data is uncorrupted as I can create the dataframe from the underlying raw parquet from spark 2.0.0 if instead of using SparkSession.sql() to create a dataframe I use SparkSession.read.parquet(). -- View this message in context:

Re: Spark 1.6.2 can read hive tables created with sqoop, but Spark 2.0.0 cannot

Using the scala api instead of the python api yields the same results. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-2-can-read-hive-tables-created-with-sqoop-but-Spark-2-0-0-cannot-tp27502p27506.html Sent from the Apache Spark User List mailing

Re: Spark 1.6.2 can read hive tables created with sqoop, but Spark 2.0.0 cannot

Hi, Is this table created as external table in Hive? Do you see data through Spark-sql or Hive thrift server. There is an issue with Zeppelin seeing data when connecting to Spark Thrift Server. Rows display null value. HTH Dr Mich Talebzadeh LinkedIn *

Re: Spark 1.6.2 can read hive tables created with sqoop, but Spark 2.0.0 cannot

Can you get all the fields back using Scala or SQL (bin/spark-sql)? On Tue, Aug 9, 2016 at 2:32 PM, cdecleene wrote: > Some details of an example table hive table that spark 2.0 could not read... > > SerDe Library: > org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe

Spark 1.6.2 can read hive tables created with sqoop, but Spark 2.0.0 cannot

Some details of an example table hive table that spark 2.0 could not read... SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat OutputFormat: