I am reading some data in a dataframe from a dynamo db table:
val data = spark.read.dynamodb("table")
data.filter($"field1".like("%hello%")).createOrReplaceTempView("temp")
spark.sql("select * from temp").show()
When I do the last statement I get results. If however I try to do:
spark.sql("select field2 from temp").show()
I get no results. The dataframe has the structure:
root
|-- field1: string (nullable = true)
|-- field2: string (nullable = true)
|-- field3: string (nullable = true)
|-- field4: long (nullable = true)
|-- field5: string (nullable = true)
Dependencies:
spark 2.2.0
scala 2.11.8
spark-dynamodb 0.0.11
Spark running on local[*]
