Hi,
I am dynamically doing union all and adding new column too
val dfresult =
> dfAcStamp.select("Col1","Col1","Col3","Col4","Col5","Col6","col7","col8","col9")
> val schemaL = dfresult.schema
> var dffiltered = sqlContext.createDataFrame(sc.emptyRDD[Row], schemaL)
> for ((key,values) <- lcrMap) {
> if(values(4) != null){
> println("Condition============="+values(4))
> val renameRepId = values(0)+"REP_ID"
> dffiltered.printSchema
> dfresult.printSchema
> dffiltered =
> dffiltered.unionAll(dfresult.withColumn(renameRepId,lit(values(3))).drop("Col9").select("Col1","Col1","Col3","Col4","Col5","Col6","Col7","Col8","Col9").where(values(4))).distinct()
> }
> }
when I am printing the schema
dfresult
root
|-- Col1: date (nullable = true)
|-- Col2: date (nullable = true)
|-- Col3: string (nullable = false)
|-- Col4: string (nullable = false)
|-- Col5: string (nullable = false)
|-- Col6: string (nullable = true)
|-- Col7: string (nullable = true)
|-- Col8: string (nullable = true)
|-- Col9: null (nullable = true)
dffiltered Schema
root
|-- Col1: date (nullable = true)
|-- Col2: date (nullable = true)
|-- Col3: string (nullable = false)
|-- Col4: string (nullable = false)
|-- Col5: string (nullable = false)
|-- Col6: string (nullable = true)
|-- Col7: string (nullable = true)
|-- Col8: string (nullable = true)
|-- Col9: null (nullable = true)
As It is priting the same schema but when I am doing UnionAll its giving me
below error
org.apache.spark.sql.AnalysisException: Union can only be performed on
tables with the same number of columns, but the left table has 9 columns
and the right has 8;
Could somebody help me in pointing out my mistake .
Thanks,
