Support nested keys in DataFrameWriter.bucketBy

[Dávid Szakállas]

Currently (In Spark 2.3.1) we cannot bucket DataFrames by nested columns, e.g 

df.write.bucketBy(10, "key.a").saveAsTable(“junk”)

will result in the following exception:

org.apache.spark.sql.AnalysisException: bucket column key.a is not defined in 
table junk, defined table columns are: key, value;
        at 
org.apache.spark.sql.catalyst.catalog.CatalogUtils$$anonfun$org$apache$spark$sql$catalyst$catalog$CatalogUtils$$normalizeColumnName$2.apply(ExternalCatalogUtils.scala:246)
        at 
org.apache.spark.sql.catalyst.catalog.CatalogUtils$$anonfun$org$apache$spark$sql$catalyst$catalog$CatalogUtils$$normalizeColumnName$2.apply(ExternalCatalogUtils.scala:246)
        at scala.Option.getOrElse(Option.scala:121)
        …

Are there plans to change this anytime soon?

Thanks, David





---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org