I'm attempting to perform a map on a Dataset[Row] but getting an error on decode when attempting to pass a custom encoder. My code looks similar to the following:
val source = spark.read.format("parquet").load("/emrdata/sources/very_large_ds") source.map{ row => { val key = row(0) } }