Yea - I'd just add a bunch of columns. Doesn't seem like that big of a deal.
On Wed, Jul 15, 2015 at 10:53 AM, RJ Nowling <rnowl...@gmail.com> wrote: > I'm considering a few approaches -- one of which is to provide new > functions like mapLeft, mapRight, filterLeft, etc. > > But this all falls shorts with DataFrames. RDDs can easily be extended > from RDD[T] to RDD[Record[T]]. I guess with DataFrames, I could add > special columns? > > On Wed, Jul 15, 2015 at 12:36 PM, Reynold Xin <r...@databricks.com> wrote: > >> How about just using two fields, one boolean field to mark good/bad, and >> another to get the source file? >> >> >> On Wed, Jul 15, 2015 at 10:31 AM, RJ Nowling <rnowl...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'm working on an ETL task with Spark. As part of this work, I'd like >>> to mark records with some info such as: >>> >>> 1. Whether the record is good or bad (e.g, Either) >>> 2. Originating file and lines >>> >>> Part of my motivation is to prevent errors with individual records from >>> stopping the entire pipeline. I'd also like to filter out and log bad >>> records at various stages. >>> >>> I could use RDD[Either[T]] for everything but that won't work for >>> DataFrames. I was wondering if anyone has had a similar situation and if >>> they found elegant ways to handle this? >>> >>> Thanks, >>> RJ >>> >> >> >
