I¹m running PySpark on YARN, and I¹m reading in SequenceFiles for which I have a custom KeyConverter class. My KeyConverter needs to have some configuration options passed to it, but I am unable to find a way to get the options to that class without modifying the Spark source. Is there a currently built-in method by which this can be done? The sequenceFile() method does not take any arguments for the KeyConverter, and the SparkContext doesn¹t seem to be accessible from my KeyConverter class, so I¹m at a loss as to how to accomplish this.
Thanks, Brett
smime.p7s
Description: S/MIME cryptographic signature