I don't think you can get a SparkContext inside an RDD function (such as mapPartitions), but you shouldn't need to. Have you considered returning the data read from the database from mapPartitions to create a new RDD and then just save it to a file like normal?
For example: rddObject.mapPartitions(x => { x.map(getDataFromDB(_)) }, true).saveAsTextFile("hdfs:///some-folder/") Does that make sense? On Wed, Oct 1, 2014 at 12:52 AM, Henry Hung <ythu...@winbond.com> wrote: > Hi All, > > > > A noob question: > > How to get SparckContext inside mapPartitions? > > > > Example: > > > > Let’s say I have rddObjects that can be split into different partitions to > be assigned to multiple executors, to speed up the export data from > database. > > > > Variable sc is created in the main program using these steps: > > val conf = new SparkConf().setAppName("ETCH VM Get FDC") > > val sc = new SparkContext(conf) > > > > and here is the mapPartitions code: > > rddObject.mapPartitions(x => { > > val uuid = java.util.UUID.randomUUID.toString > > val path = new org.apache.hadoop.fs.Path(“hdfs:///some-folder/” + uuid) > > val fs = path.getFileSystem(sc.hadoopConfiguration) > > val pw = new PrintWriter(fs.create(path)) > > while (x.hasNext) { > > // … do something here, like fetch data from database and write it to > hadoop file > > pw.println(getDataFromDB(x.next)) > > } > > }) > > > > My question is how can I use sc to get the hadoopConfiguration, thus > enable me to create Hadoop file? > > > > Best regards, > > Henry > > ------------------------------ > The privileged confidential information contained in this email is > intended for use only by the addressees as indicated by the original sender > of this email. If you are not the addressee indicated in this email or are > not responsible for delivery of the email to such a person, please kindly > reply to the sender indicating this fact and delete all copies of it from > your computer and network server immediately. Your cooperation is highly > appreciated. It is advised that any unauthorized use of confidential > information of Winbond is strictly prohibited; and any information in this > email irrelevant to the official business of Winbond shall be deemed as > neither given nor endorsed by Winbond. > -- Daniel Siegmann, Software Developer Velos Accelerating Machine Learning 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001 E: daniel.siegm...@velos.io W: www.velos.io