You can use RDD.wholeTextFiles().
For example, suppose all your files are under /tmp/ABC_input/,
val rdd = sc.wholeTextFiles("file:///tmp/ABC_input”)
val rdd1 = rdd.flatMap { case (path, content) =>
val fileName = new java.io.File(path).getName
content.split("\n").map { line =>
How I can get the file name of each record being reading?
suppose input file ABC_input_0528.txt contains
111,abc,234
222,xyz,456
suppose input file ABC_input_0531.txt contains
100,abc,299
200,xyz,499
and I need to create one final output with file name in each record using
dataframes
my output
