Hi all, I would like you to ask for an advise on how to efficiently make a join operation in Spark with tens of thousands of tiny files. A single file has a few KB and ~50 rows. In another scenario they might have 200 KB and 2000 rows.
To give you impression how they look like: File 01 ID | VALUE 01 | 10 02 | 12 03 | 55 ... File 02 ID | VALUE 01 | 33 02 | 21 03 | 53 ... and so on... ID is unique in a file, but repeats in every file. There is also a Special file which has the same form: File Special ID | VALUE 01 | 21 02 | 23 03 | 54 ... What I would like to get is a join of File 01..10000 with File Special to get a difference between values: File Result 01 = File Special - File 01 ID | VALUE 01 | 21-10 02 | 23-12 03 | 54-53 ... And save result to a csv, meaning 10000 new files. What's the best way of doing this? My idea was the following: 1. Read all Files with wholeTextFiles, each to a separate partition 2. Perform map-side join with broadcast variable inside mapPartitions (the "Special" file will be broadcasted). I am on Spark 1.3, but it can be upgraded if needed. Perhaps this could be done better in a dataframe? Then I would create one large dataframe, with additional "filename" key, i.e.: File | ID | Value 01 | 01 | 10 01 | 02 | 12 01 | 03 | 55 02 | 01 | 21 02 | 02 | 23 ... What would be then a way to make an efficient query over such dataframe? Any advice will be appreciated. Best regards, Lucas =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
