I work with a lot of data in a long format, cases in which an ID column is
repeated, followed by a variable and a value column like so:
|ID | var | value |
| A | v1 | 1.0 |
| A | v2 | 2.0 |
| B | v1 | 1.5 |
| B | v3 | -1.0 |
It seems to me that Spark doesn't provide any clear native way to transform
data of this format into a Vector() or VectorUDT() type suitable for
machine learning algorithms.
The best solution I've found so far (which isn't very good) is to group by
ID, perform a collect_list, and then use a UDF to translate the resulting
array into a vector datatype.
I can get kind of close like so:
indexer = MF.StringIndexer(inputCol = 'var', outputCol = 'varIdx')
But the resultant 'val' vector is out of index order, and still would need
to be parsed.
What's the current preferred way to solve a problem like this?