Hi, All,
I am trying to access a Map from RDDs that are on different compute nodes,
but without success. The Map is like:
val map1 = Map("aa"->1,"bb"->2,"cc"->3,...)
All RDDs will have to check against it to see if the key is in the Map or
not, so seems I have to make the Map itself global, the problem is that if
the Map is stored as RDDs and spread across the different nodes, each node
will only see a piece of the Map and the info will not be complete to check
against the Map( an then replace the key with the corresponding value) E,g:
val matchs= Vecs.map(term=>term.map{case (a,b)=>(map1(a),b)})
But if the Map is not an RDD, how to share it like sc.broadcast(map1)
Any idea about this? Thanks!
Cheers,
Dan
