It would seem that I have hit SPARK-10787, an OOME
during ClosureCleaner#ensureSerializable(). I am trying to run LSH over a
SparseVector consisting of ~4M features with no more than 3K non-zero
values per vector. I am hitting this OOME before even the hashes are
I know the issue is reported and open. My question is whether there have
since been any decent workarounds found or if anyone has any suggestions on
how I can debug whether there are specific instances of data that are
causing my issue that I might filter out prior to hitting this point.
I have set spark.serializer.objectStreamReset to 1 and it did not help. I
am rather new to Spark and I'm having a tough time following exactly where
in the execution of the job this is occurring and whether there is anything
within my control to avoid it.
Any insights would be greatly appreciated.