Ben McCann created SPARK-16688:
----------------------------------
Summary: OpenHashSet.MAX_CAPACITY is always based on Int even when
using Long
Key: SPARK-16688
URL: https://issues.apache.org/jira/browse/SPARK-16688
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.6.2, 2.0.0
Reporter: Ben McCann
MAX_CAPACITY is hardcoded to a value of 1073741824:
{code}val MAX_CAPACITY = 1 << 30
class LongHasher extends Hasher[Long] {
override def hash(o: Long): Int = (o ^ (o >>> 32)).toInt
}{code}
I'd like to stick more than 1B items in my hashmap. Spark's all about big data,
right?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
