RJ Nowling created SPARK-6522:
---------------------------------
Summary: Standardize Random Number Generation
Key: SPARK-6522
URL: https://issues.apache.org/jira/browse/SPARK-6522
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.3.0
Reporter: RJ Nowling
Priority: Minor
Generation of random numbers in Spark has to be handled carefully since
references to RNGs copy the state to the workers. As such, a separate RNG
needs to be seeded for each partition. Each time random numbers are used in
Spark's libraries, the RNG seeding is re-implemented, leaving open the
possibility of mistakes.
It would be useful if RNG seeding was standardized through utility functions or
random number generation functions that can be called in Spark pipelines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
