Olivier Girardot created SPARK-6993:
---------------------------------------
Summary: Adding default max and min methods for JavaDoubleRDD
Key: SPARK-6993
URL: https://issues.apache.org/jira/browse/SPARK-6993
Project: Spark
Issue Type: Improvement
Components: Java API
Affects Versions: 1.2.3, 1.3.1, 1.4.0
Reporter: Olivier Girardot
Priority: Minor
The default Scala API allows you when manipulating standard types like Double
to simply write
{code:scala}
val doubles : RDD[Double] = ...
doubles.max()
doubles.min()
{code}
Using the implicit ordering, on the other hand for the Java API you have to
specify a Comparator to use :
{code:java}
JavaDoubleRDD doubles = ...mapToDouble...
doubles.max(Comparator.naturalOrder()) // for example
doubles.min(Comparator.naturalOrder()) // for example
{code}
What is even less consistent is that the *stats* API doesn't ask you for any
kind of comparator and uses the default ordering by itself.
This issue, and the associated pull request, is proposing to add default max
and min functions to the JavaDoubleRDD that will use the default Comparator.
But as **java.util.Comparator.naturalOrder()** is only a Java 8 API, I used in
my PR the Guava - Ordering.natural() method and added the associated test.
This PR does not add any additional dependencies, Guava is already used by core
and shaded.
Thank you for your time,
Olivier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
