Hussein Hazimeh created SPARK-16431:
---------------------------------------
Summary: Add a unified method that accepts single instances in
transformers and predictors
Key: SPARK-16431
URL: https://issues.apache.org/jira/browse/SPARK-16431
Project: Spark
Issue Type: Improvement
Components: ML
Reporter: Hussein Hazimeh
Priority: Minor
Current transformers in spark.ml can only operate on DataFrames and don't have
a method that accepts single instances. A typical transformer has a
User-Defined Function (udf) in its *transform* method which includes a set of
operations on the features of a single instance:
{code}val column_operation = udf {operations on single instance}{code}
Adding a new method that operates directly on single instances (e.g. called
*transformInstance*) and using it in the udf instead can be useful:
{code}def transformInstance(features: featureType): OutputType = {operations on
single instance}
val column_operation = udf {transformInstance}{code}
Predictors also don’t have a public method that does predictions on single
instances. *transformInstance* can be easily added to predictors by acting as a
wrapper for the internal method predict (which takes features as input).
This simple change has (at least) three benefits.
# Providing a low-latency transformation/prediction method to support machine
learning applications that require real-time predictions. The current
*transform* method has a relatively high latency when transforming single
instances or small batches due to the overhead introduced by DataFrame
operations. I measured the latency required to classify a single instance in
the 20 Newsgroups dataset using the current *transform* method and the proposed
*transformInstance*. The ML pipeline contains a tokenizer, stopword remover,
TF hasher, IDF, scaler, and Logisitc Regression. The table below shows the
latency percentiles in milliseconds after measuring the time to classify 700
documents.
||Transformation Method||P50||P90||P99||Max||
|*transform*|31.44|39.43|67.75|126.97|
|*transformInstance*|0.16|0.38|1.16|3.2|
*transformInstance* is 200 times faster on average and can classify a document
in less than a millisecond. By profiling the code of *transform*, it turns out
that every transformer in the pipeline wastes 5 milliseconds on average in
DataFrame-related operations when transforming a single instance. This implies
that the latency increases linearly with the pipeline size which can be
problematic.
# Increasing code readability and allowing easier debugging as operations on
rows are now combined into a function that can be tested independently of the
higher-level *transform* method.
# Adding flexibility to create new models: for example, check this
[comment|https://github.com/apache/spark/pull/8883#issuecomment-215559305] on
supporting new ensemble methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]