[
https://issues.apache.org/jira/browse/SPARK-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264207#comment-15264207
]
Saikat Kanjilal commented on SPARK-14302:
-----------------------------------------
Ok I finished my initial assessment of the ml directory, here are the things
that I see duplicated:
1) The initialization of the SparkContext
2) The initialization of SqlContext
3) Creation of data frames
Xusen should we move the functionality of the above into a common python file
and have that be referenced by all the other files. I can create a Common.py
and remove all the above code from all the files and have that be referenced if
needed.
I wil next begin my assesment of the mllib directory
> Python examples code merge and clean up
> ---------------------------------------
>
> Key: SPARK-14302
> URL: https://issues.apache.org/jira/browse/SPARK-14302
> Project: Spark
> Issue Type: Sub-task
> Components: Examples
> Reporter: Xusen Yin
> Priority: Minor
> Labels: starter
>
> Duplicated code that I found in python/examples/mllib and python/examples/ml:
> * python/ml
> ** None
> * Unsure duplications, double check
> ** dataframe_example.py
> ** kmeans_example.py
> ** simple_params_example.py
> ** simple_text_classification_pipeline.py
> * python/mllib
> ** gaussian_mixture_model.py
> ** kmeans.py
> ** logistic_regression.py
> * Unsure duplications, double check
> ** correlations.py
> ** random_rdd_generation.py
> ** sampled_rdds.py
> ** word2vec.py
> When merging and cleaning those code, be sure not disturb the previous
> example on and off blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
