[spark] branch master updated: [SPARK-31610][SPARK-31668][ML] Address hashingTF saving bug and expose hashFunc property in HashingTF

2020-05-12 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e248bc7 [SPARK-31610][SPARK-31668][ML] Address

[spark] branch branch-3.0 updated: [SPARK-31610][SPARK-31668][ML] Address hashingTF saving bug and expose hashFunc property in HashingTF

2020-05-12 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new b50d53b [SPARK-31610][SPARK-31668][ML

[spark] branch branch-3.0 updated: [SPARK-31497][ML][PYSPARK] Fix Pyspark CrossValidator/TrainValidationSplit with pipeline estimator cannot save and load model

2020-04-26 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 4421178 [SPARK-31497][ML][PYSPARK] Fix

[spark] branch master updated: [SPARK-31497][ML][PYSPARK] Fix Pyspark CrossValidator/TrainValidationSplit with pipeline estimator cannot save and load model

2020-04-26 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 4a21c4c [SPARK-31497][ML][PYSPARK] Fix Pyspark

[spark] branch branch-3.0 updated: [SPARK-30667][CORE] Add allGather method to BarrierTaskContext

2020-02-19 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new f482187 [SPARK-30667][CORE] Add allGather

[spark] branch master updated: [SPARK-30667][CORE] Add allGather method to BarrierTaskContext

2020-02-19 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new af63971 [SPARK-30667][CORE] Add allGather method

[spark] branch branch-3.0 updated: [SPARK-30667][CORE] Add allGather method to BarrierTaskContext

2020-02-13 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 6001866 [SPARK-30667][CORE] Add allGather

[spark] branch master updated: [SPARK-30667][CORE] Add allGather method to BarrierTaskContext

2020-02-13 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 57254c9 [SPARK-30667][CORE] Add allGather method

[spark] branch master updated: [SPARK-30154][ML] PySpark UDF to convert MLlib vectors to dense arrays

2020-01-06 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 88542bc [SPARK-30154][ML] PySpark UDF to convert

[spark] branch master updated: [SPARK-28978][ ] Support > 256 args to python udf

2019-11-08 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8152a87 [SPARK-28978][ ] Support > 256 a

[spark] branch master updated: [SPARK-29417][CORE] Resource Scheduling - add TaskContext.resource java api

2019-10-14 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a42d894 [SPARK-29417][CORE] Resource Scheduling

[spark] branch master updated: [SPARK-28206][PYTHON] Remove the legacy Epydoc in PySpark API documentation

2019-07-05 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new fe75ff8 [SPARK-28206][PYTHON] Remove the legacy

[spark] branch master updated: [SPARK-28115][CORE][TEST] Fix flaky test: SparkContextSuite.test resource scheduling under local-cluster mode

2019-06-20 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d98a5ce [SPARK-28115][CORE][TEST] Fix flaky test

[spark] branch master updated: [SPARK-28056][PYTHON] add doc for SCALAR_ITER Pandas UDF

2019-06-17 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 1b2448b [SPARK-28056][PYTHON] add doc

[spark] branch master updated: [SPARK-26412][PYSPARK][SQL] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series

2019-06-15 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6d441dc [SPARK-26412][PYSPARK][SQL] Allow Pandas

[spark] branch master updated: [SPARK-28030][SQL] convert filePath to URI in binary file data source

2019-06-12 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 4f4829b [SPARK-28030][SQL] convert filePath

[spark] branch master updated: [SPARK-27968] ArrowEvalPythonExec.evaluate shouldn't eagerly read the first row

2019-06-06 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 4d770db [SPARK-27968

[spark] branch master updated: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-06-04 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ac808e2 [SPARK-27366][CORE] Support GPU Resources

[spark] branch master updated: [SPARK-27488][CORE] Driver interface to support GPU resources

2019-05-23 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 74e5e41 [SPARK-27488][CORE] Driver interface

[spark] branch master updated: [SPARK-27588] Binary file data source fails fast and doesn't attempt to read very large files

2019-04-29 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 618d6bf [SPARK-27588] Binary file data source

[spark] branch master updated: [SPARK-27472] add user guide for binary file data source

2019-04-29 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new fbc7942 [SPARK-27472] add user guide for binary

[spark] branch master updated: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected

2019-04-28 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 20a3ef7 [SPARK-27534][SQL] Do not load `content

[spark] branch master updated: [SPARK-27473][SQL] Support filter push down for status fields in binary file data source

2019-04-21 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9793d9e [SPARK-27473][SQL] Support filter push

[spark] branch master updated: [SPARK-25348][SQL] Data source for binary files

2019-04-16 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 1bb0c8e [SPARK-25348][SQL] Data source for binary

[spark] branch master updated: [SPARK-27454][ML][SQL] Spark image datasource fail when encounter some illegal images

2019-04-15 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d35e81f [SPARK-27454][ML][SQL] Spark image

[spark] branch master updated: [SPARK-25970][ML] Add Instrumentation to PrefixSpan

2018-12-20 Thread meng
This is an automated email from the ASF dual-hosted git repository. meng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new aa0d4ca [SPARK-25970][ML] Add Instrumentation

spark git commit: [SPARK-25321][ML] Fix local LDA model constructor

2018-09-21 Thread meng
lob/c7860af328d519cf56441b4a7cd8e6ec9d9fee59/mleap-spark/src/main/scala/org/apache/spark/ml/bundle/ops/clustering/LDAModelOp.scala#L57) ## How was this patch tested? Manual. Closes #22510 from WeichenXu123/LDA_fix. Authored-by: WeichenXu Signed-off-by: Xiangrui Meng (cherry picked from commit 40edab209bdefe793b59b650099cea026c244

spark git commit: [SPARK-25321][ML] Fix local LDA model constructor

2018-09-21 Thread meng
lob/c7860af328d519cf56441b4a7cd8e6ec9d9fee59/mleap-spark/src/main/scala/org/apache/spark/ml/bundle/ops/clustering/LDAModelOp.scala#L57) ## How was this patch tested? Manual. Closes #22510 from WeichenXu123/LDA_fix. Authored-by: WeichenXu Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Com

spark git commit: [SPARK-25321][ML] Revert SPARK-14681 to avoid API breaking change

2018-09-21 Thread meng
How was this patch tested? N/A Closes #22492 from WeichenXu123/revert_tree_change. Authored-by: WeichenXu Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/138a6316 Tree: http://git-wip-us.apache.

spark git commit: [SPARK-22666][ML][FOLLOW-UP] Improve testcase to tolerate different schema representation

2018-09-19 Thread meng
; to tolerate different schema representation. Because file:/path and file:///path are both valid URI-ifications so in some environment the testcase will fail. ## How was this patch tested? Manual. Closes #22449 from WeichenXu123/image_url. Authored-by: WeichenXu Signed-off-by: Xiangrui Meng

spark git commit: [SPARK-22666][ML][FOLLOW-UP] Improve testcase to tolerate different schema representation

2018-09-19 Thread meng
; to tolerate different schema representation. Because file:/path and file:///path are both valid URI-ifications so in some environment the testcase will fail. ## How was this patch tested? Manual. Closes #22449 from WeichenXu123/image_url. Authored-by: WeichenXu Signed-off-by: Xiangrui

spark git commit: [SPARK-25345][ML] Deprecate public APIs from ImageSchema

2018-09-08 Thread meng
rom WeichenXu123/image_api_deprecate. Authored-by: WeichenXu Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/08c02e63 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/08c02e63 Diff: http://git-

spark git commit: [SPARK-25345][ML] Deprecate public APIs from ImageSchema

2018-09-08 Thread meng
rom WeichenXu123/image_api_deprecate. Authored-by: WeichenXu Signed-off-by: Xiangrui Meng (cherry picked from commit 08c02e637ac601df2fe890b8b5a7a049bdb4541b) Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/sp

spark git commit: [SPARK-22666][ML][SQL] Spark datasource for image format

2018-09-05 Thread meng
ons): 109s - `ImageSchema.read` (set 515 partitions): 105s So we can see that my image datasource implementation (this PR) bring some performance improvement compared against old`ImageSchema.read` API. Closes #22328 from WeichenXu123/image_datasource. Authored-by: WeichenXu Signed-off-by: Xiangrui M

spark git commit: [SPARK-25248][CORE] Audit barrier Scala APIs for 2.4

2018-09-04 Thread meng
mpl` to `TaskContext`, remaining package private, and used delegate methods to avoid inheriting `TaskContextImp` and exposing unnecessary APIs. TODOs: - [x] scala doc - [x] python doc (#22261 ). Closes #22240 from mengxr/SPARK-25248. Authored-by: Xiangrui Meng Signed-off-by: Xiangrui M

spark git commit: [SPARK-25248][.1][PYSPARK] update barrier Python API

2018-08-29 Thread meng
240 . Major changes: * export the public classes * expand the docs * add doc for BarrierTaskInfo.addresss cc: jiangxb1987 Closes #22261 from mengxr/SPARK-25248.1. Authored-by: Xiangrui Meng Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-

spark git commit: [SPARK-25266][CORE] Fix memory leak in Barrier Execution Mode

2018-08-29 Thread meng
ing jmap. Closes #22258 from sarutak/fix-barrierexec-oom. Authored-by: sarutak Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3864480e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/38644

spark git commit: [SPARK-25234][SPARKR] avoid integer overflow in parallelize

2018-08-24 Thread meng
ger overflow. ## How was this patch tested? unit test Closes #5 from mengxr/SPARK-25234. Authored-by: Xiangrui Meng Signed-off-by: Xiangrui Meng (cherry picked from commit 9714fa547325ed7b6a8066a88957537936b233dd) Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-25234][SPARKR] avoid integer overflow in parallelize

2018-08-24 Thread meng
ger overflow. ## How was this patch tested? unit test Closes #5 from mengxr/SPARK-25234. Authored-by: Xiangrui Meng Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9714fa54 Tree: http://git-

spark git commit: [SPARK-25095][PYSPARK] Python support for BarrierTaskContext

2018-08-21 Thread meng
wed for barrier tasks. ## How was this patch tested? Add new tests in `tests.py` Closes #22085 from jiangxb1987/python.barrier. Authored-by: Xingbo Jiang Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/sp

spark git commit: [SPARK-25161][CORE] Fix several bugs in failure handling of barrier execution mode

2018-08-21 Thread meng
ier ResultStage failed; * RDD.isBarrier() should not rely on `ShuffleDependency`s. ## How was this patch tested? Added corresponding test cases in `DAGSchedulerSuite` and `TaskSchedulerImplSuite`. Closes #22158 from jiangxb1987/failure. Authored-by: Xingbo Jiang Signed-off-by: Xiangrui M

spark git commit: [SPARK-24819][CORE] Fail fast when no enough slots to launch the barrier stage on job submitted

2018-08-15 Thread meng
987/SPARK-24819. Lead-authored-by: Xingbo Jiang Co-authored-by: Xiangrui Meng Signed-off-by: Xiangrui Meng Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bfb74394 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bfb74

spark git commit: [SPARK-25045][CORE] Make `RDDBarrier.mapParititions` similar to `RDD.mapPartitions`

2018-08-07 Thread meng
er Commit: d90f1336d87199aac56fe227a0fe14ab0ae3a332 Parents: 66699c5 Author: Xingbo Jiang Authored: Tue Aug 7 17:32:41 2018 -0700 Committer: Xiangrui Meng Committed: Tue Aug 7 17:32:41 2018 -0700 -- .../org/apache/spark/BarrierTaskContext.sca

spark git commit: [SPARK-24954][CORE] Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-08-03 Thread meng
org/repos/asf/spark/diff/92b48842 Branch: refs/heads/master Commit: 92b48842b944a3e430472294cdc3c481bad6b804 Parents: c32dbd6 Author: Xingbo Jiang Authored: Fri Aug 3 09:36:56 2018 -0700 Committer: Xiangrui Meng Committed: Fri Aug 3 09:36:56 2018 -0

spark git commit: [SPARK-24795][CORE][FOLLOWUP] Combine BarrierTaskContext with BarrierTaskContextImpl

2018-08-02 Thread meng
ark/diff/29077a1d Branch: refs/heads/master Commit: 29077a1d15e49dfafe7f2eab963830ba9cc6b29a Parents: bbdcc3b Author: Xingbo Jiang Authored: Thu Aug 2 17:19:42 2018 -0700 Committer: Xiangrui Meng Committed: Thu Aug 2 17:19:42 2018 -0

spark git commit: [SPARK-24820][SPARK-24821][CORE] Fail fast when submitted job contains a barrier stage with unsupported RDD chain pattern

2018-08-02 Thread meng
ter: Xiangrui Meng Committed: Thu Aug 2 09:36:26 2018 -0700 -- .../apache/spark/scheduler/DAGScheduler.scala | 55 ++- .../spark/BarrierStageOnSubmittedSuite.scala| 153 +++ 2 files changed, 207 inserti

spark git commit: [SPARK-24557][ML] ClusteringEvaluator support array input

2018-08-02 Thread meng
asf/spark/diff/57d99499 Branch: refs/heads/master Commit: 57d994994d27154f57f2724924c42beb2ab2e0e7 Parents: 166f346 Author: zhengruifeng Authored: Wed Aug 1 23:46:01 2018 -0700 Committer: Xiangrui Meng Committed: Wed Aug 1 23:46:01 2018 -0

spark git commit: [SPARK-15064][ML] Locale support in StopWordsRemover

2018-06-12 Thread meng
org/repos/asf/spark/diff/5d6a53d9 Branch: refs/heads/master Commit: 5d6a53d9831cc1e2115560db5cebe0eea2565dcd Parents: 1d7db65 Author: Lee Dongjin Authored: Tue Jun 12 08:16:37 2018 -0700 Committer: Xiangrui Meng Committed: Tue Jun 12 08:16:37 2018 -0

spark git commit: [SPARK-19826][ML][PYTHON] add spark.ml Python API for PIC

2018-06-11 Thread meng
84c Branch: refs/heads/master Commit: a99d284c16cc4e00ce7c83ecdc3db6facd467552 Parents: 3e5b4ae Author: Huaxin Gao Authored: Mon Jun 11 12:15:14 2018 -0700 Committer: Xiangrui Meng Committed: Mon Jun 11 12:15:14 2018 -0700 -- pyt

spark git commit: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule in ml/__init__.py and add ImageSchema into __all__

2018-06-08 Thread meng
org/repos/asf/spark/commit/173fe450 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/173fe450 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/173fe450 Branch: refs/heads/master Commit: 173fe450df203b262b58f7e71c6b52a79db95ee0 Parents: a5d775a Author: hyukjinkwon Authored: Fri Jun 8 09:32:11 20

spark git commit: [SPARK-24300][ML] change the way to set seed in ml.cluster.LDASuite.generateLDAData

2018-06-04 Thread meng
ree: http://git-wip-us.apache.org/repos/asf/spark/tree/ff0501b0 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ff0501b0 Branch: refs/heads/master Commit: ff0501b0c27dc8149bd5fb38a19d9b0056698766 Parents: b24d3db Author: Lu WANG Authored: Mon Jun 4 16:08:27 2018 -0700 Committer: Xiangrui M

spark git commit: [SPARK-24290][ML] add support for Array input for instrumentation.logNamedValue

2018-06-04 Thread meng
ANG Authored: Mon Jun 4 14:54:31 2018 -0700 Committer: Xiangrui Meng Committed: Mon Jun 4 14:54:31 2018 -0700 -- .../apache/spark/ml/clustering/BisectingKMeans.scala | 3 +-- .../apache/spark/ml/clustering/GaussianMixture

spark git commit: [SPARK-20114][ML][FOLLOW-UP] spark.ml parity for sequential pattern mining - PrefixSpan

2018-05-23 Thread meng
ds/master Commit: df125062c8dac9fee3328d67dd438a456b7a3b74 Parents: a40ffc6 Author: WeichenXu <weichen...@databricks.com> Authored: Wed May 23 11:00:23 2018 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Wed May 2

spark git commit: [SPARK-22884][ML] ML tests for StructuredStreaming: spark.ml.clustering

2018-05-17 Thread meng
s/master Commit: d4a0895c628ca854895c3c35c46ed990af36ec61 Parents: 439c695 Author: Sandor Murakozi <smurak...@gmail.com> Authored: Thu May 17 16:33:06 2018 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Thu May 17 16:33:06 2018 -0700 --

spark git commit: [SPARK-24115] Have logging pass through instrumentation class.

2018-05-17 Thread meng
May 17 11:13:16 2018 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Thu May 17 11:13:16 2018 -0700 -- .../scala/org/apache/spark/ml/tuning/CrossValidator.scala | 10 +- .../org/a

spark git commit: [SPARK-24155][ML] Instrumentation improvements for clustering

2018-05-14 Thread meng
4 13:35:54 2018 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Mon May 14 13:35:54 2018 -0700 -- .../org/apache/spark/ml/clustering/BisectingKMeans.scala | 7 +-- .../org/apache/spark/ml/clustering/

spark git commit: [SPARK-24132][ML] Instrumentation improvement for classification

2018-05-08 Thread meng
: 9498e52 Author: Lu WANG <lu.w...@databricks.com> Authored: Tue May 8 21:20:58 2018 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Tue May 8 21:20:58 2018 -0700 -- .../spark/ml/classification/Decisio

spark git commit: [SPARK-23975][ML] Add support of array input for all clustering methods

2018-05-07 Thread meng
d: Mon May 7 20:08:41 2018 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Mon May 7 20:08:41 2018 -0700 -- .../spark/ml/clustering/BisectingKMeans.scala | 21 - .../spark/ml/clustering/Gaussia

spark git commit: [SPARK-22735][ML][DOC] Added VectorSizeHint docs and examples.

2018-01-23 Thread meng
; Closes #20285 from MrBago/sizeHintDocs. (cherry picked from commit 05839d164836e544af79c13de25802552eadd636) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f8f522c0 T

spark git commit: [SPARK-22735][ML][DOC] Added VectorSizeHint docs and examples.

2018-01-23 Thread meng
6 Branch: refs/heads/master Commit: 05839d164836e544af79c13de25802552eadd636 Parents: dc4761f Author: Bago Amirbekian <b...@databricks.com> Authored: Tue Jan 23 14:11:23 2018 -0800 Committer: Xiangrui Meng <m...@databricks.com> Committed: Tue Jan 2

spark git commit: [SPARK-20088] Do not create new SparkContext in SparkR createSparkContext

2017-03-27 Thread meng
asf/spark/tree/0588dc7c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0588dc7c Branch: refs/heads/master Commit: 0588dc7c0a9f3180dddae0dc202a6d41eb43464f Parents: 8904934 Author: Hossein <hoss...@databricks.com> Authored: Mon Mar 27 08:53:45 2017 -0700 Committer: Xiangrui Meng <m

spark git commit: [SPARK-18793][SPARK-18794][R] add spark.randomForest/spark.gbt to vignettes

2016-12-13 Thread meng
ent minimal since users can type `?spark.randomForest` to see the full doc. cc: jkbradley Author: Xiangrui Meng <m...@databricks.com> Closes #16264 from mengxr/SPARK-18793. (cherry picked from commit 594b14f1ebd0b3db9f630e504be92228f11b4d9f) Signed-off-by: Xiangrui Meng <m...@databricks.co

spark git commit: [SPARK-18793][SPARK-18794][R] add spark.randomForest/spark.gbt to vignettes

2016-12-13 Thread meng
mal since users can type `?spark.randomForest` to see the full doc. cc: jkbradley Author: Xiangrui Meng <m...@databricks.com> Closes #16264 from mengxr/SPARK-18793. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/594b

spark git commit: [SPARK-18797][SPARKR] Update spark.logit in sparkr-vignettes

2016-12-12 Thread meng
6222 from wangmiao1981/veg. (cherry picked from commit 2aa16d03db79a642cbe21f387441c34fc51a8236) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9f0e3be6 Tree: http://git-wi

spark git commit: [SPARK-18797][SPARKR] Update spark.logit in sparkr-vignettes

2016-12-12 Thread meng
er Commit: 2aa16d03db79a642cbe21f387441c34fc51a8236 Parents: 417e45c Author: wm...@hotmail.com <wm...@hotmail.com> Authored: Mon Dec 12 22:41:11 2016 -0800 Committer: Xiangrui Meng <m...@databricks.com> Committed: Mon Dec 1

spark git commit: [SPARK-18812][MLLIB] explain "Spark ML"

2016-12-09 Thread meng
AQ-like entries to the MLlib user guide to explain "Spark ML" and reduce the confusion. I check the [Spark FAQ page](http://spark.apache.org/faq.html), which seems too high-level for the content here. So I added it to the MLlib user guide instead. cc: mateiz Author: Xiangrui Meng <m

spark git commit: [SPARK-18812][MLLIB] explain "Spark ML"

2016-12-09 Thread meng
e entries to the MLlib user guide to explain "Spark ML" and reduce the confusion. I check the [Spark FAQ page](http://spark.apache.org/faq.html), which seems too high-level for the content here. So I added it to the MLlib user guide instead. cc: mateiz Author: Xiangrui Meng <m...@da

spark git commit: [SPARK-17822][R] Make JVMObjectTracker a member variable of RBackend

2016-12-09 Thread meng
tor of `SerDe.sqlSerDe` to increase readability. ## How was this patch tested? * Added unit tests for `JVMObjectTracker`. * Wait for Jenkins to run full tests. Author: Xiangrui Meng <m...@databricks.com> Closes #16154 from mengxr/SPARK-17822. (cherry picked fr

spark git commit: [SPARK-17822][R] Make JVMObjectTracker a member variable of RBackend

2016-12-09 Thread meng
tor of `SerDe.sqlSerDe` to increase readability. ## How was this patch tested? * Added unit tests for `JVMObjectTracker`. * Wait for Jenkins to run full tests. Author: Xiangrui Meng <m...@databricks.com> Closes #16154 from mengxr/SPARK-17822. (cherry picked fr

spark git commit: [SPARK-17822][R] Make JVMObjectTracker a member variable of RBackend

2016-12-09 Thread meng
tor of `SerDe.sqlSerDe` to increase readability. ## How was this patch tested? * Added unit tests for `JVMObjectTracker`. * Wait for Jenkins to run full tests. Author: Xiangrui Meng <m...@databricks.com> Closes #16154 from mengxr/SPARK-17822. Project: http://git-wip-us.apache.org/repos/asf/spark/re

spark git commit: [SPARKR][MINOR] Fix LDA doc

2016-08-29 Thread meng
9 10:23:10 2016 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Mon Aug 29 10:23:10 2016 -0700 -- R/pkg/R/mllib.R | 10 +++--- 1 file changed, 7 inser

spark git commit: [SPARKR][MINOR] Update R DESCRIPTION file

2016-08-22 Thread meng
xcheun...@hotmail.com> Closes #14764 from felixcheung/rpackagedescription. (cherry picked from commit d2b3d3e63e1a9217de6ef507c350308017664a62) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/

spark git commit: [MINOR][R] add SparkR.Rcheck/ and SparkR_*.tar.gz to R/.gitignore

2016-08-21 Thread meng
Repository: spark Updated Branches: refs/heads/branch-2.0 faff9297d -> 26d5a8b0d [MINOR][R] add SparkR.Rcheck/ and SparkR_*.tar.gz to R/.gitignore ## What changes were proposed in this pull request? Ignore temp files generated by `check-cran.sh`. Author: Xiangrui Meng <m...@databric

spark git commit: [MINOR][R] add SparkR.Rcheck/ and SparkR_*.tar.gz to R/.gitignore

2016-08-21 Thread meng
Repository: spark Updated Branches: refs/heads/master e328f577e -> ab7143463 [MINOR][R] add SparkR.Rcheck/ and SparkR_*.tar.gz to R/.gitignore ## What changes were proposed in this pull request? Ignore temp files generated by `check-cran.sh`. Author: Xiangrui Meng <m...@databric

spark git commit: [SPARK-16443][SPARKR] Alternating Least Squares (ALS) wrapper

2016-08-19 Thread meng
tabricks.com> Authored: Fri Aug 19 14:24:09 2016 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Fri Aug 19 14:24:09 2016 -0700 -- R/pkg/NAMESPACE | 3 +

spark git commit: [SPARK-16446][SPARKR][ML] Gaussian Mixture Model wrapper in SparkR

2016-08-17 Thread meng
park/tree/4d92af31 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4d92af31 Branch: refs/heads/master Commit: 4d92af310ad29ade039e4130f91f2a3d9180deef Parents: e3fec51 Author: Yanbo Liang <yblia...@gmail.com> Authored: Wed Aug 17 11:18:33 2016 -0700 Committer: Xiangrui Meng <m

spark git commit: [SPARK-16294][SQL] Labelling support for the include_example Jekyll plugin

2016-06-29 Thread meng
688.png;> Author: Cheng Lian <l...@databricks.com> Closes #13972 from liancheng/include-example-with-labels. (cherry picked from commit bde1d6a61593aeb62370f526542cead94919b0c0) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark

spark git commit: [SPARK-16294][SQL] Labelling support for the include_example Jekyll plugin

2016-06-29 Thread meng
de1d6a6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bde1d6a6 Branch: refs/heads/master Commit: bde1d6a61593aeb62370f526542cead94919b0c0 Parents: d3af673 Author: Cheng Lian <l...@databricks.com> Authored: Wed Jun 29 22:50:53 2016 -0700 Committer: Xiangrui Meng <m...@databrick

spark git commit: [SPARK-16140][MLLIB][SPARKR][DOCS] Group k-means method in generated R doc

2016-06-29 Thread meng
om commit 8c9cd0a7a719ce4286f77f35bb787e2b626a472e) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1cde325e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1cde325e Diff:

spark git commit: [SPARK-16140][MLLIB][SPARKR][DOCS] Group k-means method in generated R doc

2016-06-29 Thread meng
e Parents: c6a220d Author: Xin Ren <iamsh...@126.com> Authored: Wed Jun 29 11:25:00 2016 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Wed Jun 29 11:25:00 2016 -0700 -- R/pkg/R/generics.R | 2 ++ R/

spark git commit: [MINOR][SPARKR] Fix arguments of survreg in SparkR

2016-06-29 Thread meng
026/16447617/368e0b18-3da2-11e6-8277-45640fb11859.png) Author: Yanbo Liang <yblia...@gmail.com> Closes #13970 from yanboliang/spark-16143-followup. (cherry picked from commit c6a220d756f23ee89a0d1366b20259890c9d67c9) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project:

spark git commit: [SPARK-16245][ML] model loading backward compatibility for ml.feature.PCA

2016-06-28 Thread meng
ted? existing ut and manual test for loading models saved by Spark 1.6. Author: Yanbo Liang <yblia...@gmail.com> Closes #13937 from yanboliang/spark-16245. (cherry picked from commit 0df5ce1bc1387a58b33cd185008f4022bd3dcc69) Signed-off-by: Xiangrui Meng <m...@databricks.com> Proj

spark git commit: [SPARK-16245][ML] model loading backward compatibility for ml.feature.PCA

2016-06-28 Thread meng
-0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Tue Jun 28 19:53:07 2016 -0700 -- .../scala/org/apache/spark/ml/feature/PCA.scala | 18 -- 1 file changed, 8 insert

spark git commit: [SPARK-16143][R] group AFT survival regression methods docs in a single Rd

2016-06-27 Thread meng
igned-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4c4f7775 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4c4f7775 Diff: http://git-wip-us.apache.org/repos/asf/spar

spark git commit: [SPARK-16231][PYSPARK][ML][EXAMPLES] dataframe_example.py fails to convert ML style vectors

2016-06-27 Thread meng
igned-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/27f3462d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/27f3462d Diff: http://git-wip-us.apache.org/repos/asf/spar

spark git commit: [SPARK-16231][PYSPARK][ML][EXAMPLES] dataframe_example.py fails to convert ML style vectors

2016-06-27 Thread meng
ler <cutl...@gmail.com> Authored: Mon Jun 27 12:58:39 2016 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Mon Jun 27 12:58:39 2016 -0700 -- examples/src/main/python/ml/dataframe_example.py | 4 +++- 1 file

spark git commit: [SPARK-16187][ML] Implement util method for ML Matrix conversion in scala/java

2016-06-27 Thread meng
om commit c17b1abff8f8c6d24cb0cf4ff4f8c14a780c64b0) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e4bb31fb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e4bb31fb Diff:

spark git commit: [SPARK-16187][ML] Implement util method for ML Matrix conversion in scala/java

2016-06-27 Thread meng
0 Parents: c48c8eb Author: Yuhao Yang <yuhao.y...@intel.com> Authored: Mon Jun 27 12:27:39 2016 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Mon Jun 27 12:27:39 2016 -0700 -- .../org/apache/spark/ml/l

spark git commit: [SPARK-16133][ML] model loading backward compatibility for ml.feature

2016-06-23 Thread meng
ted? existing ut and manual test for loading 1.6 models. Author: Yuhao Yang <yuhao.y...@intel.com> Author: Yuhao Yang <hhb...@gmail.com> Closes #13844 from hhbyyh/featureComp. (cherry picked from commit cc6778ee0bf4fa7a78abd30542c4a6f80ea371c5) Signed-off-by: Xiangrui Meng <m...@databri

spark git commit: [SPARK-16142][R] group naiveBayes method docs in a single Rd

2016-06-23 Thread meng
tps://cloud.githubusercontent.com/assets/829644/16320455/aad1f6d8-394c-11e6-8ef4-13bee989f52f.png) Author: Xiangrui Meng <m...@databricks.com> Closes #13877 from mengxr/SPARK-16142. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit

spark git commit: [SPARK-16142][R] group naiveBayes method docs in a single Rd

2016-06-23 Thread meng
tps://cloud.githubusercontent.com/assets/829644/16320455/aad1f6d8-394c-11e6-8ef4-13bee989f52f.png) Author: Xiangrui Meng <m...@databricks.com> Closes #13877 from mengxr/SPARK-16142. (cherry picked from commit 4a40d43bb29704734b8128bf2a3f27802ae34e17) Signed-off-by: Xiangrui Meng <m...@da

spark git commit: [SPARK-16177][ML] model loading backward compatibility for ml.regression

2016-06-23 Thread meng
ard compatibility for ml.regression ## How was this patch tested? existing ut and manual test for loading 1.6 models. Author: Yuhao Yang <hhb...@gmail.com> Closes #13879 from hhbyyh/regreComp. (cherry picked from commit 14bc5a7f36bed19cd714a4c725a83feaccac3468) Signed-off-by: Xiangru

spark git commit: [SPARK-16177][ML] model loading backward compatibility for ml.regression

2016-06-23 Thread meng
hu Jun 23 20:43:19 2016 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Thu Jun 23 20:43:19 2016 -0700 -- .../apache/spark/ml/regression/AFTSurvivalRegression.scala | 9 + .../org/apach

spark git commit: [SPARK-16164][SQL] Update `CombineFilters` to try to construct predicates with child predicate first

2016-06-23 Thread meng
epos/asf/spark/diff/91b1ef28 Branch: refs/heads/master Commit: 91b1ef28d134313d7b6faaffa1c390f3ca4455d0 Parents: 738f134 Author: Dongjoon Hyun <dongj...@apache.org> Authored: Thu Jun 23 15:27:43 2016 -0700 Committer: Xiangrui Meng <m...@databr

spark git commit: [SPARK-16164][SQL] Update `CombineFilters` to try to construct predicates with child predicate first

2016-06-23 Thread meng
<dongj...@apache.org> Closes #13872 from dongjoon-hyun/SPARK-16164. (cherry picked from commit 91b1ef28d134313d7b6faaffa1c390f3ca4455d0) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/

spark git commit: [SPARK-16130][ML] model loading backward compatibility for ml.classfication.LogisticRegression

2016-06-23 Thread meng
Yang <hhb...@gmail.com> Authored: Thu Jun 23 11:00:00 2016 -0700 Committer: Xiangrui Meng <m...@databricks.com> Committed: Thu Jun 23 11:00:00 2016 -0700 -- .../spark/ml/classification/LogisticRegression.scala |

spark git commit: [SPARK-16130][ML] model loading backward compatibility for ml.classfication.LogisticRegression

2016-06-23 Thread meng
d6d3a48dc) Signed-off-by: Xiangrui Meng <m...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/dff3d75d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/dff3d75d Diff: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-16154][MLLIB] Update spark.ml and spark.mllib package docs

2016-06-23 Thread meng
ice to update the package docs to reflect this change. ## How was this patch tested? Manually checked generated APIs. Author: Xiangrui Meng <m...@databricks.com> Closes #13859 from mengxr/SPARK-16154. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.a

spark git commit: [SPARK-16154][MLLIB] Update spark.ml and spark.mllib package docs

2016-06-23 Thread meng
uld be nice to update the package docs to reflect this change. ## How was this patch tested? Manually checked generated APIs. Author: Xiangrui Meng <m...@databricks.com> Closes #13859 from mengxr/SPARK-16154. (cherry picked from commit 65d1f0f716f50dd14b5dfe1e7fac772f1b4d2be0) Signed-off-by:

spark git commit: [SPARK-16155][DOC] remove package grouping in Java docs

2016-06-22 Thread meng
ing in `SparkBuild.scala`. I didn't find anyone complaining about missing groups since 1.5.0 on Google. Manually checked the generated Java API docs and confirmed that they are the same as in master. Author: Xiangrui Meng <m...@databricks.com> Closes #13856 from mengxr/SPARK-16155. (cherry picked fr

  1   2   3   4   5   6   7   8   9   10   >