[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hivemall/pull/122


---

[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...

[myui]

Github user myui commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/122#discussion_r144765390
  
--- Diff: core/src/main/java/hivemall/evaluation/HitRateUDAF.java ---
@@ -71,9 +71,6 @@
 + " - Returns HitRate")
 public final class HitRateUDAF extends AbstractGenericUDAFResolver {
 
-// prevent instantiation
-private HitRateUDAF() {}
-
--- End diff --

sure.


---

[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...

[myui]

Github user myui commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/122#discussion_r144765287
  
--- Diff: core/src/main/java/hivemall/evaluation/AUCUDAF.java ---
@@ -110,7 +110,7 @@ public ClassificationEvaluator() {}
 
 @Override
 public ObjectInspector init(Mode mode, ObjectInspector[] 
parameters) throws HiveException {
-assert (parameters.length == 2 || parameters.length == 3) : 
parameters.length;
+assert (0 < parameters.length && parameters.length <= 3) : 
parameters.length;
--- End diff --

It would be a bug and should use Preconditions. I guess enable assertion 
`-ea`  is not enabled (cc: @takuti )


---

[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/122#discussion_r144753835
  
--- Diff: core/src/main/java/hivemall/evaluation/HitRateUDAF.java ---
@@ -71,9 +71,6 @@
 + " - Returns HitRate")
 public final class HitRateUDAF extends AbstractGenericUDAFResolver {
 
-// prevent instantiation
-private HitRateUDAF() {}
-
--- End diff --

This prevents Spark from loading UDAFs by using reflection. Can we remove 
this?


---

[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/122#discussion_r144753777
  
--- Diff: core/src/main/java/hivemall/evaluation/AUCUDAF.java ---
@@ -110,7 +110,7 @@ public ClassificationEvaluator() {}
 
 @Override
 public ObjectInspector init(Mode mode, ObjectInspector[] 
parameters) throws HiveException {
-assert (parameters.length == 2 || parameters.length == 3) : 
parameters.length;
+assert (0 < parameters.length && parameters.length <= 3) : 
parameters.length;
--- End diff --

In Spark, this assertion fails because Spark passes a single parameter in 
`parameters` here for final output (IIUC [`AUC` finally outputs a single 
double-typed value for each 
group](https://github.com/apache/incubator-hivemall/pull/122/files#diff-9d758588c8fad559a15d0b2362e757b2R1134)).
 In Hive, does this work well?




---

[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...

[maropu]

GitHub user maropu opened a pull request:

https://github.com/apache/incubator-hivemall/pull/122

[HIVEMALL-147][Spark] Support all Hivemall functions of v0.5-rc.1 in Spark 
Dataframe

## What changes were proposed in this pull request?
This pr added more Hivemall functions for Spark DataFrame. However, some of 
the functions are not supported here because Spark simply cannot handle them 
(e.g., unsupported types, returned types depending on options, ...).

## What type of PR is it?
Feature

## What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-147

## How was this patch tested?
Added tests in `HivemallOpsWithFeatureSuite`.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maropu/incubator-hivemall HIVEMALL-147-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hivemall/pull/122.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #122


commit 4963c2e71279c095759ba4f545cbbb47cff667b7
Author: Takeshi Yamamuro 
Date:   2017-10-14T15:11:19Z

Support all Hivemall functions of v0.5-rc.1 in Spark Dataframe




---