[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760166#comment-15760166 ] ASF GitHub Bot commented on HIVEMALL-22: Github user asfgit closed the pull request at: https://github.com/apache/incubator-hivemall/pull/11 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744469#comment-15744469 ] ASF GitHub Bot commented on HIVEMALL-22: Github user asfgit closed the pull request at: https://github.com/apache/incubator-hivemall/pull/15 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724906#comment-15724906 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/13 [WIP] Kernelized Passive-Aggressive Algorithm for Classification This PR introduce the following algorithms: - Passive-Aggressive with Polynomial Kernel - Polynomial Kernel with Inverted Indices (PKI) - Kernel Expansion - PA-I, PA-II This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/304) by @L3Sota that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-304 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/13.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13 commit 4cb84773f4c88f757d70b480a80068d597e756a7 Author: myui Date: 2016-06-16T22:59:11Z Changed the behavior of binarize_label commit adb23e3c537d9c2d2ce36c27700a85ff6852964b Author: Sotaro Sugimoto Date: 2016-04-12T06:23:12Z Added Kernelized Passive-Aggressive Algorithm for Classification Problems. commit bc228746c7f7b1abc0f6e7846d52ca1edad7fea7 Author: Sotaro Sugimoto Date: 2016-04-12T09:14:45Z Bug fixes Also newly added KernelizedPassiveAggressiveUDTFTest based on PassiveAggressiveUDTFTest, but tests are not done for newly added functionality. commit adac4e5bd0d89cc8c68768c9fb5bc1765bc3fac3 Author: Sotaro Sugimoto Date: 2016-04-14T09:51:14Z Test for news20 written; some bugs fixed. Bug still occurs for news20-small.binary where the loss does not match between udtf and udtfPKI starting at sample 51, where the label switches from all -1s to all +1s. Also, news20.binary not included due to large filesize. commit 6072a2ae7f1d0f8a0023dbe428a38c201386e38e Author: myui Date: 2016-04-14T11:51:57Z Refactored to use Bitset for support vector indices commit ecf4f3ab55ccdbe7c57bcd21b292e485465fb566 Author: myui Date: 2016-04-14T11:55:49Z Changed resource location of unit tests commit 3994cc7bb706e0b073278176eee6a1a5b78bebf6 Author: Sotaro Sugimoto Date: 2016-04-15T06:17:18Z Both normal and PKI versions of KPA now learn identically. Note that testTime was added to see how much faster PKI is, but it appears that PKI is actually slower than normal KPA for non-sparse support vectors, since PKI must go through the work of finding all matching support vectors, which is O(number of support vectors) for a completely full match. commit 31d981bd0082d4d5c40d7eb536f47f70455a3c52 Author: Sotaro Sugimoto Date: 2016-04-15T09:38:49Z Fixed incorrect alpha calculation/storage. Also fixed tests and asserts to match correct alpha calculations. commit e2812fd035f51877167f0c77aff397fa45be0bef Author: Sotaro Sugimoto Date: 2016-04-15T09:59:16Z Added FloatArrayList. Also changed alpha from DoubleArrayList to FloatArrayList. commit c793d47e0de18d32a56d2d1fd1744470f7283613 Author: Sotaro Sugimoto Date: 2016-04-20T03:14:12Z Added kernel expansion option (untested) commit 4716b34ad5038761d18e422833d9334140dd0908 Author: Sotaro Sugimoto Date: 2016-04-21T03:34:26Z Tests for KPA added. Note that kernel splitting (in KernelExpansionKPA) is incomplete. commit f986803901824ad831c7c8a2504c26797d765f23 Author: Sotaro Sugimoto Date: 2016-04-26T04:21:22Z Kernel Expansion complete with tests. Also removed Kernel Splitting (probably uses too much memory). commit b0a0179b0bc1f50403eb1f5534bfb870113f9777 Author: myui Date: 2016-12-02T07:55:49Z Merge branch 'feature/kernelized_pa' of https://github.com/L3Sota/hivemall into JIRA-22/pr-304 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.c
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724928#comment-15724928 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/6 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724922#comment-15724922 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/11 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[
https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724925#comment-15724925
]
ASF GitHub Bot commented on HIVEMALL-22:
GitHub user myui reopened a pull request:
https://github.com/apache/incubator-hivemall/pull/11
[WIP] Support Feature Selection UDFs
This PR is based on [a pending
PR](https://github.com/myui/hivemall/pull/356) by @takuti that is sent before
Hivemall entered Apache Incubator.
See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22
) for tracking the status of this issue.
---
### Sample table
| time | x |
| --: | --: |
| 1 | 182.478 |
| 2 | 176.231 |
| 3 | 183.917 |
| 4 | 177.798 |
| 5 | 165.469 |
| ... | ... |
(14398 points from [twitter
data](https://blog.twitter.com/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series))
### Usage
``` sql
create temporary function sst as
'hivemall.anomaly.SingularSpectrumTransformUDF';
```
``` sql
SELECT
time,
-- x is double or array
-- sst(x) AS res
sst(x, "-th 0.005") AS res
FROM
twitter_timeseries
ORDER BY time ASC
;
```
### Results
```
7551{"changepoint_score":0.00453049288071683,"is_changepoint":false}
7552{"changepoint_score":0.004711244102524104,"is_changepoint":false}
7553{"changepoint_score":0.004814871928978115,"is_changepoint":false}
7554{"changepoint_score":0.004968089640799422,"is_changepoint":false}
7555{"changepoint_score":0.005709056330104878,"is_changepoint":true}
7556{"changepoint_score":0.0044279766655132,"is_changepoint":false}
7557{"changepoint_score":0.0034694956722586268,"is_changepoint":false}
7558{"changepoint_score":0.002549056569322694,"is_changepoint":false}
7559{"changepoint_score":0.0017395109108403473,"is_changepoint":false}
7560{"changepoint_score":0.0010629833145070489,"is_changepoint":false}
```
On the naive SVD-based implementation, elapsed time was about 20 sec. for
the 14398 samples (vs. 10 sec on ChangeFinder)
### Observations
The change-point scores are much more stable compared to ChangeFinder, and
change-point scores are always in [0, 1]. However, since the scores are quite
noisy, too many change-points are detected. Smoothing scores like ChangeFinder
is practically important.
In terms of running time, the naive SVD-based implementation is clearly
inefficient. So, the Lanczos-based efficient variant should be supported.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-356
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/11.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11
commit 3ebd771ee4bebf14769b7c240f8b28b9d5d10e86
Author: Takuya Kitazawa
Date: 2016-09-26T08:12:01Z
Implement initial SST-based change-point detector
commit bde06e0952445bf60a9aef4bca182c0afe87e250
Author: Takuya Kitazawa
Date: 2016-09-27T05:06:20Z
Rename SSTChangePoint -> SingularSpectrumTransform
commit 2bfd1270b1e9b79185a41cbe2568f2ce968d4a71
Author: Takuya Kitazawa
Date: 2016-09-28T02:16:56Z
Add references for the original SST papers
commit 998203d5e8623d6282c2b187df24e4da7d41c16b
Author: Takuya Kitazawa
Date: 2016-09-28T10:49:48Z
Support implicit-Krylov-approximation-based efficient SST
commit cc34435155e86718acb49fa42208aff730bb756c
Author: myui
Date: 2016-12-02T07:55:23Z
Merge branch 'sst-changepoint' of https://github.com/takuti/hivemall into
JIRA-22/pr-356
> Review and merge pending Pull Requests before entering Incubator
>
>
> Key: HIVEMALL-22
> URL: https://issues.apache.org/jira/browse/HIVEMALL-22
> Project: Hivemall
> Issue Type: New Feature
>Reporter: Makoto Yui
>Assignee: Makoto Yui
>
> Need to review and merge pending Pull Requests in
> https://github.com/myui/hivemall/pulls
> * Feature Selection
> https://github.com/myui/hivemall/pull/385
> * SST change point detection
> https://github.com/myui/hivemall/pull/356
> * Checkstyle
> https://github.com/myui/hivemall/pull/343
> * System Test
> https://github.com/myui/hivemall/pull/336
> * Kernelized Passive Aggressive
> https://github.com/myui/hivemall/pull/304
> * Separate Optimizer
> https://github.com/myui/hivemall/pull/285
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724911#comment-15724911 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/14 [WIP] Separate optimizer implementations This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/285) by @maropu that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-285 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/14.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14 commit f81948c5c7b83155eb29369a59f1fc65bb607f91 Author: Takeshi YAMAMURO Date: 2016-05-02T14:43:42Z Add optimizer implementations commit 3620eb89993db22ce8aee924d3cc0df33a5f9618 Author: Takeshi YAMAMURO Date: 2016-09-20T16:52:22Z Revert some modifications commit 9ca8bce75333a3ebb43d2492bf5df815ccf869dc Author: myui Date: 2016-12-02T07:56:40Z Merge branch 'AddOptimizers' of https://github.com/maropu/hivemall into JIRA-22/pr-285 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724901#comment-15724901 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/12 [WIP] Implement System Test This PR introduces System Tests of Hivemall. Usage and example are written in `systemtest/README.md` . This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/336) by @amaya382 that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-336 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/12.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12 commit f4d23a5daaa668fa7292a0ed90aaf58ef836c5c2 Author: amaya Date: 2016-08-23T09:19:53Z add feature for testing UDF commit 0b91aca3888b0a28b6538b314fd652c8a0ae65b5 Author: amaya Date: 2016-09-05T06:04:12Z add usage commit f55fe9f9191f2bed3022f1da1ae149c571c8db83 Author: amaya Date: 2016-09-05T09:36:32Z move systemtest into specific package commit 827cde18592e636eb7239eef4744b2d64856f029 Author: amaya Date: 2016-09-05T09:37:00Z mod project settings of systemtest commit cbb14da8db352d2817c9741afae884f803c31607 Author: amaya Date: 2016-09-05T09:46:18Z add ordered/unordered result matching commit 398be69eb72bbdd43d9e989512368e1935e4e0b0 Author: amaya Date: 2016-09-05T10:08:36Z add license header commit 1e82b3eb07be3b7719c8a81b8fb2d957534e45a0 Author: amaya Date: 2016-09-07T05:01:31Z enable to use external properties commit dd8f37dc07ba25682cc83eedef86b0ab6ae6800e Author: amaya Date: 2016-09-07T05:15:35Z update README about external properties commit aa9e9ff5c5dd8b4a5d0ce0e8e2cadd988aa9133b Author: amaya Date: 2016-09-07T06:07:38Z suppress warnings commit fbbe9fe947424a77936cd4332162e9a10fd1 Author: amaya Date: 2016-09-07T06:33:25Z potential bug to exception commit 09af57a0fea6205bc619e45363309b8b69a76634 Author: amaya Date: 2016-09-07T08:04:38Z add ordered/unordered result matching for multiple queries commit 443c81788c93a1aa46699fdad2e981afcb18a0ae Author: amaya Date: 2016-09-07T08:55:57Z refine README commit f84eafc41fa15b3810320dda5a644cf8d6df3637 Author: amaya Date: 2016-09-07T09:42:25Z Merge branch 'master' into 'feature/systemtest' resolved conflict pom.xml commit 265d910d5dbe06d839c7a190018021c0745b5baa Author: amaya Date: 2016-09-08T08:43:30Z refine README commit 45cec9e72fb41b4f97e9fca0f1348d187cda9d35 Author: amaya Date: 2016-09-09T05:46:27Z support array and map in insert statement commit 2d8c794b07a5f06077817dafee868e91a5e36393 Author: amaya Date: 2016-09-09T05:49:35Z fix error messages commit 6e6512a81132c4b7e5d8e1d0e01b99cfe440b4cd Author: amaya Date: 2016-09-12T06:12:32Z make code tougher commit d13dc1125cfd81bbe6ef49c24b4c9f0ca4343e8b Author: amaya Date: 2016-09-12T06:22:45Z avoid running with no runner commit e823eb240a1a2ca10802f67e92d812e90be4647f Author: amaya Date: 2016-09-13T07:03:32Z refresh null check commit faebaf9b9082375dea8a53d3dd5b3857eee3a458 Author: amaya Date: 2016-09-13T07:26:56Z mod name and inheritance structure commit 43ca0c86936f3ccc7f825db3c4f4ecaa48087917 Author: amaya Date: 2016-11-16T06:23:49Z Update license headers commit ba91267796cbfdee53aaef02af882aff591fb8f7 Author: amaya Date: 2016-11-17T05:15:03Z Mod README commit 33eab26f383dbdbce00a209e742b611a63d953cf Author: amaya Date: 2016-11-17T05:16:14Z Add exception commit 1f3df54c0183a61390f58b94f58c12e531754a09 Author: amaya Date: 2016-11-17T16:57:31Z Make dir name static commit 3550fd30af3a01f4217c075a3b814952b406aebe Author: amaya Date: 2016-11-17T16:57:47Z Mod assert methods commit 144cb504d674d2509620ce0d315694be0f664f42 Author: amaya Date: 2016-11-17T16:58:31Z Fix process of tdprop commit 7447dde61f3a9cb8e3ba5ab278a260d0a0615524 Author: amaya Date: 2016-11-17T18:23:46Z Mod README commit ddd8dc2dbf8222c9d9d84b038dbdcd9aef1f1a87 Author: amaya Date: 2016-11-17T19:22:51Z Refine access modifiers/calls commit 798ec6a73ca37d474137fc82db1c22a92521307d Author: amaya Date: 2016-11-17T19:27:59Z Update license header commit 075f93485e41ac8f26451f68b0f38737849b04a7 Author: myui Date: 2016-12-02T07:55:34Z Merge branch 'feature/systemtest' of https://github.com/amaya382/hi
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724892#comment-15724892 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/11 [WIP] Support Feature Selection UDFs This PR introduces two feature selection schemes: `Chi-Square test` and `Signal Noise Ratio`. This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/385) by @amaya382 that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-356 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/11.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11 commit 3ebd771ee4bebf14769b7c240f8b28b9d5d10e86 Author: Takuya Kitazawa Date: 2016-09-26T08:12:01Z Implement initial SST-based change-point detector commit bde06e0952445bf60a9aef4bca182c0afe87e250 Author: Takuya Kitazawa Date: 2016-09-27T05:06:20Z Rename SSTChangePoint -> SingularSpectrumTransform commit 2bfd1270b1e9b79185a41cbe2568f2ce968d4a71 Author: Takuya Kitazawa Date: 2016-09-28T02:16:56Z Add references for the original SST papers commit 998203d5e8623d6282c2b187df24e4da7d41c16b Author: Takuya Kitazawa Date: 2016-09-28T10:49:48Z Support implicit-Krylov-approximation-based efficient SST commit cc34435155e86718acb49fa42208aff730bb756c Author: myui Date: 2016-12-02T07:55:23Z Merge branch 'sst-changepoint' of https://github.com/takuti/hivemall into JIRA-22/pr-356 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724920#comment-15724920 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/15 [WIP] Support Feature Selection UDFs This PR introduces two feature selection schemes: `Chi-Square test` and `Signal Noise Ratio`. This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/385) by @amaya382 that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-385 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/15.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15 commit 2dc176a760b553214624e98f885a719ee196cc4e Author: amaya Date: 2016-09-16T06:46:44Z add HiveUtils.isNumberListOI commit 56adf2d4e8b2591c31b846b8980016d3dafdbacc Author: amaya Date: 2016-09-16T06:48:33Z add HiveUtils.asDoubleOI commit 6f9b4fa0acebf604882240ccd5507d9df45bab2d Author: amaya Date: 2016-09-16T06:52:54Z add transpose_and_dot commit d3009be59bcf314b373038e3db8903a041396931 Author: amaya Date: 2016-09-16T07:00:58Z add chi2 and chi2_test commit d8f1005bb9fbf769b117290582bed18d7607a94a Author: amaya Date: 2016-09-20T03:01:46Z mod number format commit d0e97e6ff71b2072ec5235cc3ac169162d59da59 Author: amaya Date: 2016-09-20T03:02:28Z add HiveUtils.isNumberListListOI commit 7b07e4a6e1f700ba0a6e5b68659a040a3d89aa2f Author: amaya Date: 2016-09-20T03:03:44Z change interface of chi2 commit e9d1a94f29f31e2910a54add7c2625825d715318 Author: amaya Date: 2016-09-20T07:55:57Z add array_top_k_indices commit 1ab9b0974ca4203c00175469b7b75d5b65209547 Author: amaya Date: 2016-09-20T07:56:15Z add subarray_by_indices commit ad81b3aa5a0bbb7c248d127ba44608578c01ae00 Author: amaya Date: 2016-09-20T08:05:55Z add license and format commit be1ea37a0f5048cde4284107c04e109f0f526b42 Author: amaya Date: 2016-09-20T09:00:49Z add ddl definitions commit 89c81aacf5b13f6e125723cb5c70574c10ae Author: amaya Date: 2016-09-21T01:56:59Z change to select_k_best commit 6dc234490dc25f563b22e5659c378e6ebcf8dcdb Author: amaya Date: 2016-09-21T02:41:59Z standardize to chi2 commit a16a3fde844ba381dee7eb1e9608ddc2dcfb96fc Author: amaya Date: 2016-09-21T04:10:18Z refine chi2 commit abbf5492b95dd69e347580c59ac044a78627c547 Author: amaya Date: 2016-09-21T04:11:00Z refine transpose_and_dot commit b8cf39684496f2511e59294041d443b9438394a9 Author: amaya Date: 2016-09-21T06:02:12Z fix chi2 commit a882c5f9f8067b911254dfc43d268de06a5490f9 Author: amaya Date: 2016-09-21T07:00:36Z mod chi2 function name commit 5088ef36367df1cd51ae62f1c044933676975e2e Author: amaya Date: 2016-09-21T07:22:09Z add tests commit 22a608ee1c7239b2953183b5341f80c58b1e7045 Author: amaya Date: 2016-09-26T08:07:55Z add snr commit a1f8f958c99f3cde9e48b6d80d364004f6d98cc2 Author: amaya Date: 2016-09-27T06:58:33Z integrate chi2 and SNR into hivemall.spark commit aa7d5299739349b49ef4f50cc2c1969f5cb8a78f Author: amaya Date: 2016-09-27T07:02:02Z Merge 'master' into 'feature/feature_selection' commit 1347de985ea6f8028c9d381f8827882ad39ad3a7 Author: amaya Date: 2016-09-28T05:22:05Z refine feature selection in spark integration commit 8e2842cf8c272642feaa76bf95e8fa463b0322dc Author: amaya Date: 2016-09-28T05:24:19Z refine tests commit 4cfa4e5ac15a6535b187c23616c205696a1cd13b Author: amaya Date: 2016-09-28T09:26:01Z mod SNR for corner cases commit 80be81ecf92cd4675dcdfaa5f456d84d484d6c44 Author: amaya Date: 2016-09-28T11:01:08Z minor fix commit 8d9f0d4c00758324029d342eb4b892e046ca4a49 Author: amaya Date: 2016-09-29T02:02:14Z minor fix commit ce4a48980e33b9f16c74a62fcea6878f28b9c08b Author: amaya Date: 2016-09-30T08:05:20Z change method of testing for spark commit 67ba9631af3c231b7abd145134d17237b6aca0a5 Author: myui Date: 2016-11-21T09:19:45Z Merge branch 'feature/feature_selection' of https://github.com/amaya382/hivemall into feature_selection # Conflicts: # core/src/main/java/hivemall/utils/hadoop/HiveUtils.java # core/src/main/java/hivemall/utils/math/StatsUtils.java # spark/spark-1.6/src/main/scala/org/apache/spark/sql/hive/GroupedDataEx.scala # spark/spark-1.6/src/test/scala/org/apache/spark/sql/hive/HivemallOpsSuite.sc
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724912#comment-15724912 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/3 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724902#comment-15724902 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/5 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724907#comment-15724907 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/4 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724894#comment-15724894 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/7 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724345#comment-15724345 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui commented on the issue: https://github.com/apache/incubator-hivemall/pull/10 Good to hear that you can run a ASF development procedure :-) > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722433#comment-15722433 ] ASF GitHub Bot commented on HIVEMALL-22: Github user Lewuathe commented on the issue: https://github.com/apache/incubator-hivemall/pull/10 Thanks. I updated the site. http://hivemall.incubator.apache.org/ > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721930#comment-15721930 ] ASF GitHub Bot commented on HIVEMALL-22: Github user asfgit closed the pull request at: https://github.com/apache/incubator-hivemall/pull/10 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721695#comment-15721695 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui commented on the issue: https://github.com/apache/incubator-hivemall/pull/10 @Lewuathe Could you update the project site as well? Do you know the instructions? You can use `bin/build_site.sh` to build the site. git://git.apache.org/incubator-hivemall-site.git (asf branch) > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721669#comment-15721669 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui commented on the issue: https://github.com/apache/incubator-hivemall/pull/10 hmm... It seems the contributor is not tracked on github. When using github squash web UI, it was tracked https://github.com/apache/incubator-hivemall/commit/effe461446f2af84931901c112d542b8d2e69cfd though > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721648#comment-15721648 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui commented on the issue: https://github.com/apache/incubator-hivemall/pull/10 Merged https://github.com/apache/incubator-hivemall/commit/556503b1ac4d67bb8b0cb06e20afffdcaa8f5851 Thank you for the contribution. Could you close this issue? > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721618#comment-15721618 ] Kai Sasaki commented on HIVEMALL-22: How can we add CI on incubator-hivemall? Can we use Jenkins of ASF? > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721615#comment-15721615 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user Lewuathe opened a pull request: https://github.com/apache/incubator-hivemall/pull/10 HIVEMALL-22: Update news feed for next Apache release see: https://issues.apache.org/jira/browse/HIVEMALL-22 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Lewuathe/incubator-hivemall renew-initial-apache-release-news Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/10.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10 commit 796d348e0f73a668aa810c77a6301a063f5f9523 Author: Lewuathe Date: 2016-12-05T08:30:58Z HIVEMALL-22: Update news feed for next Apache release > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721516#comment-15721516 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/9 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721403#comment-15721403 ] ASF GitHub Bot commented on HIVEMALL-22: Github user Lewuathe closed the pull request at: https://github.com/apache/incubator-hivemall/pull/8 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721402#comment-15721402 ] ASF GitHub Bot commented on HIVEMALL-22: Github user Lewuathe commented on the issue: https://github.com/apache/incubator-hivemall/pull/8 Sure will do. Thanks. > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721388#comment-15721388 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui commented on the issue: https://github.com/apache/incubator-hivemall/pull/8 Could you please close this PR? PR from your fork is expected. I cannot delete this branch while I removed it from ASF git :-( Found a typo in `Ou*t* project repository was migrated` => `Our project repository migrated` BTW, it's a meaningless new feed. `We are planning the initial Apache release on Q1, 2017. Learn more` would be a better message. > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717683#comment-15717683 ] ASF GitHub Bot commented on HIVEMALL-22: Github user Lewuathe commented on the issue: https://github.com/apache/incubator-hivemall/pull/8 Thanks. I updated to announce repository migration. > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717425#comment-15717425 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui commented on the issue: https://github.com/apache/incubator-hivemall/pull/8 Better to remain a news feed and `learn more` button on the top, instead of removing info. > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714681#comment-15714681 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user Lewuathe opened a pull request: https://github.com/apache/incubator-hivemall/pull/8 HIVEMALL-22: Remove migrating repository alert Since migration of repository has been done, we can remove migrating alert. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/incubator-hivemall remove-migrating-repo-alert Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/8.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8 commit d05538b575559ef3b472c2bca1a6cb32f53d5b3a Author: Lewuathe Date: 2016-12-02T10:06:49Z HIVEMALL-22: Remove migrating repository alert > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714481#comment-15714481 ] Makoto Yui commented on HIVEMALL-22: [~ozawa] Could you create an issue for your CheckStyle PR [1] and send a PR? [1] https://github.com/myui/hivemall/pull/343 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714471#comment-15714471 ] Makoto Yui commented on HIVEMALL-22: Yes, PR moved to ASF github repository. https://github.com/apache/incubator-hivemall/pull/3 https://github.com/apache/incubator-hivemall/pull/4 https://github.com/apache/incubator-hivemall/pull/5 https://github.com/apache/incubator-hivemall/pull/6 https://github.com/apache/incubator-hivemall/pull/7 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714449#comment-15714449 ] Kai Sasaki commented on HIVEMALL-22: New pull requests in incubator-hivemall are integrated properly. Nice! > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714438#comment-15714438 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/7 [WIP] Support Feature Selection UDFs This PR introduces two feature selection schemes: `Chi-Square test` and `Signal Noise Ratio`. This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/385) by @amaya382 that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-385 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/7.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7 commit 2dc176a760b553214624e98f885a719ee196cc4e Author: amaya Date: 2016-09-16T06:46:44Z add HiveUtils.isNumberListOI commit 56adf2d4e8b2591c31b846b8980016d3dafdbacc Author: amaya Date: 2016-09-16T06:48:33Z add HiveUtils.asDoubleOI commit 6f9b4fa0acebf604882240ccd5507d9df45bab2d Author: amaya Date: 2016-09-16T06:52:54Z add transpose_and_dot commit d3009be59bcf314b373038e3db8903a041396931 Author: amaya Date: 2016-09-16T07:00:58Z add chi2 and chi2_test commit d8f1005bb9fbf769b117290582bed18d7607a94a Author: amaya Date: 2016-09-20T03:01:46Z mod number format commit d0e97e6ff71b2072ec5235cc3ac169162d59da59 Author: amaya Date: 2016-09-20T03:02:28Z add HiveUtils.isNumberListListOI commit 7b07e4a6e1f700ba0a6e5b68659a040a3d89aa2f Author: amaya Date: 2016-09-20T03:03:44Z change interface of chi2 commit e9d1a94f29f31e2910a54add7c2625825d715318 Author: amaya Date: 2016-09-20T07:55:57Z add array_top_k_indices commit 1ab9b0974ca4203c00175469b7b75d5b65209547 Author: amaya Date: 2016-09-20T07:56:15Z add subarray_by_indices commit ad81b3aa5a0bbb7c248d127ba44608578c01ae00 Author: amaya Date: 2016-09-20T08:05:55Z add license and format commit be1ea37a0f5048cde4284107c04e109f0f526b42 Author: amaya Date: 2016-09-20T09:00:49Z add ddl definitions commit 89c81aacf5b13f6e125723cb5c70574c10ae Author: amaya Date: 2016-09-21T01:56:59Z change to select_k_best commit 6dc234490dc25f563b22e5659c378e6ebcf8dcdb Author: amaya Date: 2016-09-21T02:41:59Z standardize to chi2 commit a16a3fde844ba381dee7eb1e9608ddc2dcfb96fc Author: amaya Date: 2016-09-21T04:10:18Z refine chi2 commit abbf5492b95dd69e347580c59ac044a78627c547 Author: amaya Date: 2016-09-21T04:11:00Z refine transpose_and_dot commit b8cf39684496f2511e59294041d443b9438394a9 Author: amaya Date: 2016-09-21T06:02:12Z fix chi2 commit a882c5f9f8067b911254dfc43d268de06a5490f9 Author: amaya Date: 2016-09-21T07:00:36Z mod chi2 function name commit 5088ef36367df1cd51ae62f1c044933676975e2e Author: amaya Date: 2016-09-21T07:22:09Z add tests commit 22a608ee1c7239b2953183b5341f80c58b1e7045 Author: amaya Date: 2016-09-26T08:07:55Z add snr commit a1f8f958c99f3cde9e48b6d80d364004f6d98cc2 Author: amaya Date: 2016-09-27T06:58:33Z integrate chi2 and SNR into hivemall.spark commit aa7d5299739349b49ef4f50cc2c1969f5cb8a78f Author: amaya Date: 2016-09-27T07:02:02Z Merge 'master' into 'feature/feature_selection' commit 1347de985ea6f8028c9d381f8827882ad39ad3a7 Author: amaya Date: 2016-09-28T05:22:05Z refine feature selection in spark integration commit 8e2842cf8c272642feaa76bf95e8fa463b0322dc Author: amaya Date: 2016-09-28T05:24:19Z refine tests commit 4cfa4e5ac15a6535b187c23616c205696a1cd13b Author: amaya Date: 2016-09-28T09:26:01Z mod SNR for corner cases commit 80be81ecf92cd4675dcdfaa5f456d84d484d6c44 Author: amaya Date: 2016-09-28T11:01:08Z minor fix commit 8d9f0d4c00758324029d342eb4b892e046ca4a49 Author: amaya Date: 2016-09-29T02:02:14Z minor fix commit ce4a48980e33b9f16c74a62fcea6878f28b9c08b Author: amaya Date: 2016-09-30T08:05:20Z change method of testing for spark commit 67ba9631af3c231b7abd145134d17237b6aca0a5 Author: myui Date: 2016-11-21T09:19:45Z Merge branch 'feature/feature_selection' of https://github.com/amaya382/hivemall into feature_selection # Conflicts: # core/src/main/java/hivemall/utils/hadoop/HiveUtils.java # core/src/main/java/hivemall/utils/math/StatsUtils.java # spark/spark-1.6/src/main/scala/org/apache/spark/sql/hive/GroupedDataEx.scala # spark/spark-1.6/src/test/scala/org/apache/spark/sql/hive/HivemallOpsSuite.scala
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[
https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714425#comment-15714425
]
ASF GitHub Bot commented on HIVEMALL-22:
GitHub user myui opened a pull request:
https://github.com/apache/incubator-hivemall/pull/6
[WIP] Implement SST-based change-point detector
This PR is based on [a pending
PR](https://github.com/myui/hivemall/pull/356) by @takuti that is sent before
Hivemall entered Apache Incubator.
See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22
) for tracking the status of this issue.
---
### Sample table
| time | x |
| --: | --: |
| 1 | 182.478 |
| 2 | 176.231 |
| 3 | 183.917 |
| 4 | 177.798 |
| 5 | 165.469 |
| ... | ... |
(14398 points from [twitter
data](https://blog.twitter.com/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series))
### Usage
``` sql
create temporary function sst as
'hivemall.anomaly.SingularSpectrumTransformUDF';
```
``` sql
SELECT
time,
-- x is double or array
-- sst(x) AS res
sst(x, "-th 0.005") AS res
FROM
twitter_timeseries
ORDER BY time ASC
;
```
### Results
```
7551{"changepoint_score":0.00453049288071683,"is_changepoint":false}
7552{"changepoint_score":0.004711244102524104,"is_changepoint":false}
7553{"changepoint_score":0.004814871928978115,"is_changepoint":false}
7554{"changepoint_score":0.004968089640799422,"is_changepoint":false}
7555{"changepoint_score":0.005709056330104878,"is_changepoint":true}
7556{"changepoint_score":0.0044279766655132,"is_changepoint":false}
7557{"changepoint_score":0.0034694956722586268,"is_changepoint":false}
7558{"changepoint_score":0.002549056569322694,"is_changepoint":false}
7559{"changepoint_score":0.0017395109108403473,"is_changepoint":false}
7560{"changepoint_score":0.0010629833145070489,"is_changepoint":false}
```
On the naive SVD-based implementation, elapsed time was about 20 sec. for
the 14398 samples (vs. 10 sec on ChangeFinder)
### Observations
The change-point scores are much more stable compared to ChangeFinder, and
change-point scores are always in [0, 1]. However, since the scores are quite
noisy, too many change-points are detected. Smoothing scores like ChangeFinder
is practically important.
In terms of running time, the naive SVD-based implementation is clearly
inefficient. So, the Lanczos-based efficient variant should be supported.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-hivemall JIRA-22/pr-356
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/6.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6
commit 3ebd771ee4bebf14769b7c240f8b28b9d5d10e86
Author: Takuya Kitazawa
Date: 2016-09-26T08:12:01Z
Implement initial SST-based change-point detector
commit bde06e0952445bf60a9aef4bca182c0afe87e250
Author: Takuya Kitazawa
Date: 2016-09-27T05:06:20Z
Rename SSTChangePoint -> SingularSpectrumTransform
commit 2bfd1270b1e9b79185a41cbe2568f2ce968d4a71
Author: Takuya Kitazawa
Date: 2016-09-28T02:16:56Z
Add references for the original SST papers
commit 998203d5e8623d6282c2b187df24e4da7d41c16b
Author: Takuya Kitazawa
Date: 2016-09-28T10:49:48Z
Support implicit-Krylov-approximation-based efficient SST
commit cc34435155e86718acb49fa42208aff730bb756c
Author: myui
Date: 2016-12-02T07:55:23Z
Merge branch 'sst-changepoint' of https://github.com/takuti/hivemall into
JIRA-22/pr-356
> Review and merge pending Pull Requests before entering Incubator
>
>
> Key: HIVEMALL-22
> URL: https://issues.apache.org/jira/browse/HIVEMALL-22
> Project: Hivemall
> Issue Type: New Feature
>Reporter: Makoto Yui
>Assignee: Makoto Yui
>
> Need to review and merge pending Pull Requests in
> https://github.com/myui/hivemall/pulls
> * Feature Selection
> https://github.com/myui/hivemall/pull/385
> * SST change point detection
> https://github.com/myui/hivemall/pull/356
> * Checkstyle
> https://github.com/myui/hivemall/pull/343
> * System Test
> https://github.com/myui/hivemall/pull/336
> * Kernelized Passive Aggressive
> https://github.com/myui/hivemall/pull/304
> * Separate Optimizer
> https://github.com/myui/hivemall/pull/285
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714418#comment-15714418 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/5 Implement System Test This PR introduces System Tests of Hivemall. Usage and example are written in `systemtest/README.md` . This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/336) by @amaya382 that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-336 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/5.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5 commit f4d23a5daaa668fa7292a0ed90aaf58ef836c5c2 Author: amaya Date: 2016-08-23T09:19:53Z add feature for testing UDF commit 0b91aca3888b0a28b6538b314fd652c8a0ae65b5 Author: amaya Date: 2016-09-05T06:04:12Z add usage commit f55fe9f9191f2bed3022f1da1ae149c571c8db83 Author: amaya Date: 2016-09-05T09:36:32Z move systemtest into specific package commit 827cde18592e636eb7239eef4744b2d64856f029 Author: amaya Date: 2016-09-05T09:37:00Z mod project settings of systemtest commit cbb14da8db352d2817c9741afae884f803c31607 Author: amaya Date: 2016-09-05T09:46:18Z add ordered/unordered result matching commit 398be69eb72bbdd43d9e989512368e1935e4e0b0 Author: amaya Date: 2016-09-05T10:08:36Z add license header commit 1e82b3eb07be3b7719c8a81b8fb2d957534e45a0 Author: amaya Date: 2016-09-07T05:01:31Z enable to use external properties commit dd8f37dc07ba25682cc83eedef86b0ab6ae6800e Author: amaya Date: 2016-09-07T05:15:35Z update README about external properties commit aa9e9ff5c5dd8b4a5d0ce0e8e2cadd988aa9133b Author: amaya Date: 2016-09-07T06:07:38Z suppress warnings commit fbbe9fe947424a77936cd4332162e9a10fd1 Author: amaya Date: 2016-09-07T06:33:25Z potential bug to exception commit 09af57a0fea6205bc619e45363309b8b69a76634 Author: amaya Date: 2016-09-07T08:04:38Z add ordered/unordered result matching for multiple queries commit 443c81788c93a1aa46699fdad2e981afcb18a0ae Author: amaya Date: 2016-09-07T08:55:57Z refine README commit f84eafc41fa15b3810320dda5a644cf8d6df3637 Author: amaya Date: 2016-09-07T09:42:25Z Merge branch 'master' into 'feature/systemtest' resolved conflict pom.xml commit 265d910d5dbe06d839c7a190018021c0745b5baa Author: amaya Date: 2016-09-08T08:43:30Z refine README commit 45cec9e72fb41b4f97e9fca0f1348d187cda9d35 Author: amaya Date: 2016-09-09T05:46:27Z support array and map in insert statement commit 2d8c794b07a5f06077817dafee868e91a5e36393 Author: amaya Date: 2016-09-09T05:49:35Z fix error messages commit 6e6512a81132c4b7e5d8e1d0e01b99cfe440b4cd Author: amaya Date: 2016-09-12T06:12:32Z make code tougher commit d13dc1125cfd81bbe6ef49c24b4c9f0ca4343e8b Author: amaya Date: 2016-09-12T06:22:45Z avoid running with no runner commit e823eb240a1a2ca10802f67e92d812e90be4647f Author: amaya Date: 2016-09-13T07:03:32Z refresh null check commit faebaf9b9082375dea8a53d3dd5b3857eee3a458 Author: amaya Date: 2016-09-13T07:26:56Z mod name and inheritance structure commit 43ca0c86936f3ccc7f825db3c4f4ecaa48087917 Author: amaya Date: 2016-11-16T06:23:49Z Update license headers commit ba91267796cbfdee53aaef02af882aff591fb8f7 Author: amaya Date: 2016-11-17T05:15:03Z Mod README commit 33eab26f383dbdbce00a209e742b611a63d953cf Author: amaya Date: 2016-11-17T05:16:14Z Add exception commit 1f3df54c0183a61390f58b94f58c12e531754a09 Author: amaya Date: 2016-11-17T16:57:31Z Make dir name static commit 3550fd30af3a01f4217c075a3b814952b406aebe Author: amaya Date: 2016-11-17T16:57:47Z Mod assert methods commit 144cb504d674d2509620ce0d315694be0f664f42 Author: amaya Date: 2016-11-17T16:58:31Z Fix process of tdprop commit 7447dde61f3a9cb8e3ba5ab278a260d0a0615524 Author: amaya Date: 2016-11-17T18:23:46Z Mod README commit ddd8dc2dbf8222c9d9d84b038dbdcd9aef1f1a87 Author: amaya Date: 2016-11-17T19:22:51Z Refine access modifiers/calls commit 798ec6a73ca37d474137fc82db1c22a92521307d Author: amaya Date: 2016-11-17T19:27:59Z Update license header commit 075f93485e41ac8f26451f68b0f38737849b04a7 Author: myui Date: 2016-12-02T07:55:34Z Merge branch 'feature/systemtest' of https://github.com/amaya382/hivemall i
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714414#comment-15714414 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/4 [WIP] Kernelized Passive-Aggressive Algorithm for Classification This PR introduce the following algorithms: - Passive-Aggressive with Polynomial Kernel - Polynomial Kernel with Inverted Indices (PKI) - Kernel Expansion - PA-I, PA-II This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/304) by @L3Sota that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-304 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/4.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4 commit 4cb84773f4c88f757d70b480a80068d597e756a7 Author: myui Date: 2016-06-16T22:59:11Z Changed the behavior of binarize_label commit adb23e3c537d9c2d2ce36c27700a85ff6852964b Author: Sotaro Sugimoto Date: 2016-04-12T06:23:12Z Added Kernelized Passive-Aggressive Algorithm for Classification Problems. commit bc228746c7f7b1abc0f6e7846d52ca1edad7fea7 Author: Sotaro Sugimoto Date: 2016-04-12T09:14:45Z Bug fixes Also newly added KernelizedPassiveAggressiveUDTFTest based on PassiveAggressiveUDTFTest, but tests are not done for newly added functionality. commit adac4e5bd0d89cc8c68768c9fb5bc1765bc3fac3 Author: Sotaro Sugimoto Date: 2016-04-14T09:51:14Z Test for news20 written; some bugs fixed. Bug still occurs for news20-small.binary where the loss does not match between udtf and udtfPKI starting at sample 51, where the label switches from all -1s to all +1s. Also, news20.binary not included due to large filesize. commit 6072a2ae7f1d0f8a0023dbe428a38c201386e38e Author: myui Date: 2016-04-14T11:51:57Z Refactored to use Bitset for support vector indices commit ecf4f3ab55ccdbe7c57bcd21b292e485465fb566 Author: myui Date: 2016-04-14T11:55:49Z Changed resource location of unit tests commit 3994cc7bb706e0b073278176eee6a1a5b78bebf6 Author: Sotaro Sugimoto Date: 2016-04-15T06:17:18Z Both normal and PKI versions of KPA now learn identically. Note that testTime was added to see how much faster PKI is, but it appears that PKI is actually slower than normal KPA for non-sparse support vectors, since PKI must go through the work of finding all matching support vectors, which is O(number of support vectors) for a completely full match. commit 31d981bd0082d4d5c40d7eb536f47f70455a3c52 Author: Sotaro Sugimoto Date: 2016-04-15T09:38:49Z Fixed incorrect alpha calculation/storage. Also fixed tests and asserts to match correct alpha calculations. commit e2812fd035f51877167f0c77aff397fa45be0bef Author: Sotaro Sugimoto Date: 2016-04-15T09:59:16Z Added FloatArrayList. Also changed alpha from DoubleArrayList to FloatArrayList. commit c793d47e0de18d32a56d2d1fd1744470f7283613 Author: Sotaro Sugimoto Date: 2016-04-20T03:14:12Z Added kernel expansion option (untested) commit 4716b34ad5038761d18e422833d9334140dd0908 Author: Sotaro Sugimoto Date: 2016-04-21T03:34:26Z Tests for KPA added. Note that kernel splitting (in KernelExpansionKPA) is incomplete. commit f986803901824ad831c7c8a2504c26797d765f23 Author: Sotaro Sugimoto Date: 2016-04-26T04:21:22Z Kernel Expansion complete with tests. Also removed Kernel Splitting (probably uses too much memory). commit b0a0179b0bc1f50403eb1f5534bfb870113f9777 Author: myui Date: 2016-12-02T07:55:49Z Merge branch 'feature/kernelized_pa' of https://github.com/L3Sota/hivemall into JIRA-22/pr-304 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714402#comment-15714402 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/3 [WIP] Separate optimizer implementations This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/285) by @maropu that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-285 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/3.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3 commit f81948c5c7b83155eb29369a59f1fc65bb607f91 Author: Takeshi YAMAMURO Date: 2016-05-02T14:43:42Z Add optimizer implementations commit 3620eb89993db22ce8aee924d3cc0df33a5f9618 Author: Takeshi YAMAMURO Date: 2016-09-20T16:52:22Z Revert some modifications commit 9ca8bce75333a3ebb43d2492bf5df815ccf869dc Author: myui Date: 2016-12-02T07:56:40Z Merge branch 'AddOptimizers' of https://github.com/maropu/hivemall into JIRA-22/pr-285 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714320#comment-15714320 ] Kai Sasaki commented on HIVEMALL-22: @myu Thanks I succeeded to create PR from JIRA-22/* branch. PRs are already integrated to JIRA? Looks like from above ASF GitHub Bot. https://github.com/myui/hivemall/pull/336 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714309#comment-15714309 ] ASF GitHub Bot commented on HIVEMALL-22: Github user myui closed the pull request at: https://github.com/apache/incubator-hivemall/pull/1 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST change point detection > https://github.com/myui/hivemall/pull/356 > * Checkstyle > https://github.com/myui/hivemall/pull/343 > * System Test > https://github.com/myui/hivemall/pull/336 > * Kernelized Passive Aggressive > https://github.com/myui/hivemall/pull/304 > * Separate Optimizer > https://github.com/myui/hivemall/pull/285 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVEMALL-22) Review and merge pending Pull Requests before entering Incubator
[ https://issues.apache.org/jira/browse/HIVEMALL-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714306#comment-15714306 ] ASF GitHub Bot commented on HIVEMALL-22: GitHub user myui opened a pull request: https://github.com/apache/incubator-hivemall/pull/1 [WIP] Separate optimizer implementations This PR is based on [a pending PR](https://github.com/myui/hivemall/pull/285) by @maropu that is sent before Hivemall entered Apache Incubator. See [JIRA](https://issues.apache.org/jira/browse/HIVEMALL-22 ) for tracking the status of this issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/myui/incubator-hivemall JIRA-22/pr-285 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hivemall/pull/1.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1 commit 4cb84773f4c88f757d70b480a80068d597e756a7 Author: myui Date: 2016-06-16T22:59:11Z Changed the behavior of binarize_label commit adb23e3c537d9c2d2ce36c27700a85ff6852964b Author: Sotaro Sugimoto Date: 2016-04-12T06:23:12Z Added Kernelized Passive-Aggressive Algorithm for Classification Problems. commit bc228746c7f7b1abc0f6e7846d52ca1edad7fea7 Author: Sotaro Sugimoto Date: 2016-04-12T09:14:45Z Bug fixes Also newly added KernelizedPassiveAggressiveUDTFTest based on PassiveAggressiveUDTFTest, but tests are not done for newly added functionality. commit adac4e5bd0d89cc8c68768c9fb5bc1765bc3fac3 Author: Sotaro Sugimoto Date: 2016-04-14T09:51:14Z Test for news20 written; some bugs fixed. Bug still occurs for news20-small.binary where the loss does not match between udtf and udtfPKI starting at sample 51, where the label switches from all -1s to all +1s. Also, news20.binary not included due to large filesize. commit 6072a2ae7f1d0f8a0023dbe428a38c201386e38e Author: myui Date: 2016-04-14T11:51:57Z Refactored to use Bitset for support vector indices commit ecf4f3ab55ccdbe7c57bcd21b292e485465fb566 Author: myui Date: 2016-04-14T11:55:49Z Changed resource location of unit tests commit 3994cc7bb706e0b073278176eee6a1a5b78bebf6 Author: Sotaro Sugimoto Date: 2016-04-15T06:17:18Z Both normal and PKI versions of KPA now learn identically. Note that testTime was added to see how much faster PKI is, but it appears that PKI is actually slower than normal KPA for non-sparse support vectors, since PKI must go through the work of finding all matching support vectors, which is O(number of support vectors) for a completely full match. commit 31d981bd0082d4d5c40d7eb536f47f70455a3c52 Author: Sotaro Sugimoto Date: 2016-04-15T09:38:49Z Fixed incorrect alpha calculation/storage. Also fixed tests and asserts to match correct alpha calculations. commit e2812fd035f51877167f0c77aff397fa45be0bef Author: Sotaro Sugimoto Date: 2016-04-15T09:59:16Z Added FloatArrayList. Also changed alpha from DoubleArrayList to FloatArrayList. commit c793d47e0de18d32a56d2d1fd1744470f7283613 Author: Sotaro Sugimoto Date: 2016-04-20T03:14:12Z Added kernel expansion option (untested) commit 4716b34ad5038761d18e422833d9334140dd0908 Author: Sotaro Sugimoto Date: 2016-04-21T03:34:26Z Tests for KPA added. Note that kernel splitting (in KernelExpansionKPA) is incomplete. commit f986803901824ad831c7c8a2504c26797d765f23 Author: Sotaro Sugimoto Date: 2016-04-26T04:21:22Z Kernel Expansion complete with tests. Also removed Kernel Splitting (probably uses too much memory). commit f81948c5c7b83155eb29369a59f1fc65bb607f91 Author: Takeshi YAMAMURO Date: 2016-05-02T14:43:42Z Add optimizer implementations commit 3620eb89993db22ce8aee924d3cc0df33a5f9618 Author: Takeshi YAMAMURO Date: 2016-09-20T16:52:22Z Revert some modifications commit 775ae4f79466265e703ec1f3a03cf6ce77459087 Author: myui Date: 2016-12-02T06:33:32Z Merge branch 'feature/kernelized_pa' of https://github.com/L3Sota/hivemall into JIRA-22/pr-304 commit 05766432c45f89627e423245e5aec3ced6d0c100 Author: myui Date: 2016-12-02T06:35:05Z Merge branch 'AddOptimizers' of https://github.com/maropu/hivemall into JIRA-22/pr-285 > Review and merge pending Pull Requests before entering Incubator > > > Key: HIVEMALL-22 > URL: https://issues.apache.org/jira/browse/HIVEMALL-22 > Project: Hivemall > Issue Type: New Feature >Reporter: Makoto Yui >Assignee: Makoto Yui > > Need to review and merge pending Pull Requests in > https://github.com/myui/hivemall/pulls > * Feature Selection > https://github.com/myui/hivemall/pull/385 > * SST
