[
https://issues.apache.org/jira/browse/HIVEMALL-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821735#comment-16821735
]
ASF GitHub Bot commented on HIVEMALL-251:
-----------------------------------------
myui commented on pull request #191: [HIVEMALL-251] Add option to return
PartOfSpeech information for tokenize_ja
URL: https://github.com/apache/incubator-hivemall/pull/191
## What changes were proposed in this pull request?
Add option to return PartOfSpeech information for `tokenize_ja` UDF.
## What type of PR is it?
Feature, Improvement
## What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-251
## How was this patch tested?
unit tests and manual tests on EMR
## How to use this feature?
```sql
WITH tmp as (
select
tokenize_ja('kuromojiを使った分かち書きのテストです。','-mode search -pos') as r
)
select
r.tokens,
r.pos,
r.tokens[0] as token0,
r.pos[0] as pos0
from
tmp;
```
| tokens |pos | token0 | pos0 |
|:-:|:-:|:-:|:-:|
| ["kuromoji","使う","分かち書き","テスト"] | ["名詞-一般","動詞-自立","名詞-一般","名詞-サ変接続"] |
kuromoji | 名詞-一般 |
## Checklist
- [x] Did you apply source code formatter, i.e., `./bin/format_code.sh`, for
your commit?
- [x] Did you run system tests on Hive (or Spark)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add option to return PartOfSpeech information for tokenize_ja
> -------------------------------------------------------------
>
> Key: HIVEMALL-251
> URL: https://issues.apache.org/jira/browse/HIVEMALL-251
> Project: Hivemall
> Issue Type: Improvement
> Affects Versions: 0.5.2
> Reporter: Makoto Yui
> Assignee: Makoto Yui
> Priority: Minor
> Fix For: 0.6.0
>
>
> tokenize_ja should return PartOfSpeech information iff "-pos" option is
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
