myui opened a new pull request #191: [HIVEMALL-251] Add option to return PartOfSpeech information for tokenize_ja URL: https://github.com/apache/incubator-hivemall/pull/191 ## What changes were proposed in this pull request? Add option to return PartOfSpeech information for `tokenize_ja` UDF. ## What type of PR is it? Feature, Improvement ## What is the Jira issue? https://issues.apache.org/jira/browse/HIVEMALL-251 ## How was this patch tested? unit tests and manual tests on EMR ## How to use this feature? ```sql WITH tmp as ( select tokenize_ja('kuromojiを使った分かち書きのテストです。','-mode search -pos') as r ) select r.tokens, r.pos, r.tokens[0] as token0, r.pos[0] as pos0 from tmp; ``` | tokens |pos | token0 | pos0 | |:-:|:-:|:-:|:-:| | ["kuromoji","使う","分かち書き","テスト"] | ["名詞-一般","動詞-自立","名詞-一般","名詞-サ変接続"] | kuromoji | 名詞-一般 | ## Checklist - [x] Did you apply source code formatter, i.e., `./bin/format_code.sh`, for your commit? - [x] Did you run system tests on Hive (or Spark)?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
