Jacob Shaw created HIVEMALL-255:
-----------------------------------
Summary: Error: java.lang.RuntimeException: Hive Runtime Error
while closing operators
Key: HIVEMALL-255
URL: https://issues.apache.org/jira/browse/HIVEMALL-255
Project: Hivemall
Issue Type: Bug
Affects Versions: 0.5.2
Environment: Ubuntu 16.04 4.4.0-146-generic
Hadoop 3.1.2
Hivemall 0.52
Reporter: Jacob Shaw
Hello when attempting to use hive-malls regression tool kit I run into errors
when attempting to build the feature representation.
I've been following this guide
[https://hivemall.incubator.apache.org/userguide/supervised_learning/tutorial.html]
and have been attempting to reproduce it. I've used the code provided however
I'm running into issues when running.
My issue seems to be with this part of the guide
{color:#569cd6}create table if not exists purchase_history{color}
{color:#c586c0}as{color}
{color:#c586c0}select{color} {color:#b5cea8}1{color}
{color:#c586c0}as{color}{color:#d4d4d4} id,
{color}{color:#ce9178}"Saturday"{color} {color:#c586c0}as{color}{color:#d4d4d4}
day_of_week, {color}{color:#ce9178}"male"{color}
{color:#c586c0}as{color}{color:#d4d4d4} gender,
{color}{color:#b5cea8}600{color} {color:#c586c0}as{color}{color:#d4d4d4} price,
{color}{color:#ce9178}"book"{color} {color:#c586c0}as{color}{color:#d4d4d4}
category, {color}{color:#b5cea8}1{color}
{color:#c586c0}as{color}{color:#d4d4d4} label{color}
{color:#c586c0}union{color} {color:#c586c0}all{color}
{color:#c586c0}select{color} {color:#b5cea8}2{color}
{color:#c586c0}as{color}{color:#d4d4d4} id,
{color}{color:#ce9178}"Friday"{color} {color:#c586c0}as{color}{color:#d4d4d4}
day_of_week, {color}{color:#ce9178}"female"{color}
{color:#c586c0}as{color}{color:#d4d4d4} gender,
{color}{color:#b5cea8}4800{color} {color:#c586c0}as{color}{color:#d4d4d4}
price, {color}{color:#ce9178}"sports"{color}
{color:#c586c0}as{color}{color:#d4d4d4} category,
{color}{color:#b5cea8}0{color} {color:#c586c0}as{color}{color:#d4d4d4}
label{color}
{color:#c586c0}union{color} {color:#c586c0}all{color}
{color:#c586c0}select{color} {color:#b5cea8}3{color}
{color:#c586c0}as{color}{color:#d4d4d4} id,
{color}{color:#ce9178}"Friday"{color} {color:#c586c0}as{color}{color:#d4d4d4}
day_of_week, {color}{color:#ce9178}"other"{color}
{color:#c586c0}as{color}{color:#d4d4d4} gender,
{color}{color:#b5cea8}18000{color} {color:#c586c0}as{color}{color:#d4d4d4}
price, {color}{color:#ce9178}"entertainment"{color}
{color:#c586c0}as{color}{color:#d4d4d4} category,
{color}{color:#b5cea8}0{color} {color:#c586c0}as{color}{color:#d4d4d4}
label{color}
{color:#c586c0}union{color} {color:#c586c0}all{color}
{color:#c586c0}select{color} {color:#b5cea8}4{color}
{color:#c586c0}as{color}{color:#d4d4d4} id,
{color}{color:#ce9178}"Thursday"{color} {color:#c586c0}as{color}{color:#d4d4d4}
day_of_week, {color}{color:#ce9178}"male"{color}
{color:#c586c0}as{color}{color:#d4d4d4} gender,
{color}{color:#b5cea8}200{color} {color:#c586c0}as{color}{color:#d4d4d4} price,
{color}{color:#ce9178}"food"{color} {color:#c586c0}as{color}{color:#d4d4d4}
category, {color}{color:#b5cea8}0{color}
{color:#c586c0}as{color}{color:#d4d4d4} label{color}
{color:#c586c0}union{color} {color:#c586c0}all{color}
{color:#c586c0}select{color} {color:#b5cea8}5{color}
{color:#c586c0}as{color}{color:#d4d4d4} id,
{color}{color:#ce9178}"Wednesday"{color}
{color:#c586c0}as{color}{color:#d4d4d4} day_of_week,
{color}{color:#ce9178}"female"{color} {color:#c586c0}as{color}{color:#d4d4d4}
gender, {color}{color:#b5cea8}1000{color}
{color:#c586c0}as{color}{color:#d4d4d4} price,
{color}{color:#ce9178}"electronics"{color}
{color:#c586c0}as{color}{color:#d4d4d4} category,
{color}{color:#b5cea8}1{color} {color:#c586c0}as{color}{color:#d4d4d4}
label{color}
{color:#d4d4d4};{color}
{color:#569cd6}create table if not exists training{color}
{color:#c586c0}as{color}
{color:#c586c0}select{color}
{color:#d4d4d4} id,{color}
{color:#d4d4d4} array_concat( {color}{color:#6a9955}-- concatenate two arrays
of quantitative and categorical features into single array{color}
{color:#d4d4d4} quantitative_features({color}
{color:#4ec9b0}array{color}{color:#d4d4d4}({color}{color:#ce9178}"price"{color}{color:#d4d4d4}),
{color}{color:#6a9955}-- quantitative feature names{color}
{color:#d4d4d4} price {color}{color:#6a9955}-- corresponding column names{color}
{color:#d4d4d4} ),{color}
{color:#d4d4d4} categorical_features({color}
{color:#4ec9b0}array{color}{color:#d4d4d4}({color}{color:#ce9178}"day of
week"{color}{color:#d4d4d4},
{color}{color:#ce9178}"gender"{color}{color:#d4d4d4},
{color}{color:#ce9178}"category"{color}{color:#d4d4d4}),
{color}{color:#6a9955}-- categorical feature names{color}
{color:#d4d4d4} day_of_week, gender, category {color}{color:#6a9955}--
corresponding column names{color}
{color:#d4d4d4} ){color}
{color:#d4d4d4} ) {color}{color:#c586c0}as{color}{color:#d4d4d4}
features,{color}
{color:#d4d4d4} label{color}
{color:#c586c0}from{color}
{color:#d4d4d4} purchase_history{color}
{color:#d4d4d4};{color}
This is copied straight from the guide. When running I am getting this error
:24,657 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2019-05-16
10:09:24,692 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Using
serializer : class
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe[[[B@43b0ade]:[id, features,
label]:[int, array<string>, int]] and formatter :
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat@2d66530f 2019-05-16
10:09:24,692 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
mapred.healthChecker.script.timeout is deprecated. Instead, use
mapreduce.tasktracker.healthchecker.script.timeout 2019-05-16 10:09:24,706 INFO
[main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH =
hdfs://localhost:9000/tmp/hive/jshaw6/970622c3-bfd6-407c-93f9-953184696ebf/hive_2019-05-16_10-09-11_357_6286825630727418123-1/-mr-10006/3f3f0199-3af0-40dd-abb4-6bad4df12ba7/map.xml
2019-05-16 10:09:24,745 ERROR [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing operators
- failing tree 2019-05-16 10:09:24,746 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
java.lang.RuntimeException: Hive Runtime Error while closing operators at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at
org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating id at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:966) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:990)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:733) at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9
more Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.exec.UDFArgumentException: argument must be a
constant value: array<string> at
org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:106)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:271)
at
org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:111)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
... 15 more Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException:
argument must be a constant value: array<string> at
hivemall.utils.hadoop.HiveUtils.getConstStringArray(HiveUtils.java:502) at
hivemall.ftvec.trans.QuantitativeFeaturesUDF.initialize(QuantitativeFeaturesUDF.java:80)
at
org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.init(VectorUDFAdaptor.java:89)
at
org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:104)
... 18 more
However when I run the same query alone, without creating a table, I get the
right results.
1 ["price:600.0","day of week#Saturday","gender#male","category#book"] 1
2 ["price:4800.0","day of week#Friday","gender#female","category#sports"]0
3 ["price:18000.0","day of
week#Friday","gender#other","category#entertainment"] 0
4 ["price:200.0","day of week#Thursday","gender#male","category#food"] 0
5 ["price:1000.0","day of
week#Wednesday","gender#female","category#electronics"] 1
Any idea why I am not able to save this information in a table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
