[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778597#comment-13778597
]
Hudson commented on HIVE-5279:
--
FAILURE: Integrated in Hive-trunk-h0.21 #2358 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/2358/])
HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis
via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526117)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
*
/hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
* /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
* /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
* /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Fix For: 0.13.0
Attachments: 5279.patch, D12963.1.patch, D12963.2.patch,
D12963.3.patch, D12963.4.patch, D12963.5.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778624#comment-13778624
]
Hudson commented on HIVE-5279:
--
ABORTED: Integrated in Hive-trunk-hadoop2 #457 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/457/])
HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis
via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526117)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
*
/hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
* /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
* /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
* /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Fix For: 0.13.0
Attachments: 5279.patch, D12963.1.patch, D12963.2.patch,
D12963.3.patch, D12963.4.patch, D12963.5.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777395#comment-13777395
]
Hudson commented on HIVE-5279:
--
FAILURE: Integrated in Hive-trunk-hadoop1-ptest #181 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/181/])
HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis
via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526117)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
*
/hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
* /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
* /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
* /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Fix For: 0.13.0
Attachments: 5279.patch, D12963.1.patch, D12963.2.patch,
D12963.3.patch, D12963.4.patch, D12963.5.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777411#comment-13777411
]
Hudson commented on HIVE-5279:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #115 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/115/])
HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis
via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526117)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
*
/hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
* /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
* /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
* /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Fix For: 0.13.0
Attachments: 5279.patch, D12963.1.patch, D12963.2.patch,
D12963.3.patch, D12963.4.patch, D12963.5.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769582#comment-13769582
]
Ashutosh Chauhan commented on HIVE-5279:
With latest patch tests in TestCliDriver TestContribCliDriver tests are no
more failing, so thats a progress. However, 4 tests in TestParse are still
failing, as previously, namely {{groupby1.q}} {{groupby2.q}} {{groupby3.q}} and
{{groupby5.q}}
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: 5279.patch, D12963.1.patch, D12963.2.patch,
D12963.3.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769989#comment-13769989
]
Leo Romanoff commented on HIVE-5279:
Regarding the issue with Kryo's exception about no-arg constructor:
Please read this section: https://code.google.com/p/kryo/#Object_creation
Basically, you can force Kryo to create objects without invoking any
constructor at all, but you should be careful with it and your class should be
written in such a way that it can handle it.
-Leo
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: 5279.patch, D12963.1.patch, D12963.2.patch,
D12963.3.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768123#comment-13768123
]
Phabricator commented on HIVE-5279:
---
ashutoshc has accepted the revision HIVE-5279 [jira] Kryo cannot instantiate
GenericUDAFEvaluator in GroupByDesc.
+1
REVISION DETAIL
https://reviews.facebook.net/D12963
BRANCH
HIVE-5279
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch, D12963.1.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768692#comment-13768692
]
Ashutosh Chauhan commented on HIVE-5279:
Additionally following tests failed too:
* TestContribCliDriver_udaf_example_avg.q
* TestContribCliDriver_udaf_example_group_concat.q
* TestContribCliDriver__udaf_example_max.q
* TestContribCliDriver_udaf_example_max_n.q
* TestContribCliDriver__udaf_example_min.q
* TestContribCliDriver__udaf_example_max_n.q
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: 5279.patch, D12963.1.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769056#comment-13769056
]
Navis commented on HIVE-5279:
-
Oh, I'll check that.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: 5279.patch, D12963.1.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767995#comment-13767995
]
Navis commented on HIVE-5279:
-
Ok, sure.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768000#comment-13768000
]
Edward Capriolo commented on HIVE-5279:
---
I just ran into this making the collect_list udaf, you need to do things
java-bean style. No-arge constructor + setter and getter. It probably was valid
before but it is not now.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768058#comment-13768058
]
Navis commented on HIVE-5279:
-
The point is we've not required anything for implementation of UDAFs till now.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766088#comment-13766088
]
Navis commented on HIVE-5279:
-
I've applied the patch and confirmed the query working. But,
reflectionUtils.newInstance() uses default constructor, which means it forces
UDAFEvaluator to have it (like GenericUDAFGroupOn$VersionedFloatGroupOnEval in
this case). I'm ok to make this to be a requirement of UDAFEvaluator. Other
ideas?
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766120#comment-13766120
]
Ashutosh Chauhan commented on HIVE-5279:
I think Kryo has other mechanisms of serialization which doesn't have that
requirement. I haven't explored Kryo fully yet. Probably, we should do that in
a follow-up. I think for now, we can get this one in and have this requirement
in the meanwhile.
[~navis] Since you have that custom udf, can you write a test case for this and
submit to let Hive QA run on the patch.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765388#comment-13765388
]
Ashutosh Chauhan commented on HIVE-5279:
I think this can be fixed by making following changes in AggregationDesc class:
1. Mark genericUDAFEvaluator as transient.
2. Store the class name of GenericUDAFEvaluator as string.
3. In getGenericUDAFEvaluator(), using reflection on className string,
instantiate GenericUDAFEvaluator and return it.
Since none of the callers of getGenericUDAFEvaluator make any use of state of
genericUDAFEvaluator, this should work. Can you give it a try?
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765406#comment-13765406
]
Navis commented on HIVE-5279:
-
I considered that. But the immediate question is, was it valid to use
UDAFEvaluator without default constructor before? I should check that, in
tomorrow.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765418#comment-13765418
]
Ashutosh Chauhan commented on HIVE-5279:
It was valid to use UDAFEvaluator without default ctor earlier on jdk6. Since,
java xml serialization was lenient on that. However, on jdk7 you are required
to provide a no-arg ctor even for java xml serialization. So, even without
Kryo, you would run into this problem, once Hive moves to jdk7.
However, patch I have uploaded eliminates this requirement altogether, since
genericUDAFEvaluator (user provided class) is no longer stored as field in
AggregatorDesc. So, if this patch works it won't matter if you have default
ctor or not either for Kryo or jdk6 or jdk7.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765445#comment-13765445
]
Brock Noland commented on HIVE-5279:
Interesting the tests didn't catch this? How did you run into it?
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5279) Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
[
https://issues.apache.org/jira/browse/HIVE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765463#comment-13765463
]
Ashutosh Chauhan commented on HIVE-5279:
Its a custom UDAFEvaluator {{GenericUDAFGroupOn}} which for some reason
contains ObjectInspector without a no-arg constructor. Probably, none of the
evaluator which we have in our tests have this. Since, evaluator is used at
runtime, so it needs to be reconstructed anyways at backend. See,
GroupByOperator::initializeOp(). Hence there is no reason good reason to
serialize the constructed instance of it from frontend to backend.
Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc
---
Key: HIVE-5279
URL: https://issues.apache.org/jira/browse/HIVE-5279
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Priority: Critical
Attachments: 5279.patch
We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how
previous serialization mechanism solved this but, kryo complaints that it's
not Serializable and fails the query.
The log below is the example,
{noformat}
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class
cannot be created (missing no-arg constructor):
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
Serialization trace:
inputOI
(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
at org.apache.h
{noformat}
If this cannot be fixed in somehow, some UDAFs should be modified to be run
on hive-0.13.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
