[
https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772356#comment-16772356
]
Mithun Radhakrishnan commented on HIVE-21000:
---------------------------------------------
[~isuller], [~kgyrtkirk], [~vihangk1], thanks so much for working on this one.
There's a chance that after porting to Thrift 0.12, we might run into noisy
logging, stemming from load-balancer health-checks. I've documented the
failures that I ran into (Hive 1.2.x + Thrift 0.12) at THRIFT-4805:
{noformat}
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
at
org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:43)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.hive.service.cli.thrift.ThriftCLIMetricsProcessor.process(ThriftCLIMetricsProcessor.java:63)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$2.run(HadoopThriftAuthBridge.java:777)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$2.run(HadoopThriftAuthBridge.java:773)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:773)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{noformat}
With Thrift 0.9.x, the {{TSaslTransportException}} was trapped and handled
separately.
With Thrift 0.12.x, it's being caught and logged as a {{TException}}. (It
doesn't seem to be wrapped in a {{RuntimeException}}). You're likely to see the
above fill up the metastore server logs.
I have also posted a tentative fix (which we're currently testing):
https://github.com/apache/thrift/pull/1746
Just a heads-up.
> Upgrade thrift to at least 0.10.0
> ---------------------------------
>
> Key: HIVE-21000
> URL: https://issues.apache.org/jira/browse/HIVE-21000
> Project: Hive
> Issue Type: Improvement
> Reporter: Zoltan Haindrich
> Assignee: Ivan Suller
> Priority: Major
> Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch,
> HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch,
> HIVE-21000.06.patch, sampler_before.png
>
>
> I was looking into some compile profiles for tables with lots of columns; and
> it turned out that [thrift 0.9.3 is allocating a
> List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348]
> during every hashcode calculation; but luckily THRIFT-2877 is improving on
> that - so I propose to upgrade to at least 0.10.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
