After applying the patch and with Hcatalog 0.4.1 + libthrift-0.8.0, the memory issue got resolved. However, we get the following exception
Any pointers would be of great help (we tried applying https://issues.apache.org/jira/browse/THRIFT-1447 which doesn't work with Hive) 2012-09-03 00:01:56,395 ERROR server.TThreadPoolServer (TThreadPoolServer.java:run(182)) - Error occurred during processing of message. java.lang.NullPointerException at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:34) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:519) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:516) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:516) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2012-09-03 02:17:46,051 ERROR server.TThreadPoolServer (TThreadPoolServer.java:run(182)) - Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:561) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:559) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1159) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:559) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 10 more ~Rajesh.B On Tue, Aug 14, 2012 at 5:54 AM, Rajesh Balamohan < rajesh.balamo...@gmail.com> wrote: > Thanks Rohini, we are planning to apply this patch on same lines. > > ~Rajesh.B > > > On Tue, Aug 14, 2012 at 12:24 AM, Rohini Palaniswamy < > rohini.adi...@gmail.com> wrote: > >> Internally in Yahoo we use a patched library of thrift in our deployments >> to solve this issue. Need to get hive to move to the libthrift 0.9 that has >> the fix for THRIFT-1468<https://issues.apache.org/jira/browse/THRIFT-1468> >> >> Regards, >> Rohini >> >> >> On Wed, Aug 8, 2012 at 8:21 PM, Rajesh Balamohan < >> rajesh.balamo...@gmail.com> wrote: >> >>> Thanks for the pointers Alan. I believe I need to apply both Hive-3008 >>> and the thrift-1468. >>> >>> Analyzing heapdump points 2 GB of space occupied by >>> TSaslServerTransport. I do not see TUGIContainingTransport occupying >>> memory. >>> >>> ~Rajesh.B >>> >>> >>> On Thu, Aug 9, 2012 at 12:25 AM, Alan Gates <ga...@hortonworks.com>wrote: >>> >>>> This looks like it may be related to HIVE-3008. I've tested back >>>> porting this to Hive 0.9. I don't know how easy it would be to backport it >>>> to Hive 0.8. >>>> >>>> Alan. >>>> >>>> On Aug 8, 2012, at 7:33 AM, Rajesh Balamohan wrote: >>>> >>>> > I took the heapdump and analyzed it. It stems from >>>> TsaslServerTransport WeakHahsMap. >>>> > >>>> > I gues, its related to >>>> https://issues.apache.org/jira/browse/THRIFT-1468. >>>> > >>>> > Are there any fixes or workarounds for this issue in hive 0.8 itself >>>> (as we wouldn't be able to move to Hive 0.9 immediately). >>>> > >>>> > Any pointers would be of great help. >>>> > >>>> > ~Rajesh.B >>>> > >>>> > On Wed, Aug 8, 2012 at 7:56 PM, Rajesh Balamohan < >>>> rajesh.balamo...@gmail.com> wrote: >>>> > Hi All, >>>> > >>>> > I am currently using HCatalog 0.4 with PIG 0.9.2. >>>> > >>>> > After 3 or 4 days, I encountered PIG exceptions and digging deep I >>>> found that it tries to connect to HCatalog server and gets socket read >>>> timeout. >>>> > >>>> > Looking at the JVM process of Hcatalog server suggest OOM. I have >>>> attached the jstat details for further reference. >>>> > >>>> > At this point in time, I am not sure which usecase causes this OOM. I >>>> would post more details as and when I find. Should I raise a JIRA for >>>> this?. Plz suggest. >>>> > >>>> > [rajesh@machineXYZ hcatalog]$ sudo /usr/java/jdk1.6.0_25/bin/jstat >>>> -gcutil 7279 250 9 >>>> > S0 S1 E O P YGC YGCT FGC FGCT >>>> GCT >>>> > 0.00 0.00 71.38 99.99 59.76 33781 478.507 6095 2358.672 >>>> 2837.179 >>>> > 0.00 0.00 71.38 99.99 59.76 33781 478.507 6095 2358.672 >>>> 2837.179 >>>> > 0.00 0.00 71.38 99.99 59.76 33781 478.507 6095 2358.672 >>>> 2837.179 >>>> > 0.00 0.00 71.38 99.99 59.76 33781 478.507 6095 2358.672 >>>> 2837.179 >>>> > 0.00 0.00 71.38 99.99 59.76 33781 478.507 6095 2358.672 >>>> 2837.179 >>>> > 0.00 0.00 71.38 99.99 59.76 33781 478.507 6095 2358.672 >>>> 2837.179 >>>> > 0.00 0.00 71.64 99.99 59.76 33781 478.507 6095 2358.672 >>>> 2837.179 >>>> > 0.00 0.00 71.64 99.99 59.76 33781 478.507 6096 2358.712 >>>> 2837.219 >>>> > 0.00 0.00 71.64 99.99 59.76 33781 478.507 6097 2358.743 >>>> 2837.249 >>>> > >>>> > -- >>>> > ~Rajesh.B >>>> > >>>> > >>>> > >>>> > -- >>>> > ~Rajesh.B >>>> >>>> >>> >>> >>> -- >>> ~Rajesh.B >>> >> >> > > > -- > ~Rajesh.B > -- ~Rajesh.B
