thanks. re #1. we need to find that Hiveserver2 process. For all i know the one you reported is hiveserver1 (which works.) chances are they use the same -Xmx value but we really shouldn't make any assumptions.
try wide format on the ps command (eg. ps -efw | grep -i Hiveserver2) re.#2. okay. so that tells us is not the number of columns blowing the heap but rather the combination of rows + columns. There's no way it stores the full result set on the heap even under normal circumstances so my guess is there's an internal number of rows it buffers. sorta like how unix buffers stdout. How and where that's set is out of my league. However, maybe you get around it by upping your heapsize again if you have the available memory of course. On Tue, Feb 18, 2014 at 8:39 AM, David Gayou <david.ga...@kxen.com> wrote: > > 1. I have no process with hiveserver2 ... > > "ps -ef | grep -i hive" return some pretty long command with a -Xmx8192 > and that's the value set in hive-env.sh > > > 2. The "select * from table limit 1" or even 100 is working correctly. > > > David. > > > On Tue, Feb 18, 2014 at 4:16 PM, Stephen Sprague <sprag...@gmail.com>wrote: > >> He lives on after all! and thanks for the continued feedback. >> >> We need the answers to these questions using HS2: >> >> >> >> 1. what is the output of "ps -ef | grep -i hiveserver2" on your >> system? in particular what is the value of -Xmx ? >> >> 2. does "select * from table limit 1" work? >> >> Thanks, >> Stephen. >> >> >> >> On Tue, Feb 18, 2014 at 6:32 AM, David Gayou <david.ga...@kxen.com>wrote: >> >>> I'm so sorry, i wrote an answer, and i forgot to sent it.... >>> And i haven't been able to work on this for a few days. >>> >>> >>> So far : >>> >>> I have a 15k columns table and 50k rows. >>> >>> I do not see any changes if i change the storage. >>> >>> >>> *Hive 12.0* >>> >>> My test query is "select * from bigtable" >>> >>> >>> If i use the hive cli, it works fine. >>> >>> If i use hiveserver1 + ODBC : it works fine >>> >>> If i use hiverserver2 + odbc or hiverserver2 + beeline,i have this java >>> exception : >>> >>> 2014-02-18 13:22:22,571 ERROR thrift.ProcessFunction >>> (ProcessFunction.java:process(41)) - Internal error processing FetchResults >>> >>> java.lang.OutOfMemoryError: Java heap space >>> at java.util.Arrays.copyOf(Arrays.java:2734) >>> at java.util.ArrayList.ensureCapacity(ArrayList.java:167) >>> at java.util.ArrayList.add(ArrayList.java:351) >>> at >>> org.apache.hive.service.cli.thrift.TRow.addToColVals(TRow.java:160) >>> at >>> org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:60) >>> at >>> org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:32) >>> at >>> org.apache.hive.service.cli.operation.SQLOperation.prepareFromRow(SQLOperation.java:270) >>> at >>> org.apache.hive.service.cli.operation.SQLOperation.decode(SQLOperation.java:262) >>> at >>> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:246) >>> at >>> org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:171) >>> at >>> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:438) >>> at >>> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:346) >>> at >>> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:407) >>> at >>> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1373) >>> >>> >>> >>> >>> *From the SVN trunk* : (for the HIVE-3746) >>> >>> With the maven change, most of the documentation and wiki are out of >>> date. >>> Compiling from trunk was not that easy and i may have failed some steps >>> but : >>> >>> It has the same behavior. It works in CLI and hiveserver1. >>> It fails with hiveserver 2. >>> >>> >>> Regards >>> >>> David Gayou >>> >>> >>> >>> >>> >>> On Thu, Feb 13, 2014 at 3:11 AM, Navis류승우 <navis....@nexr.com> wrote: >>> >>>> With HIVE-3746, which will be included in hive-0.13, HiveServer2 takes >>>> less memory than before. >>>> >>>> Could you try it with the version in trunk? >>>> >>>> >>>> 2014-02-13 10:49 GMT+09:00 Stephen Sprague <sprag...@gmail.com>: >>>> >>>> question to the original poster. closure appreciated! >>>>> >>>>> >>>>> On Fri, Jan 31, 2014 at 12:22 PM, Stephen Sprague >>>>> <sprag...@gmail.com>wrote: >>>>> >>>>>> thanks Ed. And on a separate tact lets look at Hiveserver2. >>>>>> >>>>>> >>>>>> @OP> >>>>>> >>>>>> *I've tried to look around on how i can change the thrift heap size >>>>>> but haven't found anything.* >>>>>> >>>>>> >>>>>> looking at my hiveserver2 i find this: >>>>>> >>>>>> $ ps -ef | grep -i hiveserver2 >>>>>> dwr 9824 20479 0 12:11 pts/1 00:00:00 grep -i hiveserver2 >>>>>> dwr 28410 1 0 00:05 ? 00:01:04 >>>>>> /usr/lib/jvm/java-6-sun/jre/bin/java >>>>>> *-Xmx256m*-Dhadoop.log.dir=/usr/lib/hadoop/logs >>>>>> -Dhadoop.log.file=hadoop.log >>>>>> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str= >>>>>> -Dhadoop.root.logger=INFO,console >>>>>> -Djava.library.path=/usr/lib/hadoop/lib/native >>>>>> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true >>>>>> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar >>>>>> /usr/lib/hive/lib/hive-service-0.12.0.jar >>>>>> org.apache.hive.service.server.HiveServer2 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> questions: >>>>>> >>>>>> 1. what is the output of "ps -ef | grep -i hiveserver2" on your >>>>>> system? in particular what is the value of -Xmx ? >>>>>> >>>>>> 2. can you restart your hiveserver with -Xmx1g? or some value that >>>>>> makes sense to your system? >>>>>> >>>>>> >>>>>> >>>>>> Lots of questions now. we await your answers! :) >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 31, 2014 at 11:51 AM, Edward Capriolo < >>>>>> edlinuxg...@gmail.com> wrote: >>>>>> >>>>>>> Final table compression should not effect the de serialized size of >>>>>>> the data over the wire. >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 31, 2014 at 2:49 PM, Stephen Sprague <sprag...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Excellent progress David. So. What the most important thing here >>>>>>>> we learned was that it works (!) by running hive in local mode and that >>>>>>>> this error is a limitation in the HiveServer2. That's important. >>>>>>>> >>>>>>>> so textfile storage handler and having issues converting it to ORC. >>>>>>>> hmmm. >>>>>>>> >>>>>>>> follow-ups. >>>>>>>> >>>>>>>> 1. what is your query that fails? >>>>>>>> >>>>>>>> 2. can you add a "limit 1" to the end of your query and tell us if >>>>>>>> that works? this'll tell us if it's column or row bound. >>>>>>>> >>>>>>>> 3. bonus points. run these in local mode: >>>>>>>> > set hive.exec.compress.output=true; >>>>>>>> > set mapred.output.compression.type=BLOCK; >>>>>>>> > set >>>>>>>> mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec; >>>>>>>> > create table blah stored as ORC as select * from <your >>>>>>>> table>; #i'm curious if this'll work. >>>>>>>> > show create table blah; #send output back if previous step >>>>>>>> worked. >>>>>>>> >>>>>>>> 4. extra bonus. change ORC to SEQUENCEFILE in #3 see if that works >>>>>>>> any differently. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'm wondering if compression would have any effect on the size of >>>>>>>> the internal ArrayList the thrift server uses. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 31, 2014 at 9:21 AM, David Gayou >>>>>>>> <david.ga...@kxen.com>wrote: >>>>>>>> >>>>>>>>> Ok, so here are some news : >>>>>>>>> >>>>>>>>> I tried to boost the HADOOP_HEAPSIZE to 8192, >>>>>>>>> I also setted the mapred.child.java.opts to 512M >>>>>>>>> >>>>>>>>> And it doesn't seem's to have any effect. >>>>>>>>> ------ >>>>>>>>> >>>>>>>>> I tried it using an ODBC driver => fail after few minutes. >>>>>>>>> Using a local JDBC (beeline) => running forever without any error. >>>>>>>>> >>>>>>>>> Both through hiveserver 2 >>>>>>>>> >>>>>>>>> If i use the local mode : it works! (but that not really what i >>>>>>>>> need, as i don't really how to access it with my software) >>>>>>>>> >>>>>>>>> ------ >>>>>>>>> I use a text file as storage. >>>>>>>>> I tried to use ORC, but i can't populate it with a load data (it >>>>>>>>> return an error of file format). >>>>>>>>> >>>>>>>>> Using an "ALTER TABLE orange_large_train_3 SET FILEFORMAT ORC" >>>>>>>>> after populating the table, i have a file format error on select. >>>>>>>>> >>>>>>>>> ------ >>>>>>>>> >>>>>>>>> @Edward : >>>>>>>>> >>>>>>>>> I've tried to look around on how i can change the thrift heap size >>>>>>>>> but haven't found anything. >>>>>>>>> Same thing for my client (haven't found how to change the heap >>>>>>>>> size) >>>>>>>>> >>>>>>>>> My usecase is really to have the most possible columns. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks a lot for your help >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> David >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 31, 2014 at 1:12 AM, Edward Capriolo < >>>>>>>>> edlinuxg...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Ok here are the problem(s). Thrift has frame size limits, thrift >>>>>>>>>> has to buffer rows into memory. >>>>>>>>>> >>>>>>>>>> Hove thrift has a heap size, it needs to big in this case. >>>>>>>>>> >>>>>>>>>> Your client needs a big heap size as well. >>>>>>>>>> >>>>>>>>>> The way to do this query if it is possible may be turning row >>>>>>>>>> lateral, potwntially by treating it as a list, it will make queries >>>>>>>>>> on it >>>>>>>>>> awkward. >>>>>>>>>> >>>>>>>>>> Good luck >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thursday, January 30, 2014, Stephen Sprague < >>>>>>>>>> sprag...@gmail.com> wrote: >>>>>>>>>> > oh. thinking some more about this i forgot to ask some other >>>>>>>>>> basic questions. >>>>>>>>>> > >>>>>>>>>> > a) what storage format are you using for the table (text, >>>>>>>>>> sequence, rcfile, orc or custom)? "show create table <table>" >>>>>>>>>> would yield >>>>>>>>>> that. >>>>>>>>>> > >>>>>>>>>> > b) what command is causing the stack trace? >>>>>>>>>> > >>>>>>>>>> > my thinking here is rcfile and orc are column based (i think) >>>>>>>>>> and if you don't select all the columns that could very well limit >>>>>>>>>> the size >>>>>>>>>> of the "row" being returned and hence the size of the internal >>>>>>>>>> ArrayList. >>>>>>>>>> OTOH, if you're using "select *", um, you have my sympathies. :) >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Thu, Jan 30, 2014 at 11:33 AM, Stephen Sprague < >>>>>>>>>> sprag...@gmail.com> wrote: >>>>>>>>>> > >>>>>>>>>> > thanks for the information. Up-to-date hive. Cluster on the >>>>>>>>>> smallish side. And, well, sure looks like a memory issue. :) rather >>>>>>>>>> than >>>>>>>>>> an inherent hive limitation that is. >>>>>>>>>> > >>>>>>>>>> > So. I can only speak as a user (ie. not a hive developer) but >>>>>>>>>> what i'd be interested in knowing next is is this via running hive >>>>>>>>>> in local >>>>>>>>>> mode, correct? (eg. not through hiveserver1/2). And it looks like it >>>>>>>>>> boinks on array processing which i assume to be internal code arrays >>>>>>>>>> and >>>>>>>>>> not hive data arrays - your 15K columns are all scalar/simple types, >>>>>>>>>> correct? Its clearly fetching results and looks be trying to store >>>>>>>>>> them in >>>>>>>>>> a java array - and not just one row but a *set* of rows (ArrayList) >>>>>>>>>> > >>>>>>>>>> > two things to try. >>>>>>>>>> > >>>>>>>>>> > 1. boost the heap-size. try 8192. And I don't know if >>>>>>>>>> HADOOP_HEAPSIZE is the controller of that. I woulda hoped it was >>>>>>>>>> called >>>>>>>>>> something like "HIVE_HEAPSIZE". :) Anyway, can't hurt to try. >>>>>>>>>> > >>>>>>>>>> > 2. trim down the number of columns and see where the breaking >>>>>>>>>> point is. is it 10K? is it 5K? The idea is to confirm its _the >>>>>>>>>> number of >>>>>>>>>> columns_ that is causing the memory to blow and not some other >>>>>>>>>> artifact >>>>>>>>>> unbeknownst to us. >>>>>>>>>> > >>>>>>>>>> > 3. Google around the Hive namespace for something that might >>>>>>>>>> limit or otherwise control the number of rows stored at once in >>>>>>>>>> Hive's >>>>>>>>>> internal buffer. I snoop around too. >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > That's all i got for now and maybe we'll get lucky and someone >>>>>>>>>> on this list will know something or another about this. :) >>>>>>>>>> > >>>>>>>>>> > cheers, >>>>>>>>>> > Stephen. >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Thu, Jan 30, 2014 at 2:32 AM, David Gayou < >>>>>>>>>> david.ga...@kxen.com> wrote: >>>>>>>>>> > >>>>>>>>>> > We are using the Hive 0.12.0, but it doesn't work better on >>>>>>>>>> hive 0.11.0 or hive 0.10.0 >>>>>>>>>> > Our hadoop version is 1.1.2. >>>>>>>>>> > Our cluster is 1 master + 4 slaves with 1 dual core xeon CPU >>>>>>>>>> (with hyperthreading so 4 cores per machine) + 16Gb Ram each >>>>>>>>>> > >>>>>>>>>> > The error message i get is : >>>>>>>>>> > >>>>>>>>>> > 2014-01-29 12:41:09,086 ERROR thrift.ProcessFunction >>>>>>>>>> (ProcessFunction.java:process(41)) - Internal error processing >>>>>>>>>> FetchResults >>>>>>>>>> > java.lang.OutOfMemoryError: Java heap space >>>>>>>>>> > at java.util.Arrays.copyOf(Arrays.java:2734) >>>>>>>>>> > at >>>>>>>>>> java.util.ArrayList.ensureCapacity(ArrayList.java:167) >>>>>>>>>> > at java.util.ArrayList.add(ArrayList.java:351) >>>>>>>>>> > at org.apache.hive.service.cli.Row.<init>(Row.java:47) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.RowSet.addRow(RowSet.java:61) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:235) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:170) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:417) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:306) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:386) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1373) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1358) >>>>>>>>>> > at >>>>>>>>>> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) >>>>>>>>>> > at >>>>>>>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) >>>>>>>>>> > at >>>>>>>>>> org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) >>>>>>>>>> > at java.security.AccessCont >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Sorry this was sent from mobile. Will do less grammar and spell >>>>>>>>>> check than usual. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
