question to the original poster. closure appreciated!
On Fri, Jan 31, 2014 at 12:22 PM, Stephen Sprague <[email protected]>wrote: > thanks Ed. And on a separate tact lets look at Hiveserver2. > > > @OP> > > *I've tried to look around on how i can change the thrift heap size but > haven't found anything.* > > > looking at my hiveserver2 i find this: > > $ ps -ef | grep -i hiveserver2 > dwr 9824 20479 0 12:11 pts/1 00:00:00 grep -i hiveserver2 > dwr 28410 1 0 00:05 ? 00:01:04 > /usr/lib/jvm/java-6-sun/jre/bin/java > *-Xmx256m*-Dhadoop.log.dir=/usr/lib/hadoop/logs -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str= > -Dhadoop.root.logger=INFO,console > -Djava.library.path=/usr/lib/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar > /usr/lib/hive/lib/hive-service-0.12.0.jar > org.apache.hive.service.server.HiveServer2 > > > > > questions: > > 1. what is the output of "ps -ef | grep -i hiveserver2" on your system? > in particular what is the value of -Xmx ? > > 2. can you restart your hiveserver with -Xmx1g? or some value that > makes sense to your system? > > > > Lots of questions now. we await your answers! :) > > > > On Fri, Jan 31, 2014 at 11:51 AM, Edward Capriolo > <[email protected]>wrote: > >> Final table compression should not effect the de serialized size of the >> data over the wire. >> >> >> On Fri, Jan 31, 2014 at 2:49 PM, Stephen Sprague <[email protected]>wrote: >> >>> Excellent progress David. So. What the most important thing here we >>> learned was that it works (!) by running hive in local mode and that this >>> error is a limitation in the HiveServer2. That's important. >>> >>> so textfile storage handler and having issues converting it to ORC. hmmm. >>> >>> follow-ups. >>> >>> 1. what is your query that fails? >>> >>> 2. can you add a "limit 1" to the end of your query and tell us if that >>> works? this'll tell us if it's column or row bound. >>> >>> 3. bonus points. run these in local mode: >>> > set hive.exec.compress.output=true; >>> > set mapred.output.compression.type=BLOCK; >>> > set >>> mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec; >>> > create table blah stored as ORC as select * from <your table>; >>> #i'm curious if this'll work. >>> > show create table blah; #send output back if previous step >>> worked. >>> >>> 4. extra bonus. change ORC to SEQUENCEFILE in #3 see if that works any >>> differently. >>> >>> >>> >>> I'm wondering if compression would have any effect on the size of the >>> internal ArrayList the thrift server uses. >>> >>> >>> >>> On Fri, Jan 31, 2014 at 9:21 AM, David Gayou <[email protected]>wrote: >>> >>>> Ok, so here are some news : >>>> >>>> I tried to boost the HADOOP_HEAPSIZE to 8192, >>>> I also setted the mapred.child.java.opts to 512M >>>> >>>> And it doesn't seem's to have any effect. >>>> ------ >>>> >>>> I tried it using an ODBC driver => fail after few minutes. >>>> Using a local JDBC (beeline) => running forever without any error. >>>> >>>> Both through hiveserver 2 >>>> >>>> If i use the local mode : it works! (but that not really what i need, >>>> as i don't really how to access it with my software) >>>> >>>> ------ >>>> I use a text file as storage. >>>> I tried to use ORC, but i can't populate it with a load data (it >>>> return an error of file format). >>>> >>>> Using an "ALTER TABLE orange_large_train_3 SET FILEFORMAT ORC" after >>>> populating the table, i have a file format error on select. >>>> >>>> ------ >>>> >>>> @Edward : >>>> >>>> I've tried to look around on how i can change the thrift heap size but >>>> haven't found anything. >>>> Same thing for my client (haven't found how to change the heap size) >>>> >>>> My usecase is really to have the most possible columns. >>>> >>>> >>>> Thanks a lot for your help >>>> >>>> >>>> Regards >>>> >>>> David >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Jan 31, 2014 at 1:12 AM, Edward Capriolo <[email protected] >>>> > wrote: >>>> >>>>> Ok here are the problem(s). Thrift has frame size limits, thrift has >>>>> to buffer rows into memory. >>>>> >>>>> Hove thrift has a heap size, it needs to big in this case. >>>>> >>>>> Your client needs a big heap size as well. >>>>> >>>>> The way to do this query if it is possible may be turning row lateral, >>>>> potwntially by treating it as a list, it will make queries on it awkward. >>>>> >>>>> Good luck >>>>> >>>>> >>>>> On Thursday, January 30, 2014, Stephen Sprague <[email protected]> >>>>> wrote: >>>>> > oh. thinking some more about this i forgot to ask some other basic >>>>> questions. >>>>> > >>>>> > a) what storage format are you using for the table (text, sequence, >>>>> rcfile, orc or custom)? "show create table <table>" would yield that. >>>>> > >>>>> > b) what command is causing the stack trace? >>>>> > >>>>> > my thinking here is rcfile and orc are column based (i think) and if >>>>> you don't select all the columns that could very well limit the size of >>>>> the >>>>> "row" being returned and hence the size of the internal ArrayList. OTOH, >>>>> if you're using "select *", um, you have my sympathies. :) >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > On Thu, Jan 30, 2014 at 11:33 AM, Stephen Sprague < >>>>> [email protected]> wrote: >>>>> > >>>>> > thanks for the information. Up-to-date hive. Cluster on the smallish >>>>> side. And, well, sure looks like a memory issue. :) rather than an >>>>> inherent hive limitation that is. >>>>> > >>>>> > So. I can only speak as a user (ie. not a hive developer) but what >>>>> i'd be interested in knowing next is is this via running hive in local >>>>> mode, correct? (eg. not through hiveserver1/2). And it looks like it >>>>> boinks on array processing which i assume to be internal code arrays and >>>>> not hive data arrays - your 15K columns are all scalar/simple types, >>>>> correct? Its clearly fetching results and looks be trying to store them >>>>> in >>>>> a java array - and not just one row but a *set* of rows (ArrayList) >>>>> > >>>>> > two things to try. >>>>> > >>>>> > 1. boost the heap-size. try 8192. And I don't know if >>>>> HADOOP_HEAPSIZE is the controller of that. I woulda hoped it was called >>>>> something like "HIVE_HEAPSIZE". :) Anyway, can't hurt to try. >>>>> > >>>>> > 2. trim down the number of columns and see where the breaking point >>>>> is. is it 10K? is it 5K? The idea is to confirm its _the number of >>>>> columns_ that is causing the memory to blow and not some other artifact >>>>> unbeknownst to us. >>>>> > >>>>> > 3. Google around the Hive namespace for something that might limit >>>>> or otherwise control the number of rows stored at once in Hive's internal >>>>> buffer. I snoop around too. >>>>> > >>>>> > >>>>> > That's all i got for now and maybe we'll get lucky and someone on >>>>> this list will know something or another about this. :) >>>>> > >>>>> > cheers, >>>>> > Stephen. >>>>> > >>>>> > >>>>> > >>>>> > On Thu, Jan 30, 2014 at 2:32 AM, David Gayou <[email protected]> >>>>> wrote: >>>>> > >>>>> > We are using the Hive 0.12.0, but it doesn't work better on hive >>>>> 0.11.0 or hive 0.10.0 >>>>> > Our hadoop version is 1.1.2. >>>>> > Our cluster is 1 master + 4 slaves with 1 dual core xeon CPU (with >>>>> hyperthreading so 4 cores per machine) + 16Gb Ram each >>>>> > >>>>> > The error message i get is : >>>>> > >>>>> > 2014-01-29 12:41:09,086 ERROR thrift.ProcessFunction >>>>> (ProcessFunction.java:process(41)) - Internal error processing >>>>> FetchResults >>>>> > java.lang.OutOfMemoryError: Java heap space >>>>> > at java.util.Arrays.copyOf(Arrays.java:2734) >>>>> > at java.util.ArrayList.ensureCapacity(ArrayList.java:167) >>>>> > at java.util.ArrayList.add(ArrayList.java:351) >>>>> > at org.apache.hive.service.cli.Row.<init>(Row.java:47) >>>>> > at org.apache.hive.service.cli.RowSet.addRow(RowSet.java:61) >>>>> > at >>>>> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:235) >>>>> > at >>>>> org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:170) >>>>> > at >>>>> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:417) >>>>> > at >>>>> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:306) >>>>> > at >>>>> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:386) >>>>> > at >>>>> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1373) >>>>> > at >>>>> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1358) >>>>> > at >>>>> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) >>>>> > at >>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >>>>> > at >>>>> org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) >>>>> > at >>>>> org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) >>>>> > at java.security.AccessCont >>>>> >>>>> -- >>>>> Sorry this was sent from mobile. Will do less grammar and spell check >>>>> than usual. >>>>> >>>> >>>> >>> >> >
