Hi Chuck, I have a similar complex hive tables with many fields and some are nested like array of structs (but only upto 3 levels). How did you define you ROW FORMAT as to separate the items? The COLLECTION ITEMS TERMINATED BY clause works only for the first level.How did you handle level 2 , 3, etc? Is it through SERDE FORMATs? Could you share your CREATE TABLE statement? I am having problem correctly defining my DDL to load the data file correctly.
Much appreciated. Thanks, Sadu On Mon, Oct 8, 2012 at 3:37 PM, Connell, Chuck <chuck.conn...@nuance.com>wrote: > (Follow up to the thread below...) > > I have a complex Hive table -- many fields, many nested structs. Hive > fails to create the table at all. I can't even start to load data or query > the data. Anyone else run into this? It seems to be a showstopper to using > Hive for us, which is a shame since everything else about Hive is just what > we want. > > Any suggestions? Workarounds? > > Thanks, > Chuck > > > ________________________________________ > From: Connell, Chuck > Sent: Thursday, October 04, 2012 4:31 PM > To: user@hive.apache.org > Subject: RE: Limit to columns or nesting of Hive table? > > The issue apparently is not just the number of levels of nesting. I just > created a Hive table with 20 levels of structs within each other. It > created fine. This is more levels than the table that was failing for me. > The failing table had many more fields throughout the levels. > > Chuck > > > > -----Original Message----- > From: Connell, Chuck [mailto:chuck.conn...@nuance.com] > Sent: Thursday, October 04, 2012 12:09 PM > To: user@hive.apache.org > Subject: RE: Limit to columns or nesting of Hive table? > > Thanks. So is the nesting limit 10 now? Does your 2nd paragraph mean that > this limit cannot easily be raised? > > Chuck > > -----Original Message----- > From: Edward Capriolo [mailto:edlinuxg...@gmail.com] > Sent: Thursday, October 04, 2012 11:57 AM > To: user@hive.apache.org > Subject: Re: Limit to columns or nesting of Hive table? > > There is an open jira ticket on this. There is a hard coded limit but it > could be raised with some mostly minor code changes. > > One of the bigger problems is that hive stores the definition of a column > in JDBC "column" and for some databases larger nested structs can case > issues. > > Edward > > On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck <chuck.conn...@nuance.com> > wrote: > > I am trying to create a large Hive table, with many columns and deeply > > nested structs. It is failing with > java.lang.ArrayIndexOutOfBoundsException: > > 10. > > > > > > > > Before I spend a lot of time debugging my table declaration, is there > > some limit here I should know about? Max number of columns? Max depth > > of struct nesting? > > > > > > > > Thanks, > > > > Chuck > > > > >
