We have had a ticket open for quite some time for combine input format to work across partitions. Not sure if that can help with what you are seeing as well. It could help us alot.
Edward On Thu, Jun 14, 2012 at 1:25 PM, Gang Liu <g...@fb.com> wrote: > Hey Edward, > > Thank you very much for providing comments. > > This feature is designed for use cases described in wiki. We do see them > in the real life so that we come up with the feature. > > In this first release, in order to use the feature: > 1. Hive table users need to know the skewed key in advance > 2. Hive table users need to know the skewed key is the same each > partition. > 3. If Hive table users know skewed key change, they can "alter" skewed key > via "alter" statement. > > 4. If #3 happens, old partitions have old skewed key and new partition > have new. It's expected. > > We may consider the following in the future release: > 1. Hive instruments skewed key and displays them to user > > Thanks > > Tim > > > On 6/14/12 9:34 AM, "Edward Capriolo" <edlinuxg...@gmail.com> wrote: > >>I am of the opinion this feature is too specialized to be generally >>helpful. >> >>------------------------------- >>The cardinality of 'x' is in 1000's per partition of T. Moreover, >>there is a skew for the values of 'x'. In general, there are ~10 >>values of 'x' which have a very large skew, and the remaining >>values of 'x' have a small cardinality. Also, note that this mapping >>(values of 'x' with a high cardinality can change daily). >>-------------------------- >> >>In these cases you should use clustering/bucketing. This will prevent >>the skew you are talking about. If you want more efficiency in certain >>query types build a index on top of the original table. >> >>I understand someone wanting to do this because mysql partition can do >>this, but this sounds like a management problem. Who is to say the >>skew is the same each partition? >> >>----------------------------------------- >>hive compiler to do input pruning. The list of skewed keys is stored >>at the table level (note that, this list can be initially supplied by >>the client periodically, and can be eventually updated when a new >>partition is being loaded). >>----------------------------------------- >> >>Imagine you have a table partitioned by hour and two datacenters China >>and NY. At some hours the skew will be different. Skews change over >>time. Since this property is table level I do not understand how this >>would be changed. >> >> >> >>On Thu, Jun 14, 2012 at 4:14 AM, Carl Steinbach <c...@cloudera.com> wrote: >>> Hi Tim, >>> >>> I added some comments to the wiki a couple days ago. I just wanted to >>>make >>> sure you saw them since it doesn't look like you're registered as a >>>watcher >>> for that page. >>> >>> Thanks. >>> >>> Carl >>> >>> On Mon, Jun 11, 2012 at 12:22 PM, Gang Liu <g...@fb.com> wrote: >>> >>>> Hi Carl, thanks Tim >>>> >>>> On 6/11/12 12:14 PM, "Carl Steinbach" <c...@cloudera.com> wrote: >>>> >>>> >+ hcatalog-dev >>>> > >>>> >On Mon, Jun 11, 2012 at 12:09 PM, Carl Steinbach <c...@cloudera.com> >>>> >wrote: >>>> > >>>> >> This link may work better for some people: >>>> >> >>>> >> https://cwiki.apache.org/confluence/display/Hive/ListBucketing >>>> >> >>>> >> Thanks. >>>> >> >>>> >> Carl >>>> >> >>>> >> >>>> >> On Mon, Jun 11, 2012 at 12:03 PM, Gang Liu <g...@fb.com> wrote: >>>> >> >>>> >>> Dear all hive developers, >>>> >>> >>>> >>> We are making good progress of implementing the list bucketing >>>> >>>feature. It >>>> >>> should be available soon in weeks. >>>> >>> >>>> >>> We'd like to call feature review again and please provide your >>>> >>>comments. >>>> >>> >>>> >>> Thanks >>>> >>> >>>> >>> Tim >>>> >>> >>>> >>> On 6/1/12 10:13 AM, "Gang Liu" <g...@fb.com> wrote: >>>> >>> >>>> >>> >Dear all, >>>> >>> > >>>> >>> >Please review the proposal and provide your comments: >>>> >>> > >>>> >>> >https://cwiki.apache.org/Hive/listbucketing.html >>>> >>> > >>>> >>> > >>>> >>> >Thanks >>>> >>> > >>>> >>> >Tim >>>> >>> > >>>> >>> >>>> >>> >>>> >> >>>> >>>> >
