Do not fear the big row. And yes, increase your split threshold to avoid the warning messages about a big row.
-Eric On Fri, Nov 9, 2012 at 1:29 PM, Anthony Fox <[email protected]> wrote: > Do you mean two partitions per server? In my case, that would correspond > to 30 total rows which would make each row very large ... >1G/row. Should > I increase the table.split.threshold in a corresponding way? > > > On Fri, Nov 9, 2012 at 1:09 PM, John Vines <[email protected]> wrote: > >> Glad to hear. I typically advice a minimum of 2 shards per tserver. I >> would say the maximum is actually based on the tablet size. Others in the >> country may disagree/provide better reasoning. >> >> Sent from my phone, pardon the typos and brevity. >> On Nov 9, 2012 1:03 PM, "Anthony Fox" <[email protected]> wrote: >> >>> Ok, I reingested with 1000 rows and performance for both single record >>> scans and index scans is much better. I'm going to experiment a bit with >>> the optimal number of rows. Thanks for the help, everyone. >>> >>> >>> On Fri, Nov 9, 2012 at 12:41 PM, John Vines <[email protected]> wrote: >>> >>>> The bloom filter checks only occur on a seek, and the way the column >>>> family filter works it's it seeks and then does a few scans to see if the >>>> appropriate families pop up in the short term. Bloom filter on the column >>>> family would be better if you had larger rows to encourage more >>>> seeks/minimize the number of rows to do bloom checks. >>>> >>>> The issue is that you are ultimately checking every single row for a >>>> column, which is sparse. It's not that different than doing a full table >>>> regex. If you had locality groups set up it would be more performant, until >>>> you create locality groups for everything. >>>> >>>> The intersecting iterators get their performance by being able to >>>> operate on large rows to avoid the penalty of checking each row. Minimize >>>> the number of partitions you have and it should clear up your issues. >>>> >>>> John >>>> >>>> Sent from my phone, pardon the typos and brevity. >>>> On Nov 9, 2012 12:24 PM, "William Slacum" < >>>> [email protected]> wrote: >>>> >>>>> I'll ask for someone to verify this comment for me (look @ u John W >>>>> Vines), but the bloom filter helps when you have a discrete number of >>>>> column families that will appear across many rows. >>>>> >>>>> On Fri, Nov 9, 2012 at 12:18 PM, Anthony Fox <[email protected]>wrote: >>>>> >>>>>> Ah, ok, I was under the impression that this would be really fast >>>>>> since I have a column family bloom filter turned on. Is this not >>>>>> correct? >>>>>> >>>>>> >>>>>> On Fri, Nov 9, 2012 at 12:15 PM, William Slacum < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> When I said smaller of tablets, I really mean smaller number of rows >>>>>>> :) My apologies. >>>>>>> >>>>>>> So if you're searching for a random column family in a table, like >>>>>>> with a `scan -c <cf>` in the shell, it will start at row 0 and work >>>>>>> sequentially up to row 10000000 until it finds the cf. >>>>>>> >>>>>>> >>>>>>> On Fri, Nov 9, 2012 at 12:11 PM, Anthony Fox >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> This scan is without the intersecting iterator. I'm just trying to >>>>>>>> pull back a single data record at the moment which corresponds to >>>>>>>> scanning >>>>>>>> for one column family. I'll try with a smaller number of tablets, but >>>>>>>> is >>>>>>>> the computation effort the same for the scan I am doing? >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Nov 9, 2012 at 12:02 PM, William Slacum < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> So that means you have roughly 312.5k rows per tablet, which means >>>>>>>>> about 725k column families in any given tablet. The intersecting >>>>>>>>> iterator >>>>>>>>> will work at a row per time, so I think at any given moment, it will >>>>>>>>> be >>>>>>>>> working through 32 at a time and doing a linear scan through the RFile >>>>>>>>> blocks. With RFile indices, that check is usually pretty fast, but >>>>>>>>> you're >>>>>>>>> having go through 4 orders of magnitude more data sequentially than >>>>>>>>> you can >>>>>>>>> work on. If you can experiment and re-ingest with a smaller number of >>>>>>>>> tablets, anywhere between 15 and 45, I think you will see better >>>>>>>>> performance. >>>>>>>>> >>>>>>>>> On Fri, Nov 9, 2012 at 11:53 AM, Anthony Fox <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Failed to answer the original question - 15 tablet servers, 32 >>>>>>>>>> tablets/splits. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Nov 9, 2012 at 11:52 AM, Anthony Fox < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I've tried a number of different settings of >>>>>>>>>>> table.split.threshold. I started at 1G and bumped it down to 128M >>>>>>>>>>> and the >>>>>>>>>>> cf scan is still ~30 seconds for both. I've also used less rows - >>>>>>>>>>> 00000 to >>>>>>>>>>> 99999 and still see similar performance numbers. I thought the >>>>>>>>>>> column >>>>>>>>>>> family bloom filter would help deal with large row space but >>>>>>>>>>> sparsely >>>>>>>>>>> populated column space. Is that correct? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Nov 9, 2012 at 11:49 AM, William Slacum < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I'm more inclined to believe it's because you have to search >>>>>>>>>>>> across 10M different rows to find any given column family, since >>>>>>>>>>>> they're >>>>>>>>>>>> randomly, and possibly uniformly, distributed. How many tablets >>>>>>>>>>>> are you >>>>>>>>>>>> searching across? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Nov 9, 2012 at 11:45 AM, Anthony Fox < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes, there are 10M possible partitions. I do not have a hash >>>>>>>>>>>>> from value to partition, the data is essentially randomly >>>>>>>>>>>>> balanced across >>>>>>>>>>>>> all the tablets. Unlike the bloom filter and intersecting >>>>>>>>>>>>> iterator >>>>>>>>>>>>> examples, I do not have locality groups turned on and I have data >>>>>>>>>>>>> in the cq >>>>>>>>>>>>> and the value for both index entries and record entries. Could >>>>>>>>>>>>> this be the >>>>>>>>>>>>> issue? Each record entry has approximately 30 column qualifiers >>>>>>>>>>>>> with data >>>>>>>>>>>>> in the value for each. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Nov 9, 2012 at 11:41 AM, William Slacum < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I guess assuming you have 10M possible partitions, if you're >>>>>>>>>>>>>> using a relatively uniform hash to generate your IDs, you'll >>>>>>>>>>>>>> average about >>>>>>>>>>>>>> 2 per partition. Do you have any index for term/value to >>>>>>>>>>>>>> partition? This >>>>>>>>>>>>>> will help you narrow down your search space to a subset of your >>>>>>>>>>>>>> partitions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Nov 9, 2012 at 11:39 AM, William Slacum < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> That shouldn't be a huge issue. How many rows/partitions do >>>>>>>>>>>>>>> you have? How many do you have to scan to find the specific >>>>>>>>>>>>>>> column >>>>>>>>>>>>>>> family/doc id you want? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Nov 9, 2012 at 11:26 AM, Anthony Fox < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have a table set up to use the intersecting iterator >>>>>>>>>>>>>>>> pattern. The >>>>>>>>>>>>>>>> table has about 20M records which leads to 20M column families >>>>>>>>>>>>>>>> for the >>>>>>>>>>>>>>>> data section - 1 unique column family per record. The index >>>>>>>>>>>>>>>> section of >>>>>>>>>>>>>>>> the table is not quite as large as the data section. The >>>>>>>>>>>>>>>> rowkey is a >>>>>>>>>>>>>>>> random padded integer partition between 0000000 and 9999999. >>>>>>>>>>>>>>>> I turned >>>>>>>>>>>>>>>> bloom filters on and used the ColumnFamilyFunctor to get >>>>>>>>>>>>>>>> performant >>>>>>>>>>>>>>>> column family scans without specifying a range like in the >>>>>>>>>>>>>>>> bloom filter >>>>>>>>>>>>>>>> examples in the README. However, my column family scans >>>>>>>>>>>>>>>> (without any >>>>>>>>>>>>>>>> custom iterator) are still fairly slow - ~30 seconds for a >>>>>>>>>>>>>>>> column family >>>>>>>>>>>>>>>> batch scan of one record. I've also tried RowFunctor but I see >>>>>>>>>>>>>>>> similar >>>>>>>>>>>>>>>> performance. Can anyone shed any light on the performance >>>>>>>>>>>>>>>> metrics I'm >>>>>>>>>>>>>>>> seeing? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Anthony >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >
