Ok, but I can't exactly use it for debugging unless I have some sort of a viewer - also, this is for writes, but mostly our issues are reads.
-Jack On Sun, Apr 10, 2011 at 4:58 PM, Stack <[email protected]> wrote: > That's our wal log > > > > On Apr 10, 2011, at 13:46, Jack Levin <[email protected]> wrote: > >> I would love it to be exactly like mysqld.log where you save all your >> queries there. >> >> -Jack >> >> On Sun, Apr 10, 2011 at 12:31 PM, Stack <[email protected]> wrote: >>> They don't enough detail when you enable metrics on rpc? >>> >>> It should be easy enough hacking this in. What would you like to see >>> on the query Jack? There'd be whether a Get/Delete/Put, etc., then >>> what? We'd have to give this logging a different name than that of >>> the general regionserver class because you'd never want to run a >>> production system with it on; you'd only want to enable it say via the >>> UI for a short period of time, and then turn it off, because it'd be a >>> drag on perf and it would make for reams of output (Consider server >>> doing 1k hits a second, or in your case Jack, 100k hits a second -- >>> smile -- then we'd fill the disks in short order) >>> >>> St.Ack >>> >>> >>> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[email protected]> wrote: >>>> Well, counts would help a lot, but actual queries displayed even more >>>> so... For example I just found a bug in our code that runs in infinite >>>> loop across only specific pages that cause region server CPU to >>>> overrun. It would be easier for us to ID the bugs if we had all >>>> queries logged. >>>> >>>> -Jack >>>> >>>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[email protected]> wrote: >>>>> I think HBASE-3647 was a good start but Jack asked might be a little more. >>>>> BTW 3647 isn't in 0.90.2 >>>>> >>>>> Currently regionserver.jsp displays RegionServerMetrics for every region >>>>> in >>>>> an unsorted order w.r.t. read/write request counts. For any region server >>>>> in >>>>> production, there would be hundreds of regions, it is not realistic to >>>>> spot >>>>> hot regions in the current display. >>>>> Do we need to add capability to sort the regions on regionserver.jsp >>>>> according to read/write request counts ? >>>>> >>>>> Cheers >>>>> >>>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[email protected]> wrote: >>>>> >>>>>> You mean per-region Jack? >>>>>> >>>>>> These went into TRUNK recently: >>>>>> >>>>>> HBASE-3647 Distinguish read and write request count in region >>>>>> (Ted Yu via Stack) >>>>>> HBASE-3704 Show per region request count in table.jsp >>>>>> (Ted Yu via Stack) >>>>>> >>>>>> Would they help? >>>>>> >>>>>> St.Ack >>>>>> >>>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[email protected]> wrote: >>>>>>> It would be super nice to have RPC query call logging per region >>>>>>> server. Sometimes we have 'hot' regions getting hit by users, and >>>>>>> this drives the CPU on a particular RS server up, except we have no >>>>>>> idea what table it might be. I could rig the tcpdump on port 60020, >>>>>>> and log things with AWK, but that would be super hacky. An insight >>>>>>> what traffic drives the cpu will allow us (or anyone else) model the >>>>>>> tables to user patterns accordingly. >>>>>>> >>>>>>> thanks. >>>>>>> >>>>>>> -Jack >>>>>>> >>>>>> >>>>> >>>> >>> >
