Hi, Luke, Thanks for your help. I'll test it. According to my tests, mem used by BloomFilter varied from 2B to 26MB. Perhaps sth. went wrong.
2009/7/18 Luke <[email protected]> > > Looks like the default is already 'rows', however we use first 1000 > cells to estimate number of rows in the cell store. The estimate could > be wrong. > > Increase max-approx-items might help: bloomfilter="rows > --max-approx-items 10000" > > 2009/7/17 Luke <[email protected]>: > > 2009/7/17 孔令华 <[email protected]>: > >> Thanks, Doug & Luke > >> > >> In my opinion, this kind of performance issue is not in the critical > path > >> (just in mind, no data support, but I think RPCs and disk-seeking would > cost > >> much more time). A clear log is very important, specially for normal > users. > > > > We sometime needs to turn on debugging to show a lot more data, we > > want to make sure the timing is not affected too much. Note, the > > logging system is not for range server only, it can be used for many > > other tools that could be cpu intensive. With the current format you > > can display it in any timezones easily, which would be important when > > you have data centers across the world. > > > >> BTW，according to our tests, using BloomFilter with the default params > costs > >> too many mems (a RangeServer with 1600 Ranges cost about 9GB, and about > 2GB > >> without BloomFilter). And I found that BloomFilter actually need these > mems > >> instead of fragments. > > > > Are there a lot of columns in the table? Can you use the > > bloomfilter=rows to see how much memory it uses? 7GB at 1% false > > positive rate can accommodate 6 billion items! > > > > Do you want an option to provide fixed amount memory per access group? > > > >> And at present, BloomFilter affects the whole system, which I think > should > >> be AccessGroup-wide. Using it with the table schema would be a better > >> choice(eg. appoint them when create table). > > > > You can certainly turn it off globally and use the options at create > > table time now. Doug is working on a lazy loading scheme, so that it > > uses less memory (bloomfilter and index are not loaded unless needed) > > > > __Luke > > > >> 2009/7/18 Luke <[email protected]> > >>> > >>> You can alias lcat='perl -pe "s/^\d+/localtime($&)/e"' > >>> > >>> Then you can lcat *.log to see more readable timestamps. Time > >>> conversion is very slow, especially it needs grab a global lock for > >>> timezones. Storing timestamps in timezone neutral format has a lot of > >>> benefits. > >>> > >>> On Fri, Jul 17, 2009 at 7:45 AM, Phoenix<[email protected]> > wrote: > >>> > > >>> > 1. At present, the time field of the log is very hard to read. It's > >>> > unreadable at some point. > >>> > eg. 1247799471 INFO Hypertable.Master : > >>> > > >>> > There're some configurable options that can opt the output string. > >>> > eg: > >>> > > >>> > diff --git a/src/cc/Common/Logger.cc b/src/cc/Common/Logger.cc > >>> > index 301a2d1..985f231 100644 > >>> > --- a/src/cc/Common/Logger.cc > >>> > +++ b/src/cc/Common/Logger.cc > >>> > @@ -28,6 +28,7 @@ > >>> > #include <log4cpp/Layout.hh> > >>> > #include <log4cpp/NDC.hh> > >>> > #include <log4cpp/Priority.hh> > >>> > +#include <log4cpp/PatternLayout.hh> > >>> > > >>> > #include "Logger.h" > >>> > > >>> > @@ -127,8 +128,8 @@ void > >>> > Logger::initialize(const String &name, int priority, bool > >>> > flush_per_log, > >>> > std::ostream &out) { > >>> > appender = new FlushableOstreamAppender("default", out, > >>> > flush_per_log); > >>> > - Logging::Layout* layout = new Logging::BasicLayout(); > >>> > - //Logging::Layout* layout = new MicrosecondLayout(); > >>> > + Logging::PatternLayout* layout = new Logging::PatternLayout(); > >>> > + layout->setConversionPattern("[%d{%Y-%m-%d %H:%M:%S}] %p %c %x: %m > >>> > %n"); > >>> > appender->setLayout(layout); > >>> > logger = &(Logging::Category::getInstance(name)); > >>> > logger->addAppender(appender); > >>> > > >>> > > >>> > After this mod, the output string is much better. eg: > >>> > [2009-07-17 14:53:57] INFO Hypertable.RangeServer : > >>> > > >>> > > >>> > 2. Besides, log should support rolling. At present, the log files'll > >>> > grow very large as time goes. > >>> > > > >>> > > >>> > >>> > >> > >> > >> >> > >> > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
