I wanted to get back to the group with results. We upgraded to 1.3.4 and the WA GC worked after start up. Unfortunately, I don't know if it was the restart or the upgrade that mattered as the JIRA references the fix in 1.3.6.
Thanks for all of the help. -Kris On Thu, Apr 12, 2012 at 11:02 AM, Eric Newton <[email protected]> wrote: > Running the tracer is optional. Just make an empty tracers file in the > conf directory and restart. > > -Eric > > On Thu, Apr 12, 2012 at 10:21 AM, Kristopher Kane <[email protected]>wrote: > >> Eric, thanks for the help. >> >> This is on 1.3.2, looks like we will be able to go to 1.3.4 very >> shortly. From the below JIRA, it looks like it is fixed in 1.3.6. I will >> send this up as I'm not an approving authority on installed/running >> versions and will also recommend the trace password in the site.xml. >> >> Once we upgrade to the minor version, I will report back on whether the >> WAL GC runs in that version. >> >> -Kris >> >> On Wed, Apr 11, 2012 at 1:00 PM, Eric Newton <[email protected]>wrote: >> >>> Also you will need to restart. >>> >>> What version of accumulo are you running? This problem should have been >>> fixed with ACCUMULO-243. >>> >>> -Eric >>> >>> On Wed, Apr 11, 2012 at 12:28 PM, Eric Newton <[email protected]>wrote: >>> >>>> in conf/accumulo-site.xml, add this: >>>> >>>> <property> >>>> <name>trace.password</name> >>>> <value>your-root-password-here</value> >>>> </property> >>>> >>>> -Eric >>>> >>>> On Wed, Apr 11, 2012 at 12:14 PM, Kristopher Kane <[email protected] >>>> > wrote: >>>> >>>>> I know I haven't answered some of the previous questions to me, but, >>>>> the GC talk brought me to the source of the problem. our WAL GC job is >>>>> not >>>>> successfully running. The GC logs state a bad username/password, so, will >>>>> investigate that. >>>>> >>>>> Thanks for the help! >>>>> >>>>> -Kris >>>>> >>>>> On Wed, Apr 11, 2012 at 11:18 AM, Keith Turner <[email protected]>wrote: >>>>> >>>>>> When data is written to the accumulo its written to memory and the >>>>>> write ahead logs. The data in memory is sorted and the data in the >>>>>> write ahead logs is written asis (unsorted). When the data in memory >>>>>> is flushed to HDFS, the write ahead logs that also contain that data >>>>>> are no longer needed. >>>>>> >>>>>> On Wed, Apr 11, 2012 at 11:12 AM, Eric Newton <[email protected]> >>>>>> wrote: >>>>>> > Logs are only put into HDFS during a recovery. >>>>>> > >>>>>> > Flush removes references to WALs, and the accumulo gc will ask the >>>>>> loggers >>>>>> > to delete them when there are no references to them. >>>>>> > >>>>>> > -Eric >>>>>> > >>>>>> > >>>>>> > On Wed, Apr 11, 2012 at 11:02 AM, Kristopher Kane < >>>>>> [email protected]> >>>>>> > wrote: >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Wed, Apr 11, 2012 at 10:48 AM, Keith Turner <[email protected]> >>>>>> wrote: >>>>>> >>> >>>>>> >>> How big is the partition? Are the same number of logger servers >>>>>> >>> running as tablet servers? >>>>>> >>> >>>>>> >>> >>>>>> >>> You can scan the metadata table to look for tablets that have >>>>>> alot of >>>>>> >>> write-ahead logs. I think the command below will show you how many >>>>>> >>> write-ahead logs each tablet has. Look for any tablets that have >>>>>> too >>>>>> >>> many. I think it should sort the tablets with the most tablets to >>>>>> the >>>>>> >>> top, but not positive. >>>>>> >>> >>>>>> >>> ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c >>>>>> >>> log' | cut -f 1 -d ' ' | uniq -c | sort -r -n >>>>>> >>> >>>>>> >>> I think the following command will show you how many active log >>>>>> each >>>>>> >>> logger has. This should be even. >>>>>> >>> >>>>>> >>> ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c >>>>>> >>> log' | cut -f 2 -d ' ' | cut -d ':' -f 2 | sort | uniq -c >>>>>> >>> >>>>>> >>> You can use the "flush -p" command in the shell to force data in >>>>>> >>> memory to disk and stop referencing write-ahead logs. Maybe >>>>>> execute >>>>>> >>> the commands above before and after flushing. >>>>>> >>> >>>>>> >>> Keith >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> Thanks for the replies. I read about the flush command in the >>>>>> docs but >>>>>> >> didn't make a connection between "memory" to the write ahead >>>>>> logs. Is >>>>>> >> that correct? Flush writes write ahead log data to hdfs? >>>>>> >> >>>>>> >> Thanks! >>>>>> >> >>>>>> >> -Kris >>>>>> > >>>>>> > >>>>>> >>>>> >>>>> >>>> >>> >> >
