Thanks, Doug, I would try it. Of course, I have to figure out which files fall into the filelist, firstly.
thanks -- kuer On 7月16日, 上午1时02分, Doug Judd <[email protected]> wrote: > Hi kuer, > > Thanks for the report. According to Luke (who wrote the code), these > messages are innocuous. I agree that the code should be fixed so that it > doesn't try to delete these files that don't exist. However, I think the > code does not ever delete files that are in use. This code has been in > place for about a year and we've never experienced that kind of problem. > > If you want to take a shot at fixing the non-existent file delete problem, > go for it. We'll pull it in if it looks good. > > - Doug > > On Wed, Jul 15, 2009 at 2:15 AM, kuer <[email protected]> wrote: > > > Hi, all. > > > After restarting hypertable + kfs in my work-boxes, there are some > > endless messages logging into Hypertable.Master.log : > > > 2009-07-15 16:45:42,431 1350199616 Hypertable.Master [WARN] (Master/ > > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ > > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs12 > > 2009-07-15 16:45:42,432 1350199616 Hypertable.Master [WARN] (Master/ > > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ > > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs14 > > 2009-07-15 16:45:42,432 1350199616 Hypertable.Master [WARN] (Master/ > > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ > > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs3 > > 2009-07-15 16:45:42,433 1350199616 Hypertable.Master [WARN] (Master/ > > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ > > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs4 > > ... > > ... > > ... > > 2009-07-15 16:45:42,433 1350199616 Hypertable.Master [WARN] (Master/ > > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ > > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs8 > > 2009-07-15 16:45:42,434 1350199616 Hypertable.Master [WARN] (Master/ > > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ > > storage_se/default/13C3704B81B2919AE7F6A936/cs12 > > > Then, I check the dfs.log : > > > 2009-07-15 16:55:44,060 1319373120 kosmosBroker [ERROR] (kosmos/ > > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ > > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs10' - No such file or > > directory > > 2009-07-15 16:55:44,061 1308883264 kosmosBroker [ERROR] (kosmos/ > > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ > > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs12' - No such file or > > directory > > 2009-07-15 16:55:44,061 1130555712 kosmosBroker [ERROR] (kosmos/ > > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ > > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs14' - No such file or > > directory > > 2009-07-15 16:55:44,062 1151535424 kosmosBroker [ERROR] (kosmos/ > > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ > > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs3' - No such file or > > directory > > 2009-07-15 16:55:44,062 1141045568 kosmosBroker [ERROR] (kosmos/ > > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ > > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs4' - No such file or > > directory > > ... > > ... > > ... > > 2009-07-15 16:55:44,063 1162025280 kosmosBroker [ERROR] (kosmos/ > > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ > > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs8' - No such file or > > directory > > 2009-07-15 16:55:44,064 1172515136 kosmosBroker [ERROR] (kosmos/ > > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ > > storage_se/default/13C3704B81B2919AE7F6A936/cs12' - No such file or > > directory > > > When reading source code src/cc/Hypertable/Master/MasterGc.cc : > > 182 void > > 183 reap(CountMap &files_map) { > > 184 size_t nf = 0, nf_done = 0, nd = 0, nd_done = 0; > > 185 CountMap dirs_map; // reap empty range directories as well > > 186 > > 187 foreach (const CountMap::value_type &v, files_map) { > > 188 if (!v.second) { > > 189 HT_DEBUGF("MasterGc: removing file %s", v.first); > > 190 > > 191 if (!m_dryrun) { > > 192 try { > > 193 m_fs->remove(v.first); > > 194 ++nf_done; > > 195 } > > 196 catch (Exception &e) { > > 197 HT_WARNF("%s", e.what()); > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ THIS LINE IS Logging-MESSAGE > > COME FROM > > 198 } > > 199 } > > 200 ++nf; > > 201 } > > 202 char *p = strrchr(v.first, '/'); > > 203 > > 204 if (p) { > > 205 string dir_name(v.first, p - v.first); > > 206 insert_file(dirs_map, dir_name.c_str(), v.second); > > 207 } > > 208 } > > 209 foreach (const CountMap::value_type &v, dirs_map) { > > 210 if (!v.second) { > > 211 HT_DEBUGF("MasterGc: removing directory %s", v.first); > > 212 > > > I found that reap(file_map) just remove the specified files, but do > > NOT notify/mark the status of deletion to anyone else. Recursively, > > the `file_map` was built in scan_metadata() function. Apparently, > > scan_metadata() does NOT check the existence of the file to delete > > when building deletion-file-list. > > > When Master.GC-ing, this reap operations would repeat. At present, I > > donot read the source code totally, I donot understand the rational of > > scan_metadata(). But the reap() do *delete* the exist file; So there > > is risk that reap() will delete some WORKING ( IN-USE) file by name > > which should be kept alive!!! > > > Any one can help me out ? > > > Thanks a lot. > > > -- kuer --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
