One more thing, If we implement this component and will aggregate information about all files, we can create failure handler which provided detail information about node persistence structure. It will be helpful for debugging node crash.
On Wed, May 16, 2018 at 12:46 PM, Dmitry Pavlov <dpavlov....@gmail.com> wrote: > Andrey Gura expressed in words what I also thought. > > I agree if we extract common code to some kind of file utils. > > In the same time creating common file operation framework is not possible > because of different nature of WAL, Page Store and other components using > files. > > ср, 16 мая 2018 г. в 11:27, Dmitriy Govorukhin < > dmitriy.govoruk...@gmail.com > >: > > > Andrey, > > > > My point is, it will be very cool if there is some component who will > know > > about persistence folder and files, > > and we can move all generic code, for work with files, to this > component. > > > > On Wed, May 16, 2018 at 12:59 AM, Pavel Kovalenko <jokse...@gmail.com> > > wrote: > > > > > Andrey, > > > > > > I think the main idea of that not about how to unify writing content to > > > file. It's about creating peristence files and folders. For other > > > persistence components (WAL, Checkpoint, etc.) it should be like > > > FileFactory object with methods like "public File > > > getOrCreatePartitionFile(...), public void commitFIle(..) and etc". So, > > all > > > logic about how files should look will be moved to centralized point > and > > it > > > will be more easier to understand our persistence structure. > > > > > > 2018-05-16 0:42 GMT+03:00 Andrey Gura <ag...@apache.org>: > > > > > > > Hi, > > > > > > > > I understand you idea but it just increases dependencies of different > > > > component from one that is in general bad practice. > > > > > > > > We have different components where each one can use different > approach > > > for > > > > file management. For example page store and WAL have different file > IO > > > > implementations due to performance reasons and we have to provide > > > different > > > > mechanics for work with files. > > > > > > > > Of course we can refactor mentioned components in more structured > > manner > > > > but we should not strongly link it with one implementation. > > > > > > > > вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dpavlov....@gmail.com>: > > > > > > > > > Hi Maxim, > > > > > > > > > > FileWriteAheadLogManager & FsyncModeFileWriteAheadLogManager > were > > > > > intentionally copy-pasted in hope we will soon delete FsyncManager. > > > > > > > > > > But it is still shows this tool works good. Probably we could > > integrate > > > > > this tool to our processes. > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > > вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <maxmu...@gmail.com>: > > > > > > > > > > > +1 also for something like "resource manager". > > > > > > > > > > > > Recently, I've found for myself sonarcloud.io tool for code > > > analisys. > > > > > It's > > > > > > free for open source project and I've made Ignite project initial > > run > > > > > [1]. > > > > > > > > > > > > I've prepeared analisys for mysefl and found a lot of duplicated > > code > > > > > > blocks [1]. Of course it's not the ideal tool but gave us > direction > > > of > > > > > > thoughts. E.g. these classes [3]: > > > > > > 1) FileWriteAheadLogManager.java > > > > > > 2) FsyncModeFileWriteAheadLogManager.java > > > > > > > > > > > > > > > > > > [1] https://sonarcloud.io/dashboard?id=org.apache. > > > > ignite%3Aapache-ignite > > > > > > [2] > > > > > > > > > > > > > > > > > https://sonarcloud.io/component_measures?id=org. > > > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks > > > > > > [3] > > > > > > > > > > > > > > > > > https://sonarcloud.io/component_measures?id=org. > > > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks& > > > > selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain% > > > > 2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache% > > > > 2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java > > > > > > > > > > > > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jokse...@gmail.com > >: > > > > > > > > > > > > > +1 to this approach, > > > > > > > > > > > > > > It can be also very helpful in failover scenarios when > something > > > > wrong > > > > > > > happened with disk. In this case we're reducing the number of > > > points > > > > of > > > > > > > failure. > > > > > > > > > > > > > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin < > > > > > > > dmitriy.govoruk...@gmail.com> > > > > > > > : > > > > > > > > > > > > > > > Hi Ignite, > > > > > > > > > > > > > > > > Do we have a general approach to work with a file and > > > directories? > > > > > > > > I see many duplication logic for write through .tmp file. > > > > > > > > > > > > > > > > For example, > > > > > > > > > > > > > > > > GridCacheDatabaseSharedManager.writeCheckpointEntry(); > > > > > > > > GridCacheDatabaseSharedManage.nodeStart(); > > > > > > > > FileWriteAheadLogManager.FileArchiver.archiveSegment(); > > > > > > > > > > > > > > > > All of these methods implement the same logic, write to tmp > > file > > > > and > > > > > > > rename > > > > > > > > to normal name. > > > > > > > > > > > > > > > > I guess, will be better if we stopping write duplication > logic > > > code > > > > > and > > > > > > > > start to consolidate all in one place. > > > > > > > > > > > > > > > > Also, I think that current approach to creating files is not > > > quite > > > > > > right > > > > > > > > faithful. Each internal component > > > > > > > > create/delete files inside himself, and nobody knows where > > which > > > > > files > > > > > > > > located. > > > > > > > > > > > > > > > > I suggest refactoring code and create something (maybe new > > > manager) > > > > > > that > > > > > > > > will know about all files inside the node. All internal > > > components > > > > > must > > > > > > > > create files only through this one. > > > > > > > > > > > > > > > > It makes help to write tests for persistence easier and > reduce > > > > > > > duplication > > > > > > > > code in working with files. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
