Dmitriy, Thank you for your feedback I've created IGNITE-6421 for this change.
Is it common practice to use property file in Ignite utils? I'd like to support parse and display data entries and its fields, but it is required to configure several folders: 'binary_meta' and 'marshaller' folders. My idea is to provide .conf file to configure reader once. Sincerely, Dmitriy Pavlov пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan <[email protected]>: > Thanks! > > After looking at the ticket, the way to launch this utility seems complex. > Does this utility have a shell script? If not, we should add one with > proper command line options, like we do for other scripts. > > D. > > On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev < > [email protected]> wrote: > > > Example of output (will add to wiki): > > > > [W] InitNewPageRecord [newPageId=0001001c00000067, super=PageDeltaRecord > > [grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41, > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41, > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]] > > [W] InitNewPageRecord [newPageId=0001001c00000068, super=PageDeltaRecord > > [grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41, > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41, > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]] > > [W] PagesListAddPageRecord [dataPageId=0001001c0000002a, > > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068, > > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a, > > effectivePageId=0000001c0000002a, grpId=2141373875], page = [ > > Header [ > > type=1 (DataPageIO), > > ver=1, > > crc=0, > > pageId=281595235794986(offset=0, flags=1, partId=28, index=42) > > ], > > DataPageIO [ > > 0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673, > > 3626][][free=3552, actualFree=-544 > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1], > > rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875, > > pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0, > > pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false], > > type=BTREE_PAGE_INSERT]]] > > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE, > > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450, > > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100, > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100, > > forceFlush=false], type=DATA_RECORD]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a, > > effectivePageId=000000000000000a, grpId=2141373875], page = [ > > Header [ > > type=20 (PagePartitionCountersIO), > > ver=1, > > crc=0, > > pageId=281474976710666(offset=0, flags=1, partId=0, index=10) > > ], > > PagePartitionCounters [ > > count=1, > > lastFlag=true, > > nextCountersPageId=0000000000000000, > > size={ > > -1368047377=313 > > } > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010, > > effectivePageId=0000000200000010, grpId=2141373875], page = [ > > Header [ > > type=20 (PagePartitionCountersIO), > > ver=1, > > crc=0, > > pageId=281483566645264(offset=0, flags=1, partId=2, index=16) > > ], > > PagePartitionCounters [ > > count=1, > > lastFlag=true, > > nextCountersPageId=0000000000000000, > > size={ > > -1368047377=313 > > } > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d, > > effectivePageId=000000050000000d, grpId=2141373875], page = [ > > Header [ > > type=20 (PagePartitionCountersIO), > > ver=1, > > crc=0, > > pageId=281496451547149(offset=0, flags=1, partId=5, index=13) > > ], > > PagePartitionCounters [ > > count=1, > > lastFlag=true, > > nextCountersPageId=0000000000000000, > > size={ > > -1368047377=313 > > } > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003, > > effectivePageId=0000000900000003, grpId=2141373875], page = [ > > Header [ > > type=12 (PagesListMetaIO), > > ver=1, > > crc=0, > > pageId=281513631416323(offset=0, flags=1, partId=9, index=3) > > ], > > PagesListMeta [ > > nextMetaPageId=0000000000000000, > > count=230, > > bucketData={ > > bucket=163, list=GridLongList [idx=6, > > arr=[281513631416558,281513631416559,281513631416560,281513631416561, > > 281513631416562,281513631416563]] > > bucket=166, list=GridLongList [idx=8, > > arr=[281513631416550,281513631416551,281513631416552,281513631416553, > > 281513631416554,281513631416555,281513631416556,281513631416557]] > > bucket=169, list=GridLongList [idx=8, > > arr=[281513631416542,281513631416543,281513631416544,281513631416545, > > 281513631416546,281513631416547,281513631416548,281513631416549]] > > bucket=172, list=GridLongList [idx=8, > > arr=[281513631416534,281513631416535,281513631416536,281513631416537, > > 281513631416538,281513631416539,281513631416540,281513631416541]] > > bucket=175, list=GridLongList [idx=8, > > arr=[281513631416526,281513631416527,281513631416528,281513631416529, > > 281513631416530,281513631416531,281513631416532,281513631416533]] > > bucket=178, list=GridLongList [idx=8, > > arr=[281513631416512,281513631416513,281513631416520,281513631416521, > > 281513631416522,281513631416523,281513631416524,281513631416525]] > > bucket=181, list=GridLongList [idx=8, > > arr=[281513631416510,281513631416511,281513631416514,281513631416515, > > 281513631416516,281513631416517,281513631416518,281513631416519]] > > bucket=184, list=GridLongList [idx=8, > > arr=[281513631416500,281513631416503,281513631416504,281513631416505, > > 281513631416506,281513631416507,281513631416508,281513631416509]] > > bucket=187, list=GridLongList [idx=8, > > arr=[281513631416494,281513631416495,281513631416496,281513631416497, > > 281513631416498,281513631416499,281513631416501,281513631416502]] > > bucket=190, list=GridLongList [idx=8, > > arr=[281513631416484,281513631416485,281513631416486,281513631416487, > > 281513631416488,281513631416489,281513631416492,281513631416493]] > > bucket=193, list=GridLongList [idx=8, > > arr=[281513631416448,281513631416449,281513631416480,281513631416481, > > 281513631416482,281513631416483,281513631416490,281513631416491]] > > bucket=196, list=GridLongList [idx=8, > > arr=[281513631416446,281513631416447,281513631416474,281513631416475, > > 281513631416476,281513631416477,281513631416478,281513631416479]] > > bucket=199, list=GridLongList [idx=8, > > arr=[281513631416444,281513631416445,281513631416468,281513631416469, > > 281513631416470,281513631416471,281513631416472,281513631416473]] > > bucket=202, list=GridLongList [idx=8, > > arr=[281513631416436,281513631416437,281513631416462,281513631416463, > > 281513631416464,281513631416465,281513631416466,281513631416467]] > > bucket=205, list=GridLongList [idx=8, > > arr=[281513631416434,281513631416435,281513631416456,281513631416457, > > 281513631416458,281513631416459,281513631416460,281513631416461]] > > bucket=209, list=GridLongList [idx=8, > > arr=[281513631416426,281513631416427,281513631416450,281513631416451, > > 281513631416452,281513631416453,281513631416454,281513631416455]] > > bucket=212, list=GridLongList [idx=8, > > arr=[281513631416418,281513631416419,281513631416438,281513631416439, > > 281513631416440,281513631416441,281513631416442,281513631416443]] > > bucket=215, list=GridLongList [idx=8, > > arr=[281513631416416,281513631416417,281513631416428,281513631416429, > > 281513631416430,281513631416431,281513631416432,281513631416433]] > > bucket=218, list=GridLongList [idx=8, > > arr=[281513631416414,281513631416415,281513631416420,281513631416421, > > 281513631416422,281513631416423,281513631416424,281513631416425]] > > bucket=221, list=GridLongList [idx=8, > > arr=[281513631416406,281513631416407,281513631416408,281513631416409, > > 281513631416410,281513631416411,281513631416412,281513631416413]] > > bucket=224, list=GridLongList [idx=8, > > arr=[281513631416398,281513631416399,281513631416400,281513631416401, > > 281513631416402,281513631416403,281513631416404,281513631416405]] > > bucket=227, list=GridLongList [idx=8, > > arr=[281513631416390,281513631416391,281513631416392,281513631416393, > > 281513631416394,281513631416395,281513631416396,281513631416397]] > > bucket=230, list=GridLongList [idx=8, > > arr=[281513631416381,281513631416382,281513631416383,281513631416384, > > 281513631416385,281513631416386,281513631416387,281513631416388]] > > bucket=233, list=GridLongList [idx=8, > > arr=[281513631416373,281513631416374,281513631416375,281513631416376, > > 281513631416377,281513631416378,281513631416379,281513631416380]] > > bucket=236, list=GridLongList [idx=8, > > arr=[281513631416365,281513631416366,281513631416367,281513631416368, > > 281513631416369,281513631416370,281513631416371,281513631416372]] > > bucket=239, list=GridLongList [idx=8, > > arr=[281513631416351,281513631416352,281513631416359,281513631416360, > > 281513631416361,281513631416362,281513631416363,281513631416364]] > > bucket=242, list=GridLongList [idx=8, > > arr=[281513631416348,281513631416349,281513631416353,281513631416354, > > 281513631416355,281513631416356,281513631416357,281513631416358]] > > bucket=245, list=GridLongList [idx=8, > > arr=[281513631416335,281513631416336,281513631416337,281513631416338, > > 281513631416339,281513631416340,281513631416346,281513631416347]] > > bucket=248, list=GridLongList [idx=8, > > arr=[281513631416326,281513631416327,281513631416328,281513631416329, > > 281513631416330,281513631416331,281513631416343,281513631416344]] > > } > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d, > > effectivePageId=000000090000000d, grpId=2141373875], page = [ > > Header [ > > type=20 (PagePartitionCountersIO), > > ver=1, > > crc=0, > > pageId=281513631416333(offset=0, flags=1, partId=9, index=13) > > ], > > PagePartitionCounters [ > > count=1, > > lastFlag=true, > > nextCountersPageId=0000000000000000, > > size={ > > -1368047377=198 > > } > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000, > > effectivePageId=0000000900000000, grpId=2141373875], page = [ > > Header [ > > type=14 (PagePartitionMetaIO), > > ver=1, > > crc=0, > > pageId=281513631416320(offset=0, flags=1, partId=9, index=0) > > ], > > PageMeta[ > > treeRoot=281513631416323, > > lastSuccessfulFullSnapshotId=0, > > lastSuccessfulSnapshotId=0, > > nextSnapshotTag=1, > > lastSuccessfulSnapshotTag=0, > > lastAllocatedPageCount=0, > > candidatePageCount=0 > > ], > > PagePartitionMeta[ > > size=198, > > updateCounter=198, > > globalRemoveId=1505412422861000000, > > partitionState=1(OWNING), > > countersPageId=281513631416333 > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c, > > effectivePageId=0000000b0000000c, grpId=2141373875], page = [ > > Header [ > > type=20 (PagePartitionCountersIO), > > ver=1, > > crc=0, > > pageId=281522221350924(offset=0, flags=1, partId=11, index=12) > > ], > > PagePartitionCounters [ > > count=1, > > lastFlag=true, > > nextCountersPageId=0000000000000000, > > size={ > > -1368047377=313 > > } > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f, > > effectivePageId=0000000c0000000f, grpId=2141373875], page = [ > > Header [ > > type=20 (PagePartitionCountersIO), > > ver=1, > > crc=0, > > pageId=281526516318223(offset=0, flags=1, partId=12, index=15) > > ], > > PagePartitionCounters [ > > count=1, > > lastFlag=true, > > nextCountersPageId=0000000000000000, > > size={ > > -1368047377=313 > > } > > ]], > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1, > > fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]] > > > > > > On Mon, Sep 18, 2017 at 1:10 PM, Dmitry Pavlov <[email protected]> > > wrote: > > > > > Hi Igniters, > > > > > > Page https://cwiki.apache.org/confluence/display/IGNITE/ > > Ignite+WAL+reader > > > to > > > collect related information about this prototype was created. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <[email protected] > >: > > > > > > > Can we get a sample print out from this utility? > > > > > > > > On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev < > > > > [email protected]> wrote: > > > > > > > > > Dmitry, > > > > > thank you for starting this topic. > > > > > > > > > > The IDEA of this utility is very simple - get the possibility to > look > > > > into > > > > > your WAL. > > > > > But it also requires understanding what's going on in WAL mechanism > > (We > > > > > need to describe it in more details in the wiki). > > > > > > > > > > It could be useful for a debug some weird things with recovery > > > mechanism. > > > > > Initially, it was created for dealing with the issue in > IGNITE-6342. > > It > > > > > helped to find out the error in how we deal with partition size. > > > > > > > > > > Best regards, > > > > > Ed. > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Denis, > > > > > > > > > > > > Thank you for your feedback. It is very important to us to > > understand > > > > > that > > > > > > this tool may be useful not only for Ignite tests but for all > > > > developers. > > > > > > > > > > > > I will create wiki page. > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[email protected]>: > > > > > > > > > > > > > Dmitriy, Eduard, > > > > > > > > > > > > > > Sounds like a useful thing. I’ll be happy to give a feedback > but > > > > need a > > > > > > > documentation that explains how to use the tool, what’s the > > output > > > > > > format, > > > > > > > how do decipher the output, etc (basic things). The Wiki > > > description > > > > > > should > > > > > > > be enough. > > > > > > > > > > > > > > — > > > > > > > Denis > > > > > > > > > > > > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > > > > > Eduard Shangareev created WAL converter utility which shows > WAL > > > > > > > > archive/work directory segments content in human readable > form. > > > > This > > > > > > > > utility outputs all WAL records to output stream (may be > > > redirected > > > > > to > > > > > > > > file). How to use tips can be found in issue > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6277 > > > > > > > > > > > > > > > > > > > > > > > > This utility does not require Ignite node to be up and > running > > > and > > > > > may > > > > > > be > > > > > > > > used for offline analysis of copied files. Thanks to Alexey > G. > > > for > > > > > idea > > > > > > > and > > > > > > > > review. Code is now available in modules/ignite-dev-tools. > > > > > > > > > > > > > > > > 1. Could you please write if utility can be useful for you? > > > > > > > > > > > > > > > > 2. Please respond if you need short wiki description. > > > > > > > > > > > > > > > > 3. What is your opinion what should be next steps to develop > > this > > > > > > > utility? > > > > > > > > > > > > > > > > 4. Does utility need property file to refer Ignite Work > > directory > > > > > using > > > > > > > > conf file. > > > > > > > > > > > > > > > > 5. What do you think if utility remains console non > > interactive, > > > or > > > > > > some > > > > > > > > UI/Interactive console shell may be preferable? > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > > > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
