subject:"Re\: Ignite PDS WAL analysis with human readable records"

Re: Ignite PDS WAL analysis with human readable records

2017-09-19 Thread Dmitry Pavlov

I've updated issue description.

Igniters,

who have already faced with necessity to look into WAL content and/or
latest records?

Sincerely,
Dmitriy Pavlov

пн, 18 сент. 2017 г. в 23:23, Dmitriy Setrakyan :

> Dmitriy, can you please describe your idea in the ticket?
>
> Here is the link for easier access:
> https://issues.apache.org/jira/browse/IGNITE-6421
>
> D.
>
> On Mon, Sep 18, 2017 at 7:02 AM, Dmitry Pavlov 
> wrote:
>
> > Dmitriy,
> >
> > Thank you for your feedback I've created IGNITE-6421 for this change.
> >
> > Is it common practice to use property file in Ignite utils? I'd like to
> > support parse and display data entries and its fields, but it is required
> > to configure several folders: 'binary_meta' and 'marshaller' folders.
> >
> > My idea is to provide .conf file to configure reader once.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan :
> >
> > > Thanks!
> > >
> > > After looking at the ticket, the way to launch this utility seems
> > complex.
> > > Does this utility have a shell script? If not, we should add one with
> > > proper command line options, like we do for other scripts.
> > >
> > > D.
> > >
> > > On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
> > > eduard.shangar...@gmail.com> wrote:
> > >
> > > > Example of output (will add to wiki):
> > > >
> > > > [W] InitNewPageRecord [newPageId=0001001c0067,
> > super=PageDeltaRecord
> > > > [grpId=2141373875, pageId=0001001c0067, super=WALRecord [size=41,
> > > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> > > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > > [W] InitNewPageRecord [newPageId=0001001c0068,
> > super=PageDeltaRecord
> > > > [grpId=2141373875, pageId=0001001c0068, super=WALRecord [size=41,
> > > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> > > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > > [W] PagesListAddPageRecord [dataPageId=0001001c002a,
> > > > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c0068,
> > > > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> > > > fileOffset=25488, len=37, forceFlush=false],
> > type=PAGES_LIST_ADD_PAGE]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c002a,
> > > > effectivePageId=001c002a, grpId=2141373875], page = [
> > > > Header [
> > > > type=1 (DataPageIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> > > > ],
> > > > DataPageIO [
> > > > 0001001c002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720,
> 3673,
> > > > 3626][][free=3552, actualFree=-544
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> > > > rightId=, super=PageDeltaRecord [grpId=2141373875,
> > > > pageId=0001001c0004, super=WALRecord [size=59, chainSize=0,
> > > > pos=FileWALPointer [idx=1, fileOffset=29650, len=59,
> forceFlush=false],
> > > > type=BTREE_PAGE_INSERT]]]
> > > > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377,
> > op=CREATE,
> > > > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> > > > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> > > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> > > > forceFlush=false], type=DATA_RECORD]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000a,
> > > > effectivePageId=000a, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=,
> > > > size={
> > > > -1368047377=313
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100020010,
> > > > effectivePageId=00020010, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=,
> > > > size={
> > > > -1368047377=313
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=00010005000d,
> > > >

Re: Ignite PDS WAL analysis with human readable records

2017-09-18 Thread Dmitriy Setrakyan

Dmitriy, can you please describe your idea in the ticket?

Here is the link for easier access:
https://issues.apache.org/jira/browse/IGNITE-6421

D.

On Mon, Sep 18, 2017 at 7:02 AM, Dmitry Pavlov 
wrote:

> Dmitriy,
>
> Thank you for your feedback I've created IGNITE-6421 for this change.
>
> Is it common practice to use property file in Ignite utils? I'd like to
> support parse and display data entries and its fields, but it is required
> to configure several folders: 'binary_meta' and 'marshaller' folders.
>
> My idea is to provide .conf file to configure reader once.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan :
>
> > Thanks!
> >
> > After looking at the ticket, the way to launch this utility seems
> complex.
> > Does this utility have a shell script? If not, we should add one with
> > proper command line options, like we do for other scripts.
> >
> > D.
> >
> > On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
> > eduard.shangar...@gmail.com> wrote:
> >
> > > Example of output (will add to wiki):
> > >
> > > [W] InitNewPageRecord [newPageId=0001001c0067,
> super=PageDeltaRecord
> > > [grpId=2141373875, pageId=0001001c0067, super=WALRecord [size=41,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > [W] InitNewPageRecord [newPageId=0001001c0068,
> super=PageDeltaRecord
> > > [grpId=2141373875, pageId=0001001c0068, super=WALRecord [size=41,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > [W] PagesListAddPageRecord [dataPageId=0001001c002a,
> > > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c0068,
> > > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=25488, len=37, forceFlush=false],
> type=PAGES_LIST_ADD_PAGE]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c002a,
> > > effectivePageId=001c002a, grpId=2141373875], page = [
> > > Header [
> > > type=1 (DataPageIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> > > ],
> > > DataPageIO [
> > > 0001001c002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
> > > 3626][][free=3552, actualFree=-544
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> > > rightId=, super=PageDeltaRecord [grpId=2141373875,
> > > pageId=0001001c0004, super=WALRecord [size=59, chainSize=0,
> > > pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
> > > type=BTREE_PAGE_INSERT]]]
> > > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377,
> op=CREATE,
> > > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> > > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> > > forceFlush=false], type=DATA_RECORD]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000a,
> > > effectivePageId=000a, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100020010,
> > > effectivePageId=00020010, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=00010005000d,
> > > effectivePageId=0005000d, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > >

Re: Ignite PDS WAL analysis with human readable records

2017-09-18 Thread Dmitry Pavlov

Dmitriy,

Thank you for your feedback I've created IGNITE-6421 for this change.

Is it common practice to use property file in Ignite utils? I'd like to
support parse and display data entries and its fields, but it is required
to configure several folders: 'binary_meta' and 'marshaller' folders.

My idea is to provide .conf file to configure reader once.

Sincerely,
Dmitriy Pavlov

пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan :

> Thanks!
>
> After looking at the ticket, the way to launch this utility seems complex.
> Does this utility have a shell script? If not, we should add one with
> proper command line options, like we do for other scripts.
>
> D.
>
> On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
> eduard.shangar...@gmail.com> wrote:
>
> > Example of output (will add to wiki):
> >
> > [W] InitNewPageRecord [newPageId=0001001c0067, super=PageDeltaRecord
> > [grpId=2141373875, pageId=0001001c0067, super=WALRecord [size=41,
> > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > [W] InitNewPageRecord [newPageId=0001001c0068, super=PageDeltaRecord
> > [grpId=2141373875, pageId=0001001c0068, super=WALRecord [size=41,
> > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > [W] PagesListAddPageRecord [dataPageId=0001001c002a,
> > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c0068,
> > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c002a,
> > effectivePageId=001c002a, grpId=2141373875], page = [
> > Header [
> > type=1 (DataPageIO),
> > ver=1,
> > crc=0,
> > pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> > ],
> > DataPageIO [
> > 0001001c002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
> > 3626][][free=3552, actualFree=-544
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> > rightId=, super=PageDeltaRecord [grpId=2141373875,
> > pageId=0001001c0004, super=WALRecord [size=59, chainSize=0,
> > pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
> > type=BTREE_PAGE_INSERT]]]
> > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE,
> > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> > forceFlush=false], type=DATA_RECORD]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000a,
> > effectivePageId=000a, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100020010,
> > effectivePageId=00020010, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=00010005000d,
> > effectivePageId=0005000d, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090003,
> > effectivePageId=00090003, grpId=2141373875], page = [
> > Header [
> > type=12 (PagesListMetaIO),
> > ver=1,
> > crc=0,
> > pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
> > ],
> > PagesListMeta [
> > nextMetaPageId=,
> > count=230,
> > bucketData={
> > bucket=163, list=GridLongList [idx=6,
> >

Re: Ignite PDS WAL analysis with human readable records

2017-09-18 Thread Dmitriy Setrakyan

Thanks!

After looking at the ticket, the way to launch this utility seems complex.
Does this utility have a shell script? If not, we should add one with
proper command line options, like we do for other scripts.

D.

On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
eduard.shangar...@gmail.com> wrote:

> Example of output (will add to wiki):
>
> [W] InitNewPageRecord [newPageId=0001001c0067, super=PageDeltaRecord
> [grpId=2141373875, pageId=0001001c0067, super=WALRecord [size=41,
> chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> [W] InitNewPageRecord [newPageId=0001001c0068, super=PageDeltaRecord
> [grpId=2141373875, pageId=0001001c0068, super=WALRecord [size=41,
> chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> [W] PagesListAddPageRecord [dataPageId=0001001c002a,
> super=PageDeltaRecord [grpId=2141373875, pageId=0001001c0068,
> super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c002a,
> effectivePageId=001c002a, grpId=2141373875], page = [
> Header [
> type=1 (DataPageIO),
> ver=1,
> crc=0,
> pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> ],
> DataPageIO [
> 0001001c002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
> 3626][][free=3552, actualFree=-544
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> rightId=, super=PageDeltaRecord [grpId=2141373875,
> pageId=0001001c0004, super=WALRecord [size=59, chainSize=0,
> pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
> type=BTREE_PAGE_INSERT]]]
> [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE,
> writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> forceFlush=false], type=DATA_RECORD]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000a,
> effectivePageId=000a, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=000100020010,
> effectivePageId=00020010, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=00010005000d,
> effectivePageId=0005000d, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090003,
> effectivePageId=00090003, grpId=2141373875], page = [
> Header [
> type=12 (PagesListMetaIO),
> ver=1,
> crc=0,
> pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
> ],
> PagesListMeta [
> nextMetaPageId=,
> count=230,
> bucketData={
> bucket=163, list=GridLongList [idx=6,
> arr=[281513631416558,281513631416559,281513631416560,281513631416561,
> 281513631416562,281513631416563]]
> bucket=166, list=GridLongList [idx=8,
> arr=[281513631416550,281513631416551,281513631416552,281513631416553,
> 281513631416554,281513631416555,281513631416556,281513631416557]]
> bucket=169, list=GridLongList [idx=8,
> arr=[281513631416542,281513631416543,281513631416544,281513631416545,
> 281513631416546,281513631416547,281513631416548,281513631416549]]
> bucket=172, list=GridLongList [idx=8,
> arr=[281513631416534,281513631416535,281513631416536,281513631416537,
> 281513631416538,281513631416539,281513631416540,281513631416541]]
> bucket=175, list=GridLongList [idx=8,
>

Re: Ignite PDS WAL analysis with human readable records

2017-09-18 Thread Eduard Shangareev

Example of output (will add to wiki):

[W] InitNewPageRecord [newPageId=0001001c0067, super=PageDeltaRecord
[grpId=2141373875, pageId=0001001c0067, super=WALRecord [size=41,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
[W] InitNewPageRecord [newPageId=0001001c0068, super=PageDeltaRecord
[grpId=2141373875, pageId=0001001c0068, super=WALRecord [size=41,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
[W] PagesListAddPageRecord [dataPageId=0001001c002a,
super=PageDeltaRecord [grpId=2141373875, pageId=0001001c0068,
super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c002a,
effectivePageId=001c002a, grpId=2141373875], page = [
Header [
type=1 (DataPageIO),
ver=1,
crc=0,
pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
],
DataPageIO [
0001001c002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
3626][][free=3552, actualFree=-544
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
rightId=, super=PageDeltaRecord [grpId=2141373875,
pageId=0001001c0004, super=WALRecord [size=59, chainSize=0,
pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
type=BTREE_PAGE_INSERT]]]
[W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE,
writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
forceFlush=false], type=DATA_RECORD]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000a,
effectivePageId=000a, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100020010,
effectivePageId=00020010, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=00010005000d,
effectivePageId=0005000d, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100090003,
effectivePageId=00090003, grpId=2141373875], page = [
Header [
type=12 (PagesListMetaIO),
ver=1,
crc=0,
pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
],
PagesListMeta [
nextMetaPageId=,
count=230,
bucketData={
bucket=163, list=GridLongList [idx=6,
arr=[281513631416558,281513631416559,281513631416560,281513631416561,281513631416562,281513631416563]]
bucket=166, list=GridLongList [idx=8,
arr=[281513631416550,281513631416551,281513631416552,281513631416553,281513631416554,281513631416555,281513631416556,281513631416557]]
bucket=169, list=GridLongList [idx=8,
arr=[281513631416542,281513631416543,281513631416544,281513631416545,281513631416546,281513631416547,281513631416548,281513631416549]]
bucket=172, list=GridLongList [idx=8,
arr=[281513631416534,281513631416535,281513631416536,281513631416537,281513631416538,281513631416539,281513631416540,281513631416541]]
bucket=175, list=GridLongList [idx=8,
arr=[281513631416526,281513631416527,281513631416528,281513631416529,281513631416530,281513631416531,281513631416532,281513631416533]]
bucket=178, list=GridLongList [idx=8,
arr=[281513631416512,281513631416513,281513631416520,281513631416521,281513631416522,281513631416523,281513631416524,281513631416525]]
bucket=181, list=GridLongList [idx=8,
arr=[281513631416510,281513631416511,281513631416514,281513631416515,281513631416516,281513631416517,281513631416518,281513631416519]]
bucket=184, list=GridLongList [idx=8,

Re: Ignite PDS WAL analysis with human readable records

2017-09-18 Thread Dmitry Pavlov

Hi Igniters,

Page https://cwiki.apache.org/confluence/display/IGNITE/Ignite+WAL+reader to
collect related information about this prototype was created.

Sincerely,
Dmitriy Pavlov

пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan :

> Can we get a sample print out from this utility?
>
> On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> eduard.shangar...@gmail.com> wrote:
>
> > Dmitry,
> > thank you for starting this topic.
> >
> > The IDEA of this utility is very simple - get the possibility to look
> into
> > your WAL.
> > But it also requires understanding what's going on in WAL mechanism (We
> > need to describe it in more details in the wiki).
> >
> > It could be useful for a debug some weird things with recovery mechanism.
> > Initially, it was created for dealing with the issue in IGNITE-6342. It
> > helped to find out the error in how we deal with partition size.
> >
> > Best regards,
> > Ed.
> >
> >
> >
> >
> > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov 
> > wrote:
> >
> > > Hi Denis,
> > >
> > > Thank you for your feedback. It is very important to us to understand
> > that
> > > this tool may be useful not only for Ignite tests but for all
> developers.
> > >
> > > I will create wiki page.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > сб, 16 сент. 2017 г. в 1:54, Denis Magda :
> > >
> > > > Dmitriy, Eduard,
> > > >
> > > > Sounds like a useful thing. I’ll be happy to give a feedback but
> need a
> > > > documentation that explains how to use the tool, what’s the output
> > > format,
> > > > how do decipher the output, etc (basic things). The Wiki description
> > > should
> > > > be enough.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov  >
> > > > wrote:
> > > > >
> > > > > Hi Igniters,
> > > > >
> > > > > Eduard Shangareev created WAL converter utility which shows WAL
> > > > > archive/work directory segments content in human readable form.
> This
> > > > > utility outputs all WAL records to output stream (may be redirected
> > to
> > > > > file). How to use tips can be found in issue
> > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > >
> > > > >
> > > > > This utility does not require Ignite node to be up and running and
> > may
> > > be
> > > > > used for offline analysis of copied files. Thanks to Alexey G. for
> > idea
> > > > and
> > > > > review. Code is now available in modules/ignite-dev-tools.
> > > > >
> > > > > 1. Could you please write if utility can be useful for you?
> > > > >
> > > > > 2. Please respond if you need short wiki description.
> > > > >
> > > > > 3. What is your opinion what should be next steps to develop this
> > > > utility?
> > > > >
> > > > > 4. Does utility need property file to refer Ignite Work directory
> > using
> > > > > conf file.
> > > > >
> > > > > 5. What do you think if utility remains console non interactive, or
> > > some
> > > > > UI/Interactive console shell may be preferable?
> > > > >
> > > > > Sincerely,
> > > > >
> > > > > Dmitriy Pavlov
> > > >
> > > >
> > >
> >
>

Re: Ignite PDS WAL analysis with human readable records

2017-09-17 Thread Dmitriy Setrakyan

Can we get a sample print out from this utility?

On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
eduard.shangar...@gmail.com> wrote:

> Dmitry,
> thank you for starting this topic.
>
> The IDEA of this utility is very simple - get the possibility to look into
> your WAL.
> But it also requires understanding what's going on in WAL mechanism (We
> need to describe it in more details in the wiki).
>
> It could be useful for a debug some weird things with recovery mechanism.
> Initially, it was created for dealing with the issue in IGNITE-6342. It
> helped to find out the error in how we deal with partition size.
>
> Best regards,
> Ed.
>
>
>
>
> On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov 
> wrote:
>
> > Hi Denis,
> >
> > Thank you for your feedback. It is very important to us to understand
> that
> > this tool may be useful not only for Ignite tests but for all developers.
> >
> > I will create wiki page.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 16 сент. 2017 г. в 1:54, Denis Magda :
> >
> > > Dmitriy, Eduard,
> > >
> > > Sounds like a useful thing. I’ll be happy to give a feedback but need a
> > > documentation that explains how to use the tool, what’s the output
> > format,
> > > how do decipher the output, etc (basic things). The Wiki description
> > should
> > > be enough.
> > >
> > > —
> > > Denis
> > >
> > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov 
> > > wrote:
> > > >
> > > > Hi Igniters,
> > > >
> > > > Eduard Shangareev created WAL converter utility which shows WAL
> > > > archive/work directory segments content in human readable form. This
> > > > utility outputs all WAL records to output stream (may be redirected
> to
> > > > file). How to use tips can be found in issue
> > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > >
> > > >
> > > > This utility does not require Ignite node to be up and running and
> may
> > be
> > > > used for offline analysis of copied files. Thanks to Alexey G. for
> idea
> > > and
> > > > review. Code is now available in modules/ignite-dev-tools.
> > > >
> > > > 1. Could you please write if utility can be useful for you?
> > > >
> > > > 2. Please respond if you need short wiki description.
> > > >
> > > > 3. What is your opinion what should be next steps to develop this
> > > utility?
> > > >
> > > > 4. Does utility need property file to refer Ignite Work directory
> using
> > > > conf file.
> > > >
> > > > 5. What do you think if utility remains console non interactive, or
> > some
> > > > UI/Interactive console shell may be preferable?
> > > >
> > > > Sincerely,
> > > >
> > > > Dmitriy Pavlov
> > >
> > >
> >
>

Re: Ignite PDS WAL analysis with human readable records

2017-09-17 Thread Eduard Shangareev

Dmitry,
thank you for starting this topic.

The IDEA of this utility is very simple - get the possibility to look into
your WAL.
But it also requires understanding what's going on in WAL mechanism (We
need to describe it in more details in the wiki).

It could be useful for a debug some weird things with recovery mechanism.
Initially, it was created for dealing with the issue in IGNITE-6342. It
helped to find out the error in how we deal with partition size.

Best regards,
Ed.




On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov 
wrote:

> Hi Denis,
>
> Thank you for your feedback. It is very important to us to understand that
> this tool may be useful not only for Ignite tests but for all developers.
>
> I will create wiki page.
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 16 сент. 2017 г. в 1:54, Denis Magda :
>
> > Dmitriy, Eduard,
> >
> > Sounds like a useful thing. I’ll be happy to give a feedback but need a
> > documentation that explains how to use the tool, what’s the output
> format,
> > how do decipher the output, etc (basic things). The Wiki description
> should
> > be enough.
> >
> > —
> > Denis
> >
> > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov 
> > wrote:
> > >
> > > Hi Igniters,
> > >
> > > Eduard Shangareev created WAL converter utility which shows WAL
> > > archive/work directory segments content in human readable form. This
> > > utility outputs all WAL records to output stream (may be redirected to
> > > file). How to use tips can be found in issue
> > > https://issues.apache.org/jira/browse/IGNITE-6277
> > >
> > >
> > > This utility does not require Ignite node to be up and running and may
> be
> > > used for offline analysis of copied files. Thanks to Alexey G. for idea
> > and
> > > review. Code is now available in modules/ignite-dev-tools.
> > >
> > > 1. Could you please write if utility can be useful for you?
> > >
> > > 2. Please respond if you need short wiki description.
> > >
> > > 3. What is your opinion what should be next steps to develop this
> > utility?
> > >
> > > 4. Does utility need property file to refer Ignite Work directory using
> > > conf file.
> > >
> > > 5. What do you think if utility remains console non interactive, or
> some
> > > UI/Interactive console shell may be preferable?
> > >
> > > Sincerely,
> > >
> > > Dmitriy Pavlov
> >
> >
>

Re: Ignite PDS WAL analysis with human readable records

2017-09-15 Thread Dmitry Pavlov

Hi Denis,

Thank you for your feedback. It is very important to us to understand that
this tool may be useful not only for Ignite tests but for all developers.

I will create wiki page.

Sincerely,
Dmitriy Pavlov

сб, 16 сент. 2017 г. в 1:54, Denis Magda :

> Dmitriy, Eduard,
>
> Sounds like a useful thing. I’ll be happy to give a feedback but need a
> documentation that explains how to use the tool, what’s the output format,
> how do decipher the output, etc (basic things). The Wiki description should
> be enough.
>
> —
> Denis
>
> > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov 
> wrote:
> >
> > Hi Igniters,
> >
> > Eduard Shangareev created WAL converter utility which shows WAL
> > archive/work directory segments content in human readable form. This
> > utility outputs all WAL records to output stream (may be redirected to
> > file). How to use tips can be found in issue
> > https://issues.apache.org/jira/browse/IGNITE-6277
> >
> >
> > This utility does not require Ignite node to be up and running and may be
> > used for offline analysis of copied files. Thanks to Alexey G. for idea
> and
> > review. Code is now available in modules/ignite-dev-tools.
> >
> > 1. Could you please write if utility can be useful for you?
> >
> > 2. Please respond if you need short wiki description.
> >
> > 3. What is your opinion what should be next steps to develop this
> utility?
> >
> > 4. Does utility need property file to refer Ignite Work directory using
> > conf file.
> >
> > 5. What do you think if utility remains console non interactive, or some
> > UI/Interactive console shell may be preferable?
> >
> > Sincerely,
> >
> > Dmitriy Pavlov
>
>

Re: Ignite PDS WAL analysis with human readable records

2017-09-15 Thread Denis Magda

Dmitriy, Eduard,

Sounds like a useful thing. I’ll be happy to give a feedback but need a 
documentation that explains how to use the tool, what’s the output format, how 
do decipher the output, etc (basic things). The Wiki description should be 
enough.

—
Denis

> On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov  wrote:
> 
> Hi Igniters,
> 
> Eduard Shangareev created WAL converter utility which shows WAL
> archive/work directory segments content in human readable form. This
> utility outputs all WAL records to output stream (may be redirected to
> file). How to use tips can be found in issue
> https://issues.apache.org/jira/browse/IGNITE-6277
> 
> 
> This utility does not require Ignite node to be up and running and may be
> used for offline analysis of copied files. Thanks to Alexey G. for idea and
> review. Code is now available in modules/ignite-dev-tools.
> 
> 1. Could you please write if utility can be useful for you?
> 
> 2. Please respond if you need short wiki description.
> 
> 3. What is your opinion what should be next steps to develop this utility?
> 
> 4. Does utility need property file to refer Ignite Work directory using
> conf file.
> 
> 5. What do you think if utility remains console non interactive, or some
> UI/Interactive console shell may be preferable?
> 
> Sincerely,
> 
> Dmitriy Pavlov

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

Re: Ignite PDS WAL analysis with human readable records

10 matches

Site Navigation

Mail list logo

Footer information