Re: WAL iterator unexpected behavior

[Dmitriy Govorukhin]

Dmitriy,

Sorry, my mistake, I meant unknown exception of course.

On Sun, Jun 3, 2018 at 11:53 PM, Dmitriy Setrakyan 
wrote:

> I got a bit confused by your initial statement. So, the iterator is stopped
> in case of any exception, known or unknown. In that case, sounds good.
>
> D.
>
> On Sun, Jun 3, 2018, 12:11 Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com>
> wrote:
>
> > Dmitriy,
> >
> > The iterator will be stopped. and method "it.next()" will throw the
> > exception.
> >
> > On Sat, Jun 2, 2018 at 4:27 PM, Dmitriy Setrakyan  >
> > wrote:
> >
> > > Dmitriy, what happens in case of unknown exceptions?
> > >
> > >
> > > On Thu, May 31, 2018 at 6:35 AM, Dmitriy Govorukhin <
> > > dmitriy.govoruk...@gmail.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > I created the issue to solve this issue.
> > > >
> > > > IGNITE-8661 
> > > > WALItreater
> > > > is not stopped if can not deserialize record
> > > >
> > > > I suggest to make the following changes:
> > > > 1. We should only stop iteration on known exceptions
> > > > 2. Also, need to provide ability skip records by type or some pointer
> > for
> > > > the StandaloneWalRecordsIterator
> > > >
> > > > Comments are welcome.
> > > >
> > > > On Thu, May 31, 2018 at 12:53 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > On Wed, May 30, 2018 at 8:04 AM, Dmitriy Govorukhin <
> > > > > dmitriy.govoruk...@gmail.com> wrote:
> > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > I agree that in normal mode we should stop and report that error
> > > > > according.
> > > > > > I prefer to add ability skip records for offline mode.
> > > > > >
> > > > >
> > > > > Sounds good.
> > > > >
> > > >
> > >
> >
>

Re: WAL iterator unexpected behavior

[Dmitriy Setrakyan]

I got a bit confused by your initial statement. So, the iterator is stopped
in case of any exception, known or unknown. In that case, sounds good.

D.

On Sun, Jun 3, 2018, 12:11 Dmitriy Govorukhin 
wrote:

> Dmitriy,
>
> The iterator will be stopped. and method "it.next()" will throw the
> exception.
>
> On Sat, Jun 2, 2018 at 4:27 PM, Dmitriy Setrakyan 
> wrote:
>
> > Dmitriy, what happens in case of unknown exceptions?
> >
> >
> > On Thu, May 31, 2018 at 6:35 AM, Dmitriy Govorukhin <
> > dmitriy.govoruk...@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > I created the issue to solve this issue.
> > >
> > > IGNITE-8661 
> > > WALItreater
> > > is not stopped if can not deserialize record
> > >
> > > I suggest to make the following changes:
> > > 1. We should only stop iteration on known exceptions
> > > 2. Also, need to provide ability skip records by type or some pointer
> for
> > > the StandaloneWalRecordsIterator
> > >
> > > Comments are welcome.
> > >
> > > On Thu, May 31, 2018 at 12:53 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > wrote:
> > >
> > > > On Wed, May 30, 2018 at 8:04 AM, Dmitriy Govorukhin <
> > > > dmitriy.govoruk...@gmail.com> wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > I agree that in normal mode we should stop and report that error
> > > > according.
> > > > > I prefer to add ability skip records for offline mode.
> > > > >
> > > >
> > > > Sounds good.
> > > >
> > >
> >
>

Re: WAL iterator unexpected behavior

[Dmitriy Govorukhin]

Dmitriy,

The iterator will be stopped. and method "it.next()" will throw the
exception.

On Sat, Jun 2, 2018 at 4:27 PM, Dmitriy Setrakyan 
wrote:

> Dmitriy, what happens in case of unknown exceptions?
>
>
> On Thu, May 31, 2018 at 6:35 AM, Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com> wrote:
>
> > Folks,
> >
> > I created the issue to solve this issue.
> >
> > IGNITE-8661 
> > WALItreater
> > is not stopped if can not deserialize record
> >
> > I suggest to make the following changes:
> > 1. We should only stop iteration on known exceptions
> > 2. Also, need to provide ability skip records by type or some pointer for
> > the StandaloneWalRecordsIterator
> >
> > Comments are welcome.
> >
> > On Thu, May 31, 2018 at 12:53 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > wrote:
> >
> > > On Wed, May 30, 2018 at 8:04 AM, Dmitriy Govorukhin <
> > > dmitriy.govoruk...@gmail.com> wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > I agree that in normal mode we should stop and report that error
> > > according.
> > > > I prefer to add ability skip records for offline mode.
> > > >
> > >
> > > Sounds good.
> > >
> >
>

Re: WAL iterator unexpected behavior

[Dmitriy Setrakyan]

Dmitriy, what happens in case of unknown exceptions?


On Thu, May 31, 2018 at 6:35 AM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Folks,
>
> I created the issue to solve this issue.
>
> IGNITE-8661 
> WALItreater
> is not stopped if can not deserialize record
>
> I suggest to make the following changes:
> 1. We should only stop iteration on known exceptions
> 2. Also, need to provide ability skip records by type or some pointer for
> the StandaloneWalRecordsIterator
>
> Comments are welcome.
>
> On Thu, May 31, 2018 at 12:53 AM, Dmitriy Setrakyan  >
> wrote:
>
> > On Wed, May 30, 2018 at 8:04 AM, Dmitriy Govorukhin <
> > dmitriy.govoruk...@gmail.com> wrote:
> >
> > > Dmitriy,
> > >
> > > I agree that in normal mode we should stop and report that error
> > according.
> > > I prefer to add ability skip records for offline mode.
> > >
> >
> > Sounds good.
> >
>

Re: WAL iterator unexpected behavior

[Dmitriy Govorukhin]

Folks,

I created the issue to solve this issue.

IGNITE-8661  WALItreater
is not stopped if can not deserialize record

I suggest to make the following changes:
1. We should only stop iteration on known exceptions
2. Also, need to provide ability skip records by type or some pointer for
the StandaloneWalRecordsIterator

Comments are welcome.

On Thu, May 31, 2018 at 12:53 AM, Dmitriy Setrakyan 
wrote:

> On Wed, May 30, 2018 at 8:04 AM, Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com> wrote:
>
> > Dmitriy,
> >
> > I agree that in normal mode we should stop and report that error
> according.
> > I prefer to add ability skip records for offline mode.
> >
>
> Sounds good.
>

Re: WAL iterator unexpected behavior

[Dmitriy Setrakyan]

On Wed, May 30, 2018 at 8:04 AM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Dmitriy,
>
> I agree that in normal mode we should stop and report that error according.
> I prefer to add ability skip records for offline mode.
>

Sounds good.

Re: WAL iterator unexpected behavior

[Dmitriy Govorukhin]

Dmitriy,

I agree that in normal mode we should stop and report that error according.
I prefer to add ability skip records for offline mode.

On Wed, May 30, 2018 at 5:24 PM, Dmitriy Setrakyan 
wrote:

> Dmitriy,
>
> I think the behavior for offline and online iterator is fundamentally
> different. I do not think it is OK to skip records during normal operation.
> In my view, we should report an error and stop. However, when offline, it
> is OK to report an error and continue in my view.
>
> D.
>
> On Wed, May 30, 2018 at 5:39 AM, Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com> wrote:
>
> > Yakov,
> >
> > This problem is not for offline only, it applicable for all types of the
> > iterators.
> > In general, iterator does not know which class will be needed for
> > deserialization.
> >
> > I guess we can expand WAL Iterator factory and provide a method for
> > creating the iterator with some filter,
> > which will accept predicate for skipping no interesting records. It will
> > fix 1 problem.
> >
> >
> > On Wed, May 30, 2018 at 3:18 PM, Yakov Zhdanov 
> > wrote:
> >
> > > This is for offline WAL analysis. So skipping record with proper
> message
> > is
> > > also a solution. If it is possible, iterator should output a suggestion
> > on
> > > what is missing in classpath. Option to suppress warnings should also
> > > present.
> > >
> > > Makes sense?
> > >
> > > And final question - did we look at similar utilities from other
> vendors?
> > >
> > > --Yakov
> > >
> >
>

Re: WAL iterator unexpected behavior

[Dmitriy Setrakyan]

Dmitriy,

I think the behavior for offline and online iterator is fundamentally
different. I do not think it is OK to skip records during normal operation.
In my view, we should report an error and stop. However, when offline, it
is OK to report an error and continue in my view.

D.

On Wed, May 30, 2018 at 5:39 AM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Yakov,
>
> This problem is not for offline only, it applicable for all types of the
> iterators.
> In general, iterator does not know which class will be needed for
> deserialization.
>
> I guess we can expand WAL Iterator factory and provide a method for
> creating the iterator with some filter,
> which will accept predicate for skipping no interesting records. It will
> fix 1 problem.
>
>
> On Wed, May 30, 2018 at 3:18 PM, Yakov Zhdanov 
> wrote:
>
> > This is for offline WAL analysis. So skipping record with proper message
> is
> > also a solution. If it is possible, iterator should output a suggestion
> on
> > what is missing in classpath. Option to suppress warnings should also
> > present.
> >
> > Makes sense?
> >
> > And final question - did we look at similar utilities from other vendors?
> >
> > --Yakov
> >
>

Re: WAL iterator unexpected behavior

[Dmitriy Govorukhin]

Yakov,

This problem is not for offline only, it applicable for all types of the
iterators.
In general, iterator does not know which class will be needed for
deserialization.

I guess we can expand WAL Iterator factory and provide a method for
creating the iterator with some filter,
which will accept predicate for skipping no interesting records. It will
fix 1 problem.


On Wed, May 30, 2018 at 3:18 PM, Yakov Zhdanov  wrote:

> This is for offline WAL analysis. So skipping record with proper message is
> also a solution. If it is possible, iterator should output a suggestion on
> what is missing in classpath. Option to suppress warnings should also
> present.
>
> Makes sense?
>
> And final question - did we look at similar utilities from other vendors?
>
> --Yakov
>

Re: WAL iterator unexpected behavior

[Yakov Zhdanov]

This is for offline WAL analysis. So skipping record with proper message is
also a solution. If it is possible, iterator should output a suggestion on
what is missing in classpath. Option to suppress warnings should also
present.

Makes sense?

And final question - did we look at similar utilities from other vendors?

--Yakov

Re: WAL iterator unexpected behavior

[Dmitriy Setrakyan]

Dmitriy,

Thanks for initiating the discussion!

I am not sure I understand the issue fully, but in my view we should not
allow iteration through WAL if we cannot deserialize a portion of it. We
must require that ignite-index is in the classpath, so I would just fail
right away with exception.

D.


On Tue, May 29, 2018 at 6:53 AM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Igniters,
>
> I faced a problem with iterate over WAL.
>
> Let's imagine that we write WAL and write some record which depends from
> ignite-index.
> And then move files to another machine, which does not have ignite-index in
> the classpath.
> Now try to iterate over WAL.
>
> If iterator can not deserialize some record he try to go next and
> execute advanceSegment() for switching to the next segment.
> (ignite-index not in the classpath)
>
> 1. Iterator skips part of the WAL because advanceRecord() failed to
> deserialize record.
> 2. Iterator does not signal that there was an error.
>
> In my opinion, it is not that we expected.
>
> Any comments?
>

WAL iterator unexpected behavior

[Dmitriy Govorukhin]

Igniters,

I faced a problem with iterate over WAL.

Let's imagine that we write WAL and write some record which depends from
ignite-index.
And then move files to another machine, which does not have ignite-index in
the classpath.
Now try to iterate over WAL.

If iterator can not deserialize some record he try to go next and
execute advanceSegment() for switching to the next segment.
(ignite-index not in the classpath)

1. Iterator skips part of the WAL because advanceRecord() failed to
deserialize record.
2. Iterator does not signal that there was an error.

In my opinion, it is not that we expected.

Any comments?