[jira] [Comment Edited] (OAK-4939) Isolate corrupted index and make async indexer more resilient
[ https://issues.apache.org/jira/browse/OAK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840322#comment-15840322 ] Alexander Klimetschek edited comment on OAK-4939 at 1/26/17 7:56 PM: - I added an enhancement request in OAK-5519 to be more specific and skip individual binaries/blobs that are creating issues for the indexing (blob can't be found, text extraction throws error etc.), so that the index isn't completely isolated. If the problem is a binary, then it should be isolated there, and wouldn't mean the index itself is corrupted. Of course, it makes sense to track these binaries and see if they can be indexed later when the underlying issue might have been resolved (e.g. manually through by an admin). was (Author: alexander.klimetschek): I add an enhancement request in OAK-5519 to be more specific and skip individual binaries/blobs that are creating issues for the indexing (blob can't be found, text extraction throws error etc.), so that the index isn't completely isolated. If the problem is a binary, then it should be isolated there, and wouldn't mean the index itself is corrupted. Of course, it makes sense to track these binaries and see if they can be indexed later when the underlying issue might have been resolved (e.g. manually through by an admin). > Isolate corrupted index and make async indexer more resilient > - > > Key: OAK-4939 > URL: https://issues.apache.org/jira/browse/OAK-4939 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, query >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.5.15, 1.6 > > Attachments: corrupt-index-mbean.png, OAK-4939-v1.diff, > OAK-4939-v2.diff > > > Currently if any one of the async index gets corrupted it brings down the > whole async indexer and no other index gets updated untill system > administrator reindexes the problamatic async index. > Instead of fail all we should isolate such corrupted index and mark them as > corrupted. And still let async indexer progress for other working indexes. > This would ensure that one corrupted index does not affect the whole system > and allow the application to work partially. > Feature branch - > https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-4939?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4939) Isolate corrupted index and make async indexer more resilient
[
https://issues.apache.org/jira/browse/OAK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686435#comment-15686435
]
Chetan Mehrotra edited comment on OAK-4939 at 11/22/16 11:18 AM:
-
Can be done. For now {{failing}} boolean attribute of IndexStatsMBean would be
set to true if index is failing
was (Author: chetanm):
Can be done. For now {{failing}} boolean attribute would be set to true if
index is failing
> Isolate corrupted index and make async indexer more resilient
> -
>
> Key: OAK-4939
> URL: https://issues.apache.org/jira/browse/OAK-4939
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene, query
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6, 1.5.15
>
> Attachments: OAK-4939-v1.diff, corrupt-index-mbean.png
>
>
> Currently if any one of the async index gets corrupted it brings down the
> whole async indexer and no other index gets updated untill system
> administrator reindexes the problamatic async index.
> Instead of fail all we should isolate such corrupted index and mark them as
> corrupted. And still let async indexer progress for other working indexes.
> This would ensure that one corrupted index does not affect the whole system
> and allow the application to work partially.
> Feature branch -
> https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-4939?expand=1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4939) Isolate corrupted index and make async indexer more resilient
[ https://issues.apache.org/jira/browse/OAK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686263#comment-15686263 ] Chetan Mehrotra edited comment on OAK-4939 at 11/22/16 9:59 AM: [~alex.parvulescu] [~catholicon] [~tmueller] Please review and provide feedback on implemented approach. Commits message in branch provide more details around implementation details was (Author: chetanm): [~alex.parvulescu] [~catholicon] [~tmueller] Please review and provide feedback on implemented approach. > Isolate corrupted index and make async indexer more resilient > - > > Key: OAK-4939 > URL: https://issues.apache.org/jira/browse/OAK-4939 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, query >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.6 > > Attachments: OAK-4939-v1.diff, corrupt-index-mbean.png > > > Currently if any one of the async index gets corrupted it brings down the > whole async indexer and no other index gets updated untill system > administrator reindexes the problamatic async index. > Instead of fail all we should isolate such corrupted index and mark them as > corrupted. And still let async indexer progress for other working indexes. > This would ensure that one corrupted index does not affect the whole system > and allow the application to work partially. > Feature branch - > https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-4939?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4939) Isolate corrupted index and make async indexer more resilient
[
https://issues.apache.org/jira/browse/OAK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581189#comment-15581189
]
Chetan Mehrotra edited comment on OAK-4939 at 10/17/16 4:43 AM:
One way to implement this
# {{IndexUpdate}} while making call to editor looks for any exception thrown.
In case of any exception received for any specific editor it would set
{{corrupted}} to {{true}} for that index definition and let that indexing cycle
fail
# Upon next cycle run it would ignore such corrupted indexes and only work with
editor instance for working indexes.
# However it would log a warning for such corrupted index mentioning that
system admin should reindex.
# In addition the IndexStatsMBean would also expose such corrupted index so
monitoring system can look for that and issue required alerts
# Upon reindexing the {{IndexUpdate}} would clear the {{corrupted}} flag
# Async indexer MBean would have a new operation which allows system admin to
do an indexing run which include such corrupted index to re check what is the
failure cause
Such an approach would ensure that even if one of the index gets corrupted the
asyn index continue to work fine for other indexes. So only queries which make
use of such corrupted index would suffer instead of the whole application. This
would be specially useful for system where an index is being created per tenant
[~alexparvulescu] [~catholicon] [~tmueller] Thoughts?
was (Author: chetanm):
One way to implement this
# {{IndexUpdate}} while making call to editor looks for any exception thrown.
In case of any exception received for any specific editor it would set
{{corrupted}} to {{true}} for that index definition and let that indexing cycle
fail
# Upon next cycle run it would ignore such corrupted indexes and only work with
editor instance for working indexes.
# However it would log a warning for such corrupted index mentioning that
system admin should reindex.
# In addition the IndexStatsMBean would also expose such corrupted index so
monitoring system can look for that and issue required alerts
# Upon reindexing the {{IndexUpdate}} would clear the {{corrupted}} flag
Such an approach would ensure that even if one of the index gets corrupted the
asyn index continue to work fine for other indexes. So only queries which make
use of such corrupted index would suffer instead of the whole application. This
would be specially useful for system where an index is being created per tenant
[~alexparvulescu] [~catholicon] [~tmueller] Thoughts?
> Isolate corrupted index and make async indexer more resilient
> -
>
> Key: OAK-4939
> URL: https://issues.apache.org/jira/browse/OAK-4939
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene, query
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
>
> Currently if any one of the async index gets corrupted it brings down the
> whole async indexer and no other index gets updated untill system
> administrator reindexes the problamatic async index.
> Instead of fail all we should isolate such corrupted index and mark them as
> corrupted. And still let async indexer progress for other working indexes.
> This would ensure that one corrupted index does not affect the whole system
> and allow the application to work partially.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4939) Isolate corrupted index and make async indexer more resilient
[
https://issues.apache.org/jira/browse/OAK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581189#comment-15581189
]
Chetan Mehrotra edited comment on OAK-4939 at 10/17/16 4:41 AM:
One way to implement this
# {{IndexUpdate}} while making call to editor looks for any exception thrown.
In case of any exception received for any specific editor it would set
{{corrupted}} to {{true}} for that index definition and let that indexing cycle
fail
# Upon next cycle run it would ignore such corrupted indexes and only work with
editor instance for working indexes.
# However it would log a warning for such corrupted index mentioning that
system admin should reindex.
# In addition the IndexStatsMBean would also expose such corrupted index so
monitoring system can look for that and issue required alerts
# Upon reindexing the {{IndexUpdate}} would clear the {{corrupted}} flag
Such an approach would ensure that even if one of the index gets corrupted the
asyn index continue to work fine for other indexes. So only queries which make
use of such corrupted index would suffer instead of the whole application. This
would be specially useful for system where an index is being created per tenant
[~alexparvulescu] [~catholicon] [~tmueller] Thoughts?
was (Author: chetanm):
One way to implement this
# {{IndexUpdate}} while making call to editor looks for any exception thrown.
In case of any exception received for any specific editor it would set
{{corrupted}} to {{true}} for that index definition and let that indexing cycle
fail
# Upon next cycle run it would ignore such corrupted indexes and only work with
editor instance for working indexes.
# However it would log a warning for such corrupted index mentioning that
system admin should reindex.
# In addition the IndexStatsMBean would also expose such corrupted index so
monitoring system can look for that and issue required alerts
Such an approach would ensure that even if one of the index gets corrupted the
asyn index continue to work fine for other indexes. So only queries which make
use of such corrupted index would suffer instead of the whole application. This
would be specially useful for system where an index is being created per tenant
[~alexparvulescu] [~catholicon] [~tmueller] Thoughts?
> Isolate corrupted index and make async indexer more resilient
> -
>
> Key: OAK-4939
> URL: https://issues.apache.org/jira/browse/OAK-4939
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene, query
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
>
> Currently if any one of the async index gets corrupted it brings down the
> whole async indexer and no other index gets updated untill system
> administrator reindexes the problamatic async index.
> Instead of fail all we should isolate such corrupted index and mark them as
> corrupted. And still let async indexer progress for other working indexes.
> This would ensure that one corrupted index does not affect the whole system
> and allow the application to work partially.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
