[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-16 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206943#comment-16206943
 ] 

ASF subversion and git services commented on SOLR-11443:


Commit 58730dcd6751427cf901552b4453eb817dfc631c in lucene-solr's branch 
refs/heads/branch_7x from [~caomanhdat]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=58730dc ]

SOLR-11443: Update CHANGES.txt


> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-16 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206942#comment-16206942
 ] 

ASF subversion and git services commented on SOLR-11443:


Commit 9fac59ef55a134ff363c8bc4f0e5589769cf4962 in lucene-solr's branch 
refs/heads/master from [~caomanhdat]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9fac59e ]

SOLR-11443: Update CHANGES.txt


> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-16 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205626#comment-16205626
 ] 

ASF subversion and git services commented on SOLR-11443:


Commit 265cf35503ccd50f38d4f5ba576910c25c8dd326 in lucene-solr's branch 
refs/heads/branch_7x from [~caomanhdat]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=265cf35 ]

SOLR-11443: Remove the usage of workqueue for Overseer


> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-14 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204530#comment-16204530
 ] 

ASF GitHub Bot commented on SOLR-11443:
---

Github user asfgit closed the pull request at:

https://github.com/apache/lucene-solr/pull/262


> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-14 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204529#comment-16204529
 ] 

ASF subversion and git services commented on SOLR-11443:


Commit 9543e85460b6d1264857c42b568d4a7f59c06007 in lucene-solr's branch 
refs/heads/master from [~caomanhdat]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9543e85 ]

SOLR-11443: Remove the usage of workqueue for Overseer


> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-13 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1620#comment-1620
 ] 

ASF GitHub Bot commented on SOLR-11443:
---

Github user CaoManhDat commented on the issue:

https://github.com/apache/lucene-solr/pull/262
  
:+1: 


> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-13 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204443#comment-16204443
 ] 

Scott Blum commented on SOLR-11443:
---

LGTM

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-13 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204436#comment-16204436
 ] 

ASF GitHub Bot commented on SOLR-11443:
---

GitHub user CaoManhDat opened a pull request:

https://github.com/apache/lucene-solr/pull/262

SOLR-11443: Remove the usage of workqueue for Overseer

SOLR-11443: Remove the usage of workqueue for Overseer

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/CaoManhDat/lucene-solr jira/SOLR-11443

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #262


commit 9543e85460b6d1264857c42b568d4a7f59c06007
Author: Cao Manh Dat 
Date:   2017-10-14T02:33:17Z

SOLR-11443: Remove the usage of workqueue for Overseer




> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-13 Thread Cao Manh Dat (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204325#comment-16204325
 ] 

Cao Manh Dat commented on SOLR-11443:
-

[~dragonsinth] We should not, unconditionally clear and set dirty, cause this 
will trigger get zk node names ( which is very expensive ). Why should we do 
that if the cache still valid?? Yeah, knowChildren.size() == 0 should be 
addressed too.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-13 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204001#comment-16204001
 ] 

Scott Blum commented on SOLR-11443:
---

SOLR-11447 looks interesting, might well address that comment.

{code}
int cacheSizeBefore = knownChildren.size();
knownChildren.removeAll(paths);
if (cacheSizeBefore - paths.size() == knownChildren.size()) {
  stats.setQueueLength(knownChildren.size());
} else {
  // There are elements get deleted but not present in the cache,
  // the cache seems not valid anymore
  knownChildren.clear();
  isDirty = true;
}
{code}

I just kind of feel like you should unconditionally clear and set dirty, to 
catch any weird edge cases.  What if post removal, knownChildren.size() == 0 in 
the above code?  Having knownChildren empty and !isDirty seems runs the risk of 
report false queue empty status when in fact we just need to pull more nodes 
from ZK.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-13 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203992#comment-16203992
 ] 

Scott Blum commented on SOLR-11443:
---

BTW, have you tried out github PRs?  It would be so much easier to review in 
that tool. :)

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-13 Thread Cao Manh Dat (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203212#comment-16203212
 ] 

Cao Manh Dat commented on SOLR-11443:
-

Thank [~dragonsinth] for reviewing!
bq. I might be thinking about this wrong, but the test seems to trying to 
thread an invisible needle, I guess we're trying to shut down overseer halfway 
through the list of updates? But we might very well just complete all 
operations quickly and restart overseer after they're all done.
bq. I feel like the maybeFlushBefore, maybeFlushAfter bits need a little more 
thinking. Seems pretty arbitrary to only check the firstCommand; maybe we 
should completely separate command-specific flush trigger from general purpose 
flush trigger? Then you could check command-level flushing on each command, if 
that's even still necessary.
bq.When would numUpdates diverge from updates.size()?

That all relates to SOLR-11447 changes. 
For the first one, I assume that you're talking about {{testDownNodeFailover}}, 
DOWNNODE message is converted to multiple ZKWriteCommands, so the test proves 
that if we flush clusterstate when processing the first command and Overseer 
get restarted right after the flushing, the rest of ZkWriteCommands will never 
get executed.
For the second comment, I fixed that in last patch of SOLR-11447. 
The numUpdates count number of ZkWriteCommand was processed, updates.size() 
indicates how many collections get affected ( many ZkWriteCommands can affect 
single collection )

bq. Seems like you could just always set this dirty; but if you're trying to 
in-memory surgery as an optimization, I don't understand the need for the 
containsAll check.

It is a sanity check ( which can never happen ). But If we know that there are 
nodes get deleted but not present in the cache, the cache seems in the dirty 
state. Here is slightly better version of that code block.
{code}
int cacheSizeBefore = knownChildren.size();
knownChildren.removeAll(paths);
if (cacheSizeBefore - paths.size() == knownChildren.size()) {
  stats.setQueueLength(knownChildren.size());
} else {
  // There are elements get deleted but not present in the cache,
  // the cache seems not valid anymore
  knownChildren.clear();
  isDirty = true;
}
{code}



> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201129#comment-16201129
 ] 

Scott Blum commented on SOLR-11443:
---

Should probably deprecate and update all the doc on the legacy workQueue to 
note that it's only to support the previous version since we don't add anything 
to it anymore.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201095#comment-16201095
 ] 

Scott Blum commented on SOLR-11443:
---

Can you talk me through:

{code}
if (knownChildren.containsAll(paths)) {
  knownChildren.removeAll(paths);
  stats.setQueueLength(knownChildren.size());
} else {
  knownChildren.clear();
  isDirty = true;
}
{code}

Seems like you could just always set this dirty; but if you're trying to 
in-memory surgery as an optimization, I don't understand the need for the 
containsAll check.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201120#comment-16201120
 ] 

Scott Blum commented on SOLR-11443:
---

Just a few nits / questions, otherwise LGTM.  Super great performance 
improvement and simplification.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201107#comment-16201107
 ] 

Scott Blum commented on SOLR-11443:
---

it might be more clear to do the `processedNodes.add(head.first());` _after_ 
calling processQueueItem, maybe conditionally based on whether onWriteAfter was 
called.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201081#comment-16201081
 ] 

Scott Blum commented on SOLR-11443:
---

When would numUpdates diverge from updates.size()?

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201074#comment-16201074
 ] 

Scott Blum commented on SOLR-11443:
---

I feel like the maybeFlushBefore, maybeFlushAfter bits need a little more 
thinking.  Seems pretty arbitrary to only check the firstCommand; maybe we 
should completely separate command-specific flush trigger from general purpose 
flush trigger?  Then you could check command-level flushing on each command, if 
that's even still necessary.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201064#comment-16201064
 ] 

Scott Blum commented on SOLR-11443:
---

I might be thinking about this wrong, but the test seems to trying to thread an 
invisible needle, I guess we're trying to shut down overseer halfway through 
the list of updates?  But we might very well just complete all operations 
quickly and restart overseer after they're all done.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11443) Remove the usage of workqueue for Overseer

2017-10-11 Thread Scott Blum (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200881#comment-16200881
 ] 

Scott Blum commented on SOLR-11443:
---

Love the idea; having a separate queue-work never made much sense to me.  I can 
look at the patch in a bit.

> Remove the usage of workqueue for Overseer
> --
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in 
> Overseer, hence boost performance a lot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org