[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-08-12 Thread Yago Riveiro (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419184#comment-15419184 ] Yago Riveiro commented on SOLR-8586: Then I do not understand, how this is possible:

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-08-12 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419145#comment-15419145 ] Yonik Seeley commented on SOLR-8586: You can set the environment variable solr.disableFingerprint to

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-08-12 Thread Yago Riveiro (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419078#comment-15419078 ] Yago Riveiro commented on SOLR-8586: My index has 12T of data indexed with 4.0, the _version_ field

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-08-12 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418799#comment-15418799 ] Yonik Seeley commented on SOLR-8586: If the _version_ field doesn't have docValues, then it will be

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-08-11 Thread Yago Riveiro (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417460#comment-15417460 ] Yago Riveiro commented on SOLR-8586: Is this operation memory bound? I'm trying to update my SolrCloud

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-10 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142215#comment-15142215 ] Yonik Seeley commented on SOLR-8586: bq. Yep, I've been looping a custom version of the

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-08 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137248#comment-15137248 ] Yonik Seeley commented on SOLR-8586: bq. The first thing I see is to compare hashes between the shards

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-08 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137227#comment-15137227 ] Yonik Seeley commented on SOLR-8586: OK, I did some basic performance testing... On an index w/ 5M

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-08 Thread Joel Bernstein (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136924#comment-15136924 ] Joel Bernstein commented on SOLR-8586: -- Now that this is in place it may make sense to combine this

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-08 Thread Joel Bernstein (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137272#comment-15137272 ] Joel Bernstein commented on SOLR-8586: -- I think there would need to be a system of truth involved,

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-05 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134387#comment-15134387 ] ASF subversion and git services commented on SOLR-8586: --- Commit

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-05 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134476#comment-15134476 ] ASF subversion and git services commented on SOLR-8586: --- Commit

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-05 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134529#comment-15134529 ] ASF subversion and git services commented on SOLR-8586: --- Commit

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-05 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134542#comment-15134542 ] ASF subversion and git services commented on SOLR-8586: --- Commit

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-05 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134545#comment-15134545 ] ASF subversion and git services commented on SOLR-8586: --- Commit

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-04 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133217#comment-15133217 ] Erick Erickson commented on SOLR-8586: -- OK, does this mean I can commit SOLR-8500 (after this is

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-04 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133278#comment-15133278 ] ASF subversion and git services commented on SOLR-8586: --- Commit

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-04 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133286#comment-15133286 ] Mark Miller commented on SOLR-8586: --- I think that we should warn that it can result in more often needing

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-04 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133458#comment-15133458 ] Erick Erickson commented on SOLR-8586: -- Yeah, this is kind of a "use at your own risk in very

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-04 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132688#comment-15132688 ] Mark Miller commented on SOLR-8586: --- Any chaos monkey test results yet? > Implement hash over all

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-04 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132725#comment-15132725 ] Yonik Seeley commented on SOLR-8586: Yep, I've been looping a custom version of the HDFS-nothing-safe

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-04 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132892#comment-15132892 ] ASF subversion and git services commented on SOLR-8586: --- Commit

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-01 Thread Stephan Lagraulet (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126086#comment-15126086 ] Stephan Lagraulet commented on SOLR-8586: - I'm trying to gather all issues related to SolrCloud

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-02-01 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126352#comment-15126352 ] Yonik Seeley commented on SOLR-8586: {quote} PeerSync always returned "true" if the core doing the sync

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-28 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122039#comment-15122039 ] Yonik Seeley commented on SOLR-8586: So the basic idea is that when a replica coming back up syncs to a

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-28 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122496#comment-15122496 ] Yonik Seeley commented on SOLR-8586: bq. I am thinking if per-segment caching would conflict with any

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-28 Thread Ishan Chattopadhyaya (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122410#comment-15122410 ] Ishan Chattopadhyaya commented on SOLR-8586: The approach looks good to me. {quote} {code} +

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-28 Thread David Smith (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122311#comment-15122311 ] David Smith commented on SOLR-8586: --- Trying to understand this without knowing the internals of the sync

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-28 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122263#comment-15122263 ] Yonik Seeley commented on SOLR-8586: PeerSync always returned "true" if the core doing the sync was

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-28 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122527#comment-15122527 ] Yonik Seeley commented on SOLR-8586: bq. It isn't stated, but I assume the replica does a full sync if

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-28 Thread Joel Bernstein (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122872#comment-15122872 ] Joel Bernstein commented on SOLR-8586: -- This exactly what we need for implementing alerts (SOLR-8577).

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-27 Thread Ishan Chattopadhyaya (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119093#comment-15119093 ] Ishan Chattopadhyaya commented on SOLR-8586: numRecordsToKeep can be configured.

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-27 Thread Ishan Chattopadhyaya (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119099#comment-15119099 ] Ishan Chattopadhyaya commented on SOLR-8586: Just want to understand that those updates (at a

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-27 Thread Stephan Lagraulet (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119107#comment-15119107 ] Stephan Lagraulet commented on SOLR-8586: - Thanks I missed this Solr5 enhancement... > Implement

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-27 Thread Stephan Lagraulet (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119062#comment-15119062 ] Stephan Lagraulet commented on SOLR-8586: - Would it be possible to increase this "100 updates"

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-27 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119180#comment-15119180 ] Yonik Seeley commented on SOLR-8586: The latter. We're looking at that is in the index, and that will

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-26 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118596#comment-15118596 ] Yonik Seeley commented on SOLR-8586: bq. I wonder if adding hashes of the version might be prone to

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-26 Thread David Smiley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15117982#comment-15117982 ] David Smiley commented on SOLR-8586: Could you please clarify what this issue is all about? I don't get

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-26 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118332#comment-15118332 ] Yonik Seeley commented on SOLR-8586: bq. Could you please clarify what this issue is all about? I don't

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-26 Thread David Smiley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118579#comment-15118579 ] David Smiley commented on SOLR-8586: Ah, ok. I wasn't familiar with PeerSync; thanks for educating me.

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-22 Thread Ishan Chattopadhyaya (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112702#comment-15112702 ] Ishan Chattopadhyaya commented on SOLR-8586: A bloom filter with all versions, maybe? >

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-22 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112725#comment-15112725 ] Yonik Seeley commented on SOLR-8586: A bloom filter would allow one to estimate (with a known error) if

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-22 Thread Ishan Chattopadhyaya (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112734#comment-15112734 ] Ishan Chattopadhyaya commented on SOLR-8586: I see.. My initial thought was that a bloom filter

[jira] [Commented] (SOLR-8586) Implement hash over all documents to check for shard synchronization

2016-01-22 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112751#comment-15112751 ] Yonik Seeley commented on SOLR-8586: bq. My initial thought was that a bloom filter from one replica