[GitHub] [lucene-solr] zacharymorn commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
zacharymorn commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748563011 Oh wow thanks for the updates there, and sorry to hear the current changes are not yet ready! Yes the original implementation hasn't been used for a while now (as

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748539067 I am giving up for today. The whole implementation is plain broken and the test never catched it. It was writen in earlier times of Lucene when IndexOutputs were able to

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748535427 Hi, I modified the test to allow some enforced mode, so the direcory is fully tests (basically, I made it to always return direct IndexInput/Outputs. After this it

[GitHub] [lucene-solr] jpountz commented on pull request #993: LUCENE-9303: maybe can simpler

2020-12-19 Thread GitBox
jpountz commented on pull request #993: URL: https://github.com/apache/lucene-solr/pull/993#issuecomment-748532390 Sorry for the time to respond on this PR @kkewwei. It looks like the problem you pointed out no longer exists in master?

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748530259 I removed the file length handling and cleaned up the whole stuff. I am not sure if the truncate is still needed, but I left it for now. In addition, the

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748529191 There's more broken, the file size on writing is incorrect sometimes. More rewrite is required. This is

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748528098 Hi, I rewrote most of the ByteBuffer handling. The bugs came from the fact that we used an additional outside bufferSize and bufferPos. All calcuations were done based

[GitHub] [lucene-solr] murblanc commented on a change in pull request #2148: SOLR-15052: Per-replica states for reducing overseer bottlenecks

2020-12-19 Thread GitBox
murblanc commented on a change in pull request #2148: URL: https://github.com/apache/lucene-solr/pull/2148#discussion_r546277620 ## File path: solr/core/src/java/org/apache/solr/cloud/overseer/CollectionMutator.java ## @@ -136,8 +154,13 @@ public ZkWriteCommand

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748519083 I will rewrite that stuff a bit, so you can pass a mergeBufferSize when creating the directory, but the value is adjusted according to the file system. I don't want

[GitHub] [lucene-solr] murblanc commented on a change in pull request #2148: SOLR-15052: Per-replica states for reducing overseer bottlenecks

2020-12-19 Thread GitBox
murblanc commented on a change in pull request #2148: URL: https://github.com/apache/lucene-solr/pull/2148#discussion_r546277475 ## File path: solr/solrj/src/java/org/apache/solr/common/cloud/PerReplicaStates.java ## @@ -0,0 +1,587 @@ +/* + * Licensed to the Apache Software

[GitHub] [lucene-solr] murblanc commented on a change in pull request #2148: SOLR-15052: Per-replica states for reducing overseer bottlenecks

2020-12-19 Thread GitBox
murblanc commented on a change in pull request #2148: URL: https://github.com/apache/lucene-solr/pull/2148#discussion_r546277475 ## File path: solr/solrj/src/java/org/apache/solr/common/cloud/PerReplicaStates.java ## @@ -0,0 +1,587 @@ +/* + * Licensed to the Apache Software

[GitHub] [lucene-solr] murblanc commented on a change in pull request #2148: SOLR-15052: Per-replica states for reducing overseer bottlenecks

2020-12-19 Thread GitBox
murblanc commented on a change in pull request #2148: URL: https://github.com/apache/lucene-solr/pull/2148#discussion_r546277024 ## File path: solr/solrj/src/java/org/apache/solr/common/cloud/PerReplicaStates.java ## @@ -0,0 +1,587 @@ +/* + * Licensed to the Apache Software

[GitHub] [lucene-solr] murblanc commented on a change in pull request #2148: SOLR-15052: Per-replica states for reducing overseer bottlenecks

2020-12-19 Thread GitBox
murblanc commented on a change in pull request #2148: URL: https://github.com/apache/lucene-solr/pull/2148#discussion_r546277024 ## File path: solr/solrj/src/java/org/apache/solr/common/cloud/PerReplicaStates.java ## @@ -0,0 +1,587 @@ +/* + * Licensed to the Apache Software

[jira] [Commented] (SOLR-15052) Reducing overseer bottlenecks using per-replica states

2020-12-19 Thread Ishan Chattopadhyaya (Jira)
[ https://issues.apache.org/jira/browse/SOLR-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252260#comment-17252260 ] Ishan Chattopadhyaya commented on SOLR-15052: - Thanks [~ilan] and [~mdrob] for the reviews.

[GitHub] [lucene-solr] uschindler edited a comment on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler edited a comment on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748517193 Yes that's the issue. On windows it looks like allocateDirect does not allign to beginning of page at some memory location not aligned to 512 bytes. The

[GitHub] [lucene-solr] murblanc commented on pull request #2148: SOLR-15052: Per-replica states for reducing overseer bottlenecks

2020-12-19 Thread GitBox
murblanc commented on pull request #2148: URL: https://github.com/apache/lucene-solr/pull/2148#issuecomment-748518170 Added my high level comments as a

[jira] [Commented] (SOLR-15052) Reducing overseer bottlenecks using per-replica states

2020-12-19 Thread Ilan Ginzburg (Jira)
[ https://issues.apache.org/jira/browse/SOLR-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252259#comment-17252259 ] Ilan Ginzburg commented on SOLR-15052: -- I think this goes in the right direction and thank you for

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748517370 Nevertheless what I don't like: why do we need this ALIGN masks and so on. As we have the official block size we can use that for our calculations instead of relying on

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748517193 Yes that's the issue. On windows it looks like allocateDirect does not allign to beginning of page at some memory location not aligned to 512 bytes. The alignedSlice is

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748516524 I think I know the problem: when you call alignedSlice, the buffer's remaining() may be no longer the bufferSize. All calculations afterwards are then using the

[GitHub] [lucene-solr] zacharymorn commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
zacharymorn commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748509146 > > Hmm I don't have a windows box to test this. The blockSize is dynamically computed though. If this works on linux/darwin but fails on windows, does it signal

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748508063 > Hmm I don't have a windows box to test this. The blockSize is dynamically computed though. If this works on linux/darwin but fails on windows, does it signal another

[GitHub] [lucene-solr] zacharymorn commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
zacharymorn commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748506650 > Hi, I implemented the approach and also cleaned up the lookup code to not require unchecked casts. I also made the code safer for exceptions, you only catched

[GitHub] [lucene-solr] jpountz commented on a change in pull request #2141: LUCENE-9346: Support minimumNumberShouldMatch in WANDScorer

2020-12-19 Thread GitBox
jpountz commented on a change in pull request #2141: URL: https://github.com/apache/lucene-solr/pull/2141#discussion_r546265505 ## File path: lucene/core/src/java/org/apache/lucene/search/WANDScorer.java ## @@ -271,6 +280,19 @@ public boolean matches() throws IOException {

[jira] [Commented] (LUCENE-9645) Delete useless variable negativeDeltas in BKDReader$IndexTree

2020-12-19 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252248#comment-17252248 ] Adrien Grand commented on LUCENE-9645: -- Would you like to open a pull request to fix this? >

[GitHub] [lucene-solr] jpountz commented on a change in pull request #2141: LUCENE-9346: Support minimumNumberShouldMatch in WANDScorer

2020-12-19 Thread GitBox
jpountz commented on a change in pull request #2141: URL: https://github.com/apache/lucene-solr/pull/2141#discussion_r546262630 ## File path: lucene/core/src/java/org/apache/lucene/search/WANDScorer.java ## @@ -130,10 +130,19 @@ private static long scaleMinScore(float

[GitHub] [lucene-solr] jpountz commented on a change in pull request #2141: LUCENE-9346: Support minimumNumberShouldMatch in WANDScorer

2020-12-19 Thread GitBox
jpountz commented on a change in pull request #2141: URL: https://github.com/apache/lucene-solr/pull/2141#discussion_r546262588 ## File path: lucene/core/src/java/org/apache/lucene/search/WANDScorer.java ## @@ -130,10 +130,19 @@ private static long scaleMinScore(float

[jira] [Resolved] (LUCENE-9617) FieldNumbers.clear() should reset lowestUnassignedFieldNumber

2020-12-19 Thread Michael McCandless (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-9617. Fix Version/s: 8.8 Resolution: Fixed Thank you [~msfroh]! >

[jira] [Commented] (LUCENE-9617) FieldNumbers.clear() should reset lowestUnassignedFieldNumber

2020-12-19 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252202#comment-17252202 ] ASF subversion and git services commented on LUCENE-9617: - Commit

[jira] [Commented] (LUCENE-9617) FieldNumbers.clear() should reset lowestUnassignedFieldNumber

2020-12-19 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252201#comment-17252201 ] ASF subversion and git services commented on LUCENE-9617: - Commit

[jira] [Commented] (LUCENE-9617) FieldNumbers.clear() should reset lowestUnassignedFieldNumber

2020-12-19 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252199#comment-17252199 ] ASF subversion and git services commented on LUCENE-9617: - Commit

[jira] [Commented] (LUCENE-9617) FieldNumbers.clear() should reset lowestUnassignedFieldNumber

2020-12-19 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252200#comment-17252200 ] ASF subversion and git services commented on LUCENE-9617: - Commit

[jira] [Updated] (SOLR-11191) Shard Split doesn't work on indexes that have nested documents

2020-12-19 Thread David Smiley (Jira)
[ https://issues.apache.org/jira/browse/SOLR-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-11191: Labels: NestedDocuments (was: ) > Shard Split doesn't work on indexes that have nested documents

[jira] [Assigned] (SOLR-11191) Shard Split doesn't work on indexes that have nested documents

2020-12-19 Thread David Smiley (Jira)
[ https://issues.apache.org/jira/browse/SOLR-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley reassigned SOLR-11191: --- Assignee: David Smiley > Shard Split doesn't work on indexes that have nested documents >

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748465821 Nevertheless, on Windows I see the test failing all the time (JDK 11 and JDK 15 were tested): ``` org.apache.lucene.misc.store.TestDirectIODirectory >

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748465633 Hi, I implemented the approach and also cleaned up the lookup code to not require unchecked casts. I also made the code safer for exceptions, you only catched

[GitHub] [lucene-solr] dweiss edited a comment on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
dweiss edited a comment on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748454464 The master branch of the JDK always emits this exception, it seems:

[GitHub] [lucene-solr] dweiss commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
dweiss commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748454464 The master branch of the JDK always emits this exception, it seems:

[GitHub] [lucene-solr] iverase commented on pull request #2154: LUCENE-9642: Rotate triangle points before checking triangle orientation

2020-12-19 Thread GitBox
iverase commented on pull request #2154: URL: https://github.com/apache/lucene-solr/pull/2154#issuecomment-748449367 good point @madrob, unit test added. This is an automated message from the Apache Git Service. To respond

[GitHub] [lucene-solr] zacharymorn commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
zacharymorn commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748443350 > You can do it statically, look at how the unmapper in Mmapdir works: > Look up the reflective in static intitalizer and save the result at very end in static final.

[GitHub] [lucene-solr] uschindler commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
uschindler commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748442053 You can do it statically, look at how the unmapper in Mmapdir works: Look up the reflective in static intitalizer and save the result at very end in static final.

[GitHub] [lucene-solr] zacharymorn commented on pull request #2052: LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory

2020-12-19 Thread GitBox
zacharymorn commented on pull request #2052: URL: https://github.com/apache/lucene-solr/pull/2052#issuecomment-748441602 > I'll try to help out later today, Zach. Sounds good. Thanks Dawid! This is an automated