[jira] [Commented] (SOLR-9284) The HDFS BlockDirectoryCache should not let it's keysToRelease or names maps grow indefinitely.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666417#comment-15666417
 ] 

Mark Miller commented on SOLR-9284:
---

bq. But if it's hard to achieve

It's not exactly simple - that is why the current delete methods take a file 
name, but also do not release any of the block keys. I think if we wanted to 
that, we either have to do long scans of cache keys, or start storing cache key 
list maps keyed by file name.

> The HDFS BlockDirectoryCache should not let it's keysToRelease or names maps 
> grow indefinitely.
> ---
>
> Key: SOLR-9284
> URL: https://issues.apache.org/jira/browse/SOLR-9284
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: hdfs
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 6.2, master (7.0)
>
> Attachments: SOLR-9284.patch, SOLR-9284.patch
>
>
> https://issues.apache.org/jira/browse/SOLR-9284



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-6.x - Build # 181 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-6.x/181/

No tests ran.

Build Log:
[...truncated 40552 lines...]
prepare-release-no-sign:
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist
 [copy] Copying 476 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist/lucene
 [copy] Copying 245 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist/solr
   [smoker] Java 1.8 JAVA_HOME=/home/jenkins/tools/java/latest1.8
   [smoker] NOTE: output encoding is UTF-8
   [smoker] 
   [smoker] Load release URL 
"file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist/"...
   [smoker] 
   [smoker] Test Lucene...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.01 sec (17.8 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download lucene-6.4.0-src.tgz...
   [smoker] 30.2 MB in 0.04 sec (858.5 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-6.4.0.tgz...
   [smoker] 64.7 MB in 0.07 sec (868.8 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-6.4.0.zip...
   [smoker] 75.5 MB in 0.09 sec (878.3 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack lucene-6.4.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6108 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-6.4.0.zip...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6108 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-6.4.0-src.tgz...
   [smoker] make sure no JARs/WARs in src dist...
   [smoker] run "ant validate"
   [smoker] run tests w/ Java 8 and testArgs='-Dtests.slow=false'...
   [smoker] test demo with 1.8...
   [smoker]   got 227 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] generate javadocs w/ Java 8...
   [smoker] 
   [smoker] Crawl/parse...
   [smoker] 
   [smoker] Verify...
   [smoker]   confirm all releases have coverage in TestBackwardsCompatibility
   [smoker] find all past Lucene releases...
   [smoker] run TestBackwardsCompatibility..
   [smoker] success!
   [smoker] 
   [smoker] Test Solr...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.00 sec (201.4 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download solr-6.4.0-src.tgz...
   [smoker] 39.6 MB in 0.05 sec (788.0 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-6.4.0.tgz...
   [smoker] 139.4 MB in 0.17 sec (816.2 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-6.4.0.zip...
   [smoker] 148.5 MB in 0.19 sec (793.2 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack solr-6.4.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] unpack lucene-6.4.0.tgz...
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.4.0/contrib/dataimporthandler-extras/lib/javax.mail-1.5.1.jar:
 it has javax.* classes
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.4.0/contrib/dataimporthandler-extras/lib/activation-1.1.1.jar:
 it has javax.* classes
   [smoker] copying unpacked distribution for Java 8 ...
   [smoker] test solr example w/ Java 8...
   [smoker]   start Solr instance 
(log=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.4.0-java8/solr-example.log)...
   [smoker] No process found for Solr node running on port 8983
   [smoker]   Running techproducts example on port 8983 from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.4.0-java8
   [smoker] Creating Solr home directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.4.0-java8/example/techproducts/solr
   [smoker] 
   [smoker] Starting up Solr on port 8983 using command:
   [smoker] bin/solr start -p 8983 -s "example/techproducts/solr"
   [smoker] 
   [smoker] Waiting up to 180 seconds to see Solr running on port 8983 [|]  
 [/]   [-]   [\]  
   [smoker] Started Solr server on port 8983 (pid=2539). Happy searching!
   [smoker] 
   [smoker] 
   

[jira] [Updated] (LUCENE-7526) Improvements to UnifiedHighlighter OffsetStrategies

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley updated LUCENE-7526:
-
Attachment: LUCENE-7526.patch

I understand RE MultiValueTokenStream.  I was motivated by considering 
highlighting a TokenStream without a MemoryIndex resulted in some complexity in 
various places (not introduced here, I mean pre-existing); some of it is 
addressed with this patch, and others still linger.  Another hack that could 
move is the getPayoad() returning the Term.  If this offset strategy returned a 
subclassed OffsetEnum, that quirk could be better isolated.  Any way; let that 
be for another day.

Attached is the patch delta from master.  I'll commit tomorrow night.

> Improvements to UnifiedHighlighter OffsetStrategies
> ---
>
> Key: LUCENE-7526
> URL: https://issues.apache.org/jira/browse/LUCENE-7526
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.4
>
> Attachments: LUCENE-7526.patch
>
>
> This ticket improves several of the UnifiedHighlighter FieldOffsetStrategies 
> by reducing reliance on creating or re-creating TokenStreams.
> The primary changes are as follows:
> * AnalysisOffsetStrategy - split into two offset strategies
>   ** MemoryIndexOffsetStrategy - the primary analysis mode that utilizes a 
> MemoryIndex for producing Offsets
>   ** TokenStreamOffsetStrategy - an offset strategy that avoids creating a 
> MemoryIndex.  Can only be used if the query distills down to terms and 
> automata.
> * TokenStream removal 
>   ** MemoryIndexOffsetStrategy - previously a TokenStream was created to fill 
> the memory index and then once consumed a new one was generated by 
> uninverting the MemoryIndex back into a TokenStream if there were automata 
> (wildcard/mtq queries) involved.  Now this is avoided, which should save 
> memory and avoid a second pass over the data.
>   ** TermVectorOffsetStrategy - this was refactored in a similar way to avoid 
> generating a TokenStream if automata are involved.
>   ** PostingsWithTermVectorsOffsetStrategy - similar refactoring
> * CompositePostingsEnum - aggregates several underlying PostingsEnums for 
> wildcard/mtq queries.  This should improve relevancy by providing unified 
> metrics for a wildcard across all it's term matches
> * Added a HighlightFlag for enabling the newly separated 
> TokenStreamOffsetStrategy since it can adversely affect passage relevancy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9633) Limit FastLRUCache by RAM Usage

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar resolved SOLR-9633.
-
Resolution: Fixed
  Assignee: Shalin Shekhar Mangar

Thanks Yonik and Michael!

> Limit FastLRUCache by RAM Usage
> ---
>
> Key: SOLR-9633
> URL: https://issues.apache.org/jira/browse/SOLR-9633
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9633.patch, SOLR-9633.patch
>
>
> SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on 
> memory usage. That helps with the query result cache but not with the filter 
> cache which defaults to FastLRUCache. This issue intends to add the same 
> feature to FastLRUCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9766) Solr precommit is failing

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666167#comment-15666167
 ] 

Erick Erickson commented on SOLR-9766:
--

Thanks guys!

> Solr precommit is failing
> -
>
> Key: SOLR-9766
> URL: https://issues.apache.org/jira/browse/SOLR-9766
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
>Assignee: Steve Rowe
>
> The Solr precommit is failing due to following errors,
> [ecj-lint] 36. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 31)
>  [ecj-lint]   import org.apache.solr.client.solrj.SolrQuery;
>  [ecj-lint]  ^^
>  [ecj-lint] The import org.apache.solr.client.solrj.SolrQuery is never used
>  [ecj-lint] --
>  [ecj-lint] 37. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 34)
>  [ecj-lint]   import org.apache.solr.client.solrj.impl.HttpSolrClient;
>  [ecj-lint]  
>  [ecj-lint] The import org.apache.solr.client.solrj.impl.HttpSolrClient is 
> never used
>  [ecj-lint] --
>  [ecj-lint] 38. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 52)
>  [ecj-lint]   import org.apache.solr.client.solrj.response.QueryResponse;
>  [ecj-lint]  ^^^
>  [ecj-lint] The import org.apache.solr.client.solrj.response.QueryResponse is 
> never used
>  [ecj-lint] --
>  [ecj-lint] 39. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 55)
>  [ecj-lint]   import org.apache.solr.common.SolrDocument;
>  [ecj-lint]  ^^^
>  [e
> These seem to be introduced by changes committed as part of SOLR-9166 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666163#comment-15666163
 ] 

ASF subversion and git services commented on SOLR-9633:
---

Commit b57a5e41f82418621d8ac7e42edf0cd08dcee6be in lucene-solr's branch 
refs/heads/master from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b57a5e4 ]

SOLR-9633: Fix issue number in CHANGES.txt


> Limit FastLRUCache by RAM Usage
> ---
>
> Key: SOLR-9633
> URL: https://issues.apache.org/jira/browse/SOLR-9633
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9633.patch, SOLR-9633.patch
>
>
> SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on 
> memory usage. That helps with the query result cache but not with the filter 
> cache which defaults to FastLRUCache. This issue intends to add the same 
> feature to FastLRUCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666166#comment-15666166
 ] 

ASF subversion and git services commented on SOLR-9633:
---

Commit d3420c31a92180653b767850e82f55ec3d20fda2 in lucene-solr's branch 
refs/heads/branch_6x from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d3420c3 ]

SOLR-9633: Fix issue number in CHANGES.txt

(cherry picked from commit b57a5e4)


> Limit FastLRUCache by RAM Usage
> ---
>
> Key: SOLR-9633
> URL: https://issues.apache.org/jira/browse/SOLR-9633
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9633.patch, SOLR-9633.patch
>
>
> SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on 
> memory usage. That helps with the query result cache but not with the filter 
> cache which defaults to FastLRUCache. This issue intends to add the same 
> feature to FastLRUCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage

[Shalin Shekhar Mangar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666161#comment-15666161
 ] 

Shalin Shekhar Mangar commented on SOLR-9633:
-

I put SOLR-9366 instead of SOLR-9633 in the commit message by mistake. The 
commit messages are:
{code}
Commit 487b0976eb3e98b78ab492f4969a2aa0373b626f in lucene-solr's branch 
refs/heads/master from Shalin Shekhar Mangar
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=487b097 ]

SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config 
parameter
{code}

{code}
Commit 53dad4f0d1c8f563810626ef9ab470fb3e5d1da9 in lucene-solr's branch 
refs/heads/branch_6x from Shalin Shekhar Mangar
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=53dad4f ]

SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config 
parameter
(cherry picked from commit 487b097)
{code}

> Limit FastLRUCache by RAM Usage
> ---
>
> Key: SOLR-9633
> URL: https://issues.apache.org/jira/browse/SOLR-9633
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9633.patch, SOLR-9633.patch
>
>
> SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on 
> memory usage. That helps with the query result cache but not with the filter 
> cache which defaults to FastLRUCache. This issue intends to add the same 
> feature to FastLRUCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Deleted] (SOLR-9366) replicas in LIR seem to still be participating in search requests

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar updated SOLR-9366:

Comment: was deleted

(was: Commit 487b0976eb3e98b78ab492f4969a2aa0373b626f in lucene-solr's branch 
refs/heads/master from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=487b097 ]

SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config 
parameter
)

> replicas in LIR seem to still be participating in search requests
> -
>
> Key: SOLR-9366
> URL: https://issues.apache.org/jira/browse/SOLR-9366
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
> Attachments: SOLR-9366.txt.gz
>
>
> Spinning this off from SOLR-9363 where sarowe's jenkins encountered a strange 
> test failure when TestInjection is used causing replicas to return errors on 
> some requests.
> Reading over the logs it appears that when this happens, and the replicas are 
> put into LIR they then ignore an explicit user requested commit (ie: 
> {{Ignoring commit while not ACTIVE - state: BUFFERING replay: false}}) but 
> still participate in queries -- apparently before they finish recovery (or at 
> the very least before / with-out doing the commit/openSearcher that they 
> previously ignored.
> Details and logs to follow...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Deleted] (SOLR-9366) replicas in LIR seem to still be participating in search requests

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar updated SOLR-9366:

Comment: was deleted

(was: Commit 53dad4f0d1c8f563810626ef9ab470fb3e5d1da9 in lucene-solr's branch 
refs/heads/branch_6x from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=53dad4f ]

SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config 
parameter

(cherry picked from commit 487b097)
)

> replicas in LIR seem to still be participating in search requests
> -
>
> Key: SOLR-9366
> URL: https://issues.apache.org/jira/browse/SOLR-9366
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
> Attachments: SOLR-9366.txt.gz
>
>
> Spinning this off from SOLR-9363 where sarowe's jenkins encountered a strange 
> test failure when TestInjection is used causing replicas to return errors on 
> some requests.
> Reading over the logs it appears that when this happens, and the replicas are 
> put into LIR they then ignore an explicit user requested commit (ie: 
> {{Ignoring commit while not ACTIVE - state: BUFFERING replay: false}}) but 
> still participate in queries -- apparently before they finish recovery (or at 
> the very least before / with-out doing the commit/openSearcher that they 
> previously ignored.
> Details and logs to follow...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9366) replicas in LIR seem to still be participating in search requests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666150#comment-15666150
 ] 

ASF subversion and git services commented on SOLR-9366:
---

Commit 53dad4f0d1c8f563810626ef9ab470fb3e5d1da9 in lucene-solr's branch 
refs/heads/branch_6x from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=53dad4f ]

SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config 
parameter

(cherry picked from commit 487b097)


> replicas in LIR seem to still be participating in search requests
> -
>
> Key: SOLR-9366
> URL: https://issues.apache.org/jira/browse/SOLR-9366
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
> Attachments: SOLR-9366.txt.gz
>
>
> Spinning this off from SOLR-9363 where sarowe's jenkins encountered a strange 
> test failure when TestInjection is used causing replicas to return errors on 
> some requests.
> Reading over the logs it appears that when this happens, and the replicas are 
> put into LIR they then ignore an explicit user requested commit (ie: 
> {{Ignoring commit while not ACTIVE - state: BUFFERING replay: false}}) but 
> still participate in queries -- apparently before they finish recovery (or at 
> the very least before / with-out doing the commit/openSearcher that they 
> previously ignored.
> Details and logs to follow...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9366) replicas in LIR seem to still be participating in search requests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666147#comment-15666147
 ] 

ASF subversion and git services commented on SOLR-9366:
---

Commit 487b0976eb3e98b78ab492f4969a2aa0373b626f in lucene-solr's branch 
refs/heads/master from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=487b097 ]

SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config 
parameter


> replicas in LIR seem to still be participating in search requests
> -
>
> Key: SOLR-9366
> URL: https://issues.apache.org/jira/browse/SOLR-9366
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
> Attachments: SOLR-9366.txt.gz
>
>
> Spinning this off from SOLR-9363 where sarowe's jenkins encountered a strange 
> test failure when TestInjection is used causing replicas to return errors on 
> some requests.
> Reading over the logs it appears that when this happens, and the replicas are 
> put into LIR they then ignore an explicit user requested commit (ie: 
> {{Ignoring commit while not ACTIVE - state: BUFFERING replay: false}}) but 
> still participate in queries -- apparently before they finish recovery (or at 
> the very least before / with-out doing the commit/openSearcher that they 
> previously ignored.
> Details and logs to follow...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_102) - Build # 2183 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/2183/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
at 
__randomizedtesting.SeedInfo.seed([DE0F8B6CBB011F57:E3D7254083EF4127]:0)
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:252)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:49)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:525)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$200(AbstractPollingIoAcceptor.java:67)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:409)
at 
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12306 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberosAlt
   [junit4]   2> 1734798 WARN  
(TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[DE0F8B6CBB011F57]) [] 
o.a.d.s.c.DefaultDirectoryService You didn't change the admin password of 
directory service instance 'DefaultKrbServer'.  Please update the admin 
password as soon as possible to prevent a possible security breach.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.method=testBasics 
-Dtests.seed=DE0F8B6CBB011F57 -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=sr-Latn-RS -Dtests.timezone=Europe/Ulyanovsk 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   8.65s J1 | TestSolrCloudWithKerberosAlt.testBasics <<<
   [junit4]> Throwable #1: java.net.BindException: Address already in use
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([DE0F8B6CBB011F57:E3D7254083EF4127]:0)
   [junit4]>at sun.nio.ch.Net.bind0(Native Method)
   [junit4]>at sun.nio.ch.Net.bind(Net.java:433)
   [junit4]>at sun.nio.ch.Net.bind(Net.java:425)
   [junit4]>at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
   [junit4]>at 
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
   [junit4]>at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:252)
   [junit4]>at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:49)
   [junit4]>at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:525)
   [junit4]>at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$200(AbstractPollingIoAcceptor.java:67)
   [junit4]>at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:409)
   [junit4]>at 
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
   [junit4]>at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]>at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]>at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberosAlt_DE0F8B6CBB011F57-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=sr-Latn-RS, 
timezone=Europe/Ulyanovsk
   [junit4]   2> NOTE: Linux 4.4.0-42-generic i386/Oracle Corporation 1.8.0_102 
(32-bit)/cpus=12,threads=1,free=219915688,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [ConfigSetsAPITest, 
AliasIntegrationTest, TestCSVResponseWriter, TestFieldTypeResource, 
ZkStateWriterTest, SortByFunctionTest, DeleteNodeTest, FullHLLTest, 
FileUtilsTest, ZkControllerTest, SimplePostToolTest, AnalysisErrorHandlingTest, 
HdfsTlogReplayBufferedWhileIndexingTest, TestTrie, TestCollectionAPI, 
TestTolerantUpdateProcessorRandomCloud, ConjunctionSolrSpellCheckerTest, 
TestSolrIndexConfig, TermsComponentTest, TestSQLHandlerNonCloud, 
HdfsNNFailoverTest, TolerantUpdateProcessorTest, 

[jira] [Comment Edited] (SOLR-9736) HttpSolrCall always prefer leader

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665726#comment-15665726
 ] 

Cao Manh Dat edited comment on SOLR-9736 at 11/15/16 2:04 AM:
--

[~shalinmangar] Thank you for review my patch, because the cores are chosen 
randomly so we can miss some core is chosen in the test.

Updated patch, in this patch I modified the test to run more request, to make 
sure that all the 
cores are chosen ( I ran the test about 300 times and it still passed )


was (Author: caomanhdat):
Updated patch, in this patch I modified the test to run more request, to make 
sure that all the cores are chosen.

> HttpSolrCall always prefer leader
> -
>
> Key: SOLR-9736
> URL: https://issues.apache.org/jira/browse/SOLR-9736
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-9736.patch, SOLR-9736.patch, SOLR-9736.patch, 
> SOLR-9736.patch
>
>
> Currently, `HttpSolrCall.getCoreByCollection` always picks the first 
> available leader ( or first replica ) of the first slice. It puts undue 
> pressure on leaders and quite possibly on the wrong ones



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9736) HttpSolrCall always prefer leader

[Cao Manh Dat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cao Manh Dat updated SOLR-9736:
---
Attachment: SOLR-9736.patch

Updated patch, in this patch I modified the test to run more request, to make 
sure that all the cores are chosen.

> HttpSolrCall always prefer leader
> -
>
> Key: SOLR-9736
> URL: https://issues.apache.org/jira/browse/SOLR-9736
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-9736.patch, SOLR-9736.patch, SOLR-9736.patch, 
> SOLR-9736.patch
>
>
> Currently, `HttpSolrCall.getCoreByCollection` always picks the first 
> available leader ( or first replica ) of the first slice. It puts undue 
> pressure on leaders and quite possibly on the wrong ones



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 199 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/199/

13 tests failed.
FAILED:  org.apache.lucene.search.TestFuzzyQuery.testRandom

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([DBDEA4DBCB14C93B]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestFuzzyQuery

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([DBDEA4DBCB14C93B]:0)


FAILED:  
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchBoundaries

Error Message:
Timeout waiting for CDCR replication to complete @source_collection:shard1

Stack Trace:
java.lang.RuntimeException: Timeout waiting for CDCR replication to complete 
@source_collection:shard1
at 
__randomizedtesting.SeedInfo.seed([9607C0FE3F7862F0:B42819AB46DCA317]:0)
at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForReplicationToComplete(BaseCdcrDistributedZkTest.java:795)
at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchBoundaries(CdcrReplicationDistributedZkTest.java:557)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (SOLR-9513) Introduce a generic authentication plugin which delegates all functionality to Hadoop authentication framework

[Hrishikesh Gadre (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665641#comment-15665641
 ] 

Hrishikesh Gadre commented on SOLR-9513:


[~ichattopadhyaya] Can you please take a look?

> Introduce a generic authentication plugin which delegates all functionality 
> to Hadoop authentication framework
> --
>
> Key: SOLR-9513
> URL: https://issues.apache.org/jira/browse/SOLR-9513
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
>
> Currently Solr kerberos authentication plugin delegates the core logic to 
> Hadoop authentication framework. But the configuration parameters required by 
> the Hadoop authentication framework are hardcoded in the plugin code itself. 
> https://github.com/apache/lucene-solr/blob/5b770b56d012279d334f41e4ef7fe652480fd3cf/solr/core/src/java/org/apache/solr/security/KerberosPlugin.java#L119
> The problem with this approach is that we need to make code changes in Solr 
> to expose new capabilities added in Hadoop authentication framework. e.g. 
> HADOOP-12082
> We should implement a generic Solr authentication plugin which will accept 
> configuration parameters via security.json (in Zookeeper) and delegate them 
> to Hadoop authentication framework. This will allow to utilize new features 
> in Hadoop without code changes in Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9513) Introduce a generic authentication plugin which delegates all functionality to Hadoop authentication framework

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665638#comment-15665638
 ] 

ASF GitHub Bot commented on SOLR-9513:
--

GitHub user hgadre opened a pull request:

https://github.com/apache/lucene-solr/pull/114

SOLR-9513 A generic Solr authentication plugin to integrate with Hadoop

- Added a generic Solr authentication plugin to integrate with Hadoop
- Added delegation tokens support
- Added proxy users support

Currently the unit tests are using kerberos authentication handler in
Hadoop. After Solr is updated to use Hadoop 3, these tests will be
modified to use multi-auth support in Hadoop.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hgadre/lucene-solr SOLR-9513_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #114


commit 7562e0cecfc758aff4399909ef147114813204f8
Author: Hrishikesh Gadre 
Date:   2016-11-08T21:18:34Z

SOLR-9513 A generic Solr authentication plugin to integrate with Hadoop

- Added a generic Solr authentication plugin to integrate with Hadoop
- Added delegation tokens support
- Added proxy users support

Currently the unit tests are using kerberos authentication handler in
Hadoop. After Solr is updated to use Hadoop 3, these tests will be
modified to use multi-auth support in Hadoop.




> Introduce a generic authentication plugin which delegates all functionality 
> to Hadoop authentication framework
> --
>
> Key: SOLR-9513
> URL: https://issues.apache.org/jira/browse/SOLR-9513
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
>
> Currently Solr kerberos authentication plugin delegates the core logic to 
> Hadoop authentication framework. But the configuration parameters required by 
> the Hadoop authentication framework are hardcoded in the plugin code itself. 
> https://github.com/apache/lucene-solr/blob/5b770b56d012279d334f41e4ef7fe652480fd3cf/solr/core/src/java/org/apache/solr/security/KerberosPlugin.java#L119
> The problem with this approach is that we need to make code changes in Solr 
> to expose new capabilities added in Hadoop authentication framework. e.g. 
> HADOOP-12082
> We should implement a generic Solr authentication plugin which will accept 
> configuration parameters via security.json (in Zookeeper) and delegate them 
> to Hadoop authentication framework. This will allow to utilize new features 
> in Hadoop without code changes in Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #114: SOLR-9513 A generic Solr authentication plugi...

[hgadre]

GitHub user hgadre opened a pull request:

https://github.com/apache/lucene-solr/pull/114

SOLR-9513 A generic Solr authentication plugin to integrate with Hadoop

- Added a generic Solr authentication plugin to integrate with Hadoop
- Added delegation tokens support
- Added proxy users support

Currently the unit tests are using kerberos authentication handler in
Hadoop. After Solr is updated to use Hadoop 3, these tests will be
modified to use multi-auth support in Hadoop.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hgadre/lucene-solr SOLR-9513_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #114


commit 7562e0cecfc758aff4399909ef147114813204f8
Author: Hrishikesh Gadre 
Date:   2016-11-08T21:18:34Z

SOLR-9513 A generic Solr authentication plugin to integrate with Hadoop

- Added a generic Solr authentication plugin to integrate with Hadoop
- Added delegation tokens support
- Added proxy users support

Currently the unit tests are using kerberos authentication handler in
Hadoop. After Solr is updated to use Hadoop 3, these tests will be
modified to use multi-auth support in Hadoop.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9166) Export handler returns zero for numeric fields that are not in the original doc

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665603#comment-15665603
 ] 

ASF subversion and git services commented on SOLR-9166:
---

Commit ca80ba6b80be619ebeea1f6b8f0864832ebbfec8 in lucene-solr's branch 
refs/heads/master from [~steve_rowe]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ca80ba6 ]

SOLR-9166: fix precommit


> Export handler returns zero for numeric fields that are not in the original 
> doc
> ---
>
> Key: SOLR-9166
> URL: https://issues.apache.org/jira/browse/SOLR-9166
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Erick Erickson
> Fix For: trunk, 6.4
>
> Attachments: SOLR-9166-6x.patch, SOLR-9166.patch, SOLR-9166.patch, 
> SOLR-9166.patch, SOLR-9166.patch
>
>
> From the dev list discussion:
> My original post.
> Zero is different from not
> existing. And let's claim that I want to process a stream and, say,
> facet on in integer field over the result set. There's no way on the
> client side to distinguish between a document that has a zero in the
> field and one that didn't have the field in the first place so I'll
> over-count the zero bucket.
> From Dennis Gove:
> Is this true for non-numeric fields as well? I agree that this seems like a 
> very bad thing.
> I can't imagine that a fix would cause a problem with Streaming Expressions, 
> ParallelSQL, or other given that the /select handler is not returning 0 for 
> these missing fields (the /select handler is the default handler for the 
> Streaming API so if nulls were a problem I imagine we'd have already seen 
> it). 
> That said, within Streaming Expressions there is a select(...) function which 
> supports a replace(...) operation which allows you to replace one value (or 
> null) with some other value. If a 0 were necessary one could use a 
> select(...) to replace null with 0 using an expression like this 
>select(, replace(fieldA, null, withValue=0)). 
> The end result of that would be that the field fieldA would never have a null 
> value and for all tuples where a null value existed it would be replaced with 
> 0.
> Details on the select function can be found at 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61330338#StreamingExpressions-select.
> And to answer Denis' question, null gets returned for string DocValues fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9766) Solr precommit is failing

[Steve Rowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe resolved SOLR-9766.
--
Resolution: Fixed
  Assignee: Steve Rowe

Thanks [~hgadre], I removed the unused imports, looks like it was an issue only 
on master.

> Solr precommit is failing
> -
>
> Key: SOLR-9766
> URL: https://issues.apache.org/jira/browse/SOLR-9766
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
>Assignee: Steve Rowe
>
> The Solr precommit is failing due to following errors,
> [ecj-lint] 36. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 31)
>  [ecj-lint]   import org.apache.solr.client.solrj.SolrQuery;
>  [ecj-lint]  ^^
>  [ecj-lint] The import org.apache.solr.client.solrj.SolrQuery is never used
>  [ecj-lint] --
>  [ecj-lint] 37. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 34)
>  [ecj-lint]   import org.apache.solr.client.solrj.impl.HttpSolrClient;
>  [ecj-lint]  
>  [ecj-lint] The import org.apache.solr.client.solrj.impl.HttpSolrClient is 
> never used
>  [ecj-lint] --
>  [ecj-lint] 38. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 52)
>  [ecj-lint]   import org.apache.solr.client.solrj.response.QueryResponse;
>  [ecj-lint]  ^^^
>  [ecj-lint] The import org.apache.solr.client.solrj.response.QueryResponse is 
> never used
>  [ecj-lint] --
>  [ecj-lint] 39. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 55)
>  [ecj-lint]   import org.apache.solr.common.SolrDocument;
>  [ecj-lint]  ^^^
>  [e
> These seem to be introduced by changes committed as part of SOLR-9166 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9766) Solr precommit is failing

[Hrishikesh Gadre (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665596#comment-15665596
 ] 

Hrishikesh Gadre commented on SOLR-9766:


[~erickerickson] Can you please take a look?

> Solr precommit is failing
> -
>
> Key: SOLR-9766
> URL: https://issues.apache.org/jira/browse/SOLR-9766
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
>
> The Solr precommit is failing due to following errors,
> [ecj-lint] 36. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 31)
>  [ecj-lint]   import org.apache.solr.client.solrj.SolrQuery;
>  [ecj-lint]  ^^
>  [ecj-lint] The import org.apache.solr.client.solrj.SolrQuery is never used
>  [ecj-lint] --
>  [ecj-lint] 37. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 34)
>  [ecj-lint]   import org.apache.solr.client.solrj.impl.HttpSolrClient;
>  [ecj-lint]  
>  [ecj-lint] The import org.apache.solr.client.solrj.impl.HttpSolrClient is 
> never used
>  [ecj-lint] --
>  [ecj-lint] 38. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 52)
>  [ecj-lint]   import org.apache.solr.client.solrj.response.QueryResponse;
>  [ecj-lint]  ^^^
>  [ecj-lint] The import org.apache.solr.client.solrj.response.QueryResponse is 
> never used
>  [ecj-lint] --
>  [ecj-lint] 39. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 55)
>  [ecj-lint]   import org.apache.solr.common.SolrDocument;
>  [ecj-lint]  ^^^
>  [e
> These seems to be introduced by changes committed as part of SOLR-9166 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9766) Solr precommit is failing

[Hrishikesh Gadre (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hrishikesh Gadre updated SOLR-9766:
---
Description: 
The Solr precommit is failing due to following errors,

[ecj-lint] 36. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 31)
 [ecj-lint] import org.apache.solr.client.solrj.SolrQuery;
 [ecj-lint]^^
 [ecj-lint] The import org.apache.solr.client.solrj.SolrQuery is never used
 [ecj-lint] --
 [ecj-lint] 37. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 34)
 [ecj-lint] import org.apache.solr.client.solrj.impl.HttpSolrClient;
 [ecj-lint]
 [ecj-lint] The import org.apache.solr.client.solrj.impl.HttpSolrClient is 
never used
 [ecj-lint] --
 [ecj-lint] 38. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 52)
 [ecj-lint] import org.apache.solr.client.solrj.response.QueryResponse;
 [ecj-lint]^^^
 [ecj-lint] The import org.apache.solr.client.solrj.response.QueryResponse is 
never used
 [ecj-lint] --
 [ecj-lint] 39. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 55)
 [ecj-lint] import org.apache.solr.common.SolrDocument;
 [ecj-lint]^^^
 [e

These seem to be introduced by changes committed as part of SOLR-9166 

  was:
The Solr precommit is failing due to following errors,

[ecj-lint] 36. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 31)
 [ecj-lint] import org.apache.solr.client.solrj.SolrQuery;
 [ecj-lint]^^
 [ecj-lint] The import org.apache.solr.client.solrj.SolrQuery is never used
 [ecj-lint] --
 [ecj-lint] 37. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 34)
 [ecj-lint] import org.apache.solr.client.solrj.impl.HttpSolrClient;
 [ecj-lint]
 [ecj-lint] The import org.apache.solr.client.solrj.impl.HttpSolrClient is 
never used
 [ecj-lint] --
 [ecj-lint] 38. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 52)
 [ecj-lint] import org.apache.solr.client.solrj.response.QueryResponse;
 [ecj-lint]^^^
 [ecj-lint] The import org.apache.solr.client.solrj.response.QueryResponse is 
never used
 [ecj-lint] --
 [ecj-lint] 39. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 55)
 [ecj-lint] import org.apache.solr.common.SolrDocument;
 [ecj-lint]^^^
 [e

These seems to be introduced by changes committed as part of SOLR-9166 


> Solr precommit is failing
> -
>
> Key: SOLR-9766
> URL: https://issues.apache.org/jira/browse/SOLR-9766
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
>
> The Solr precommit is failing due to following errors,
> [ecj-lint] 36. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 31)
>  [ecj-lint]   import org.apache.solr.client.solrj.SolrQuery;
>  [ecj-lint]  ^^
>  [ecj-lint] The import org.apache.solr.client.solrj.SolrQuery is never used
>  [ecj-lint] --
>  [ecj-lint] 37. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 34)
>  [ecj-lint]   import org.apache.solr.client.solrj.impl.HttpSolrClient;
>  [ecj-lint]  
>  [ecj-lint] The import org.apache.solr.client.solrj.impl.HttpSolrClient is 
> never used
>  [ecj-lint] --
>  [ecj-lint] 38. ERROR in 
> /Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
>  (at line 52)
>  [ecj-lint]   import org.apache.solr.client.solrj.response.QueryResponse;
>  [ecj-lint]  

[jira] [Created] (SOLR-9766) Solr precommit is failing

[Hrishikesh Gadre (JIRA)]

Hrishikesh Gadre created SOLR-9766:
--

 Summary: Solr precommit is failing
 Key: SOLR-9766
 URL: https://issues.apache.org/jira/browse/SOLR-9766
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Hrishikesh Gadre


The Solr precommit is failing due to following errors,

[ecj-lint] 36. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 31)
 [ecj-lint] import org.apache.solr.client.solrj.SolrQuery;
 [ecj-lint]^^
 [ecj-lint] The import org.apache.solr.client.solrj.SolrQuery is never used
 [ecj-lint] --
 [ecj-lint] 37. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 34)
 [ecj-lint] import org.apache.solr.client.solrj.impl.HttpSolrClient;
 [ecj-lint]
 [ecj-lint] The import org.apache.solr.client.solrj.impl.HttpSolrClient is 
never used
 [ecj-lint] --
 [ecj-lint] 38. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 52)
 [ecj-lint] import org.apache.solr.client.solrj.response.QueryResponse;
 [ecj-lint]^^^
 [ecj-lint] The import org.apache.solr.client.solrj.response.QueryResponse is 
never used
 [ecj-lint] --
 [ecj-lint] 39. ERROR in 
/Users/hgadre/git-repo/upstream/lucene-solr/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java
 (at line 55)
 [ecj-lint] import org.apache.solr.common.SolrDocument;
 [ecj-lint]^^^
 [e

These seems to be introduced by changes committed as part of SOLR-9166 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9751) PreAnalyzedField can cause managed schema corruption

[Steve Rowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe resolved SOLR-9751.
--
   Resolution: Fixed
Fix Version/s: 6.4
   master (7.0)

> PreAnalyzedField can cause managed schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9751.patch, SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9764) Design a memory efficient DocSet if a query returns all docs

[Michael Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Sun updated SOLR-9764:
--
Description: 
In some use cases, particularly use cases with time series data, using 
collection alias and partitioning data into multiple small collections using 
timestamp, a filter query can match all documents in a collection. Currently 
BitDocSet is used which contains a large array of long integers with every bits 
set to 1. After querying, the resulted DocSet saved in filter cache is large 
and becomes one of the main memory consumers in these use cases.

For example. suppose a Solr setup has 14 collections for data in last 14 days, 
each collection with one day of data. A filter query for last one week data 
would result in at least six DocSet in filter cache which matches all documents 
in six collections respectively.   

This is to design a new DocSet that is memory efficient for such a use case.  
The new DocSet removes the large array, reduces memory usage and GC pressure 
without losing advantage of large filter cache.

In particular, for use cases when using time series data, collection alias and 
partition data into multiple small collections using timestamp, the gain can be 
large.

For further optimization, it may be helpful to design a DocSet with run length 
encoding. Thanks [~mmokhtar] for suggestion. 

  was:
In some use cases, particularly use cases with time series data, using 
collection alias and partitioning data into multiple small collections using 
timestamp, a filter query can match all documents in a collection. Currently 
BitDocSet is used which contains a large array of long integers with every bits 
set to 1. After querying, the resulted DocSet saved in filter cache is large 
and becomes one of the main memory consumers in these use cases.

For example. suppose a Solr setup has 14 collections for data in last 14 days, 
each collection with one day of data. A filter query for last one week data 
would result in at least six DocSet in filter cache which matches all documents 
in six collections respectively.   

This is to design a new DocSet that is memory efficient for such a use case.  
The new DocSet removes the large array, reduces memory usage and GC pressure 
without losing advantage of large filter cache.

In particular, for use cases when using time series data, collection alias and 
partition data into multiple small collections using timestamp, the gain can be 
large.

For further optimization, it may be helpful to design a DocSet with run length 
encoding.


> Design a memory efficient DocSet if a query returns all docs
> 
>
> Key: SOLR-9764
> URL: https://issues.apache.org/jira/browse/SOLR-9764
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Sun
> Attachments: SOLR-9764.patch
>
>
> In some use cases, particularly use cases with time series data, using 
> collection alias and partitioning data into multiple small collections using 
> timestamp, a filter query can match all documents in a collection. Currently 
> BitDocSet is used which contains a large array of long integers with every 
> bits set to 1. After querying, the resulted DocSet saved in filter cache is 
> large and becomes one of the main memory consumers in these use cases.
> For example. suppose a Solr setup has 14 collections for data in last 14 
> days, each collection with one day of data. A filter query for last one week 
> data would result in at least six DocSet in filter cache which matches all 
> documents in six collections respectively.   
> This is to design a new DocSet that is memory efficient for such a use case.  
> The new DocSet removes the large array, reduces memory usage and GC pressure 
> without losing advantage of large filter cache.
> In particular, for use cases when using time series data, collection alias 
> and partition data into multiple small collections using timestamp, the gain 
> can be large.
> For further optimization, it may be helpful to design a DocSet with run 
> length encoding. Thanks [~mmokhtar] for suggestion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9751) PreAnalyzedField can cause managed schema corruption

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665591#comment-15665591
 ] 

Steve Rowe commented on SOLR-9751:
--

I opened SOLR-9765 to explore dealing with the general case of mixed schema 
modification request success/failure.  (On this issue, modification requests 
succeeded on the coordinating replica, but failed on other replicas.)

> PreAnalyzedField can cause managed schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-9751.patch, SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9765) In SolrCloud mode, schema modification should fail the operation and rollback to previous schema on all replicas if any replica fails to load the new schema

[Steve Rowe (JIRA)]

Steve Rowe created SOLR-9765:


 Summary: In SolrCloud mode, schema modification should fail the 
operation and rollback to previous schema on all replicas if any replica fails 
to load the new schema
 Key: SOLR-9765
 URL: https://issues.apache.org/jira/browse/SOLR-9765
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Steve Rowe


When a schema modification request succeeds on the coordinating replica but 
fails on one or more other replicas, the request should fail and the schema 
should be rolled back to the pre-request schema on all replicas.

An example of where this should have happened but didn't: SOLR-9751.

Another scenario where this might occur: if nodes don't all have the same 
classpath, schema modifications might succeed on one node but not on another.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9764) Design a memory efficient DocSet if a query returns all docs

[Michael Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Sun updated SOLR-9764:
--
Description: 
In some use cases, particularly use cases with time series data, using 
collection alias and partitioning data into multiple small collections using 
timestamp, a filter query can match all documents in a collection. Currently 
BitDocSet is used which contains a large array of long integers with every bits 
set to 1. After querying, the resulted DocSet saved in filter cache is large 
and becomes one of the main memory consumers in these use cases.

For example. suppose a Solr setup has 14 collections for data in last 14 days, 
each collection with one day of data. A filter query for last one week data 
would result in at least six DocSet in filter cache which matches all documents 
in six collections respectively.   

This is to design a new DocSet that is memory efficient for such a use case.  
The new DocSet removes the large array, reduces memory usage and GC pressure 
without losing advantage of large filter cache.

In particular, for use cases when using time series data, collection alias and 
partition data into multiple small collections using timestamp, the gain can be 
large.

For further optimization, it may be helpful to design a DocSet with run length 
encoding.

  was:
In some use cases, particularly use cases with time series data, using 
collection alias and partitioning data into multiple small collections using 
timestamp, a filter query can match all documents in a collection. Currently 
BitDocSet is used which contains a large array of long integers with every bits 
set to 1. After querying, the resulted DocSet saved in filter cache is large 
and becomes one of the main memory consumers in these use cases.

For example. suppose a Solr setup has 14 collections for data in last 14 days, 
each collection with one day of data. A filter query for last one week data 
would result in at least six DocSet in filter cache which matches all documents 
in six collections respectively.   

This is to design a new DocSet that is memory efficient for such a use case.  
The new DocSet removes the large array, reduces memory usage and GC pressure 
without losing advantage of large filter cache.

In particular, for use cases when using time series data, collection alias and 
partition data into multiple small collections using timestamp, the gain can be 
large.




> Design a memory efficient DocSet if a query returns all docs
> 
>
> Key: SOLR-9764
> URL: https://issues.apache.org/jira/browse/SOLR-9764
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Sun
> Attachments: SOLR-9764.patch
>
>
> In some use cases, particularly use cases with time series data, using 
> collection alias and partitioning data into multiple small collections using 
> timestamp, a filter query can match all documents in a collection. Currently 
> BitDocSet is used which contains a large array of long integers with every 
> bits set to 1. After querying, the resulted DocSet saved in filter cache is 
> large and becomes one of the main memory consumers in these use cases.
> For example. suppose a Solr setup has 14 collections for data in last 14 
> days, each collection with one day of data. A filter query for last one week 
> data would result in at least six DocSet in filter cache which matches all 
> documents in six collections respectively.   
> This is to design a new DocSet that is memory efficient for such a use case.  
> The new DocSet removes the large array, reduces memory usage and GC pressure 
> without losing advantage of large filter cache.
> In particular, for use cases when using time series data, collection alias 
> and partition data into multiple small collections using timestamp, the gain 
> can be large.
> For further optimization, it may be helpful to design a DocSet with run 
> length encoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9751) PreAnalyzedField can cause managed schema corruption

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665520#comment-15665520
 ] 

ASF subversion and git services commented on SOLR-9751:
---

Commit 8989c774783a80cab6902e45f111cfe60ed15d49 in lucene-solr's branch 
refs/heads/branch_6x from [~steve_rowe]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8989c77 ]

SOLR-9751: PreAnalyzedField can cause managed schema corruption


> PreAnalyzedField can cause managed schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-9751.patch, SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9751) PreAnalyzedField can cause managed schema corruption

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665521#comment-15665521
 ] 

ASF subversion and git services commented on SOLR-9751:
---

Commit 76b439a0bdf8a3e74f53130571535bbfdec5c771 in lucene-solr's branch 
refs/heads/master from [~steve_rowe]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=76b439a ]

SOLR-9751: PreAnalyzedField can cause managed schema corruption


> PreAnalyzedField can cause managed schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-9751.patch, SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9751) PreAnalyzedField can cause managed schema corruption

[Steve Rowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe updated SOLR-9751:
-
Attachment: SOLR-9751.patch

Patch, fixes a precommit issue in the last patch (unused imports), and removes 
the warnings I added for analyzers specified on non-{{TextField}}-s, since 
{{FieldType.set\{Index,Query\}Analyzer()}} already handles this case as a 
severe error.

Precommit and all tests pass.  Committing shortly.

> PreAnalyzedField can cause managed schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-9751.patch, SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Collection API for performance monitoring?

[Walter Underwood]

Because the current stats are not usable. They really should be removed from 
the code.

They calculate percentiles since the last collection load. We need to know 95th 
percentile
during the peak hour last night, not the 95th for the last month.

Right now, we run eleven collections in our Solr 4 cluster. In each collection, 
we have
several different handlers. Usually, one for autosuggest (instant results), one 
for the SRP,
and one for mobile, though we also have SEO requests and so on. We can track 
performance
for each of these.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 14, 2016, at 3:54 PM, Erick Erickson  wrote:
> 
> Point taken, and thanks for the link. The stats I'm referring to in
> this thread are available now, and would (I think) be a quick win. I
> don't have a huge amount of investment in it though, more "why didn't
> we think of this before?" followed by "maybe there's a very good
> reason not to bother". This may be it since we now standardize on
> Jetty. My question of course is whether this would be supported moving
> forward to netty or whatever...
> 
> Best,
> Erick
> 
> On Mon, Nov 14, 2016 at 3:44 PM, Walter Underwood  
> wrote:
>> I’m not fond of polling for performance stats. I’d rather have the app
>> report them.
>> 
>> We could integrate existing Jetty monitoring:
>> 
>> http://metrics.dropwizard.io/3.1.0/manual/jetty/
>> 
>> From our experience with a similar approach, we might need some
>> Solr-specific metric
>> conflation. SolrJ sends a request to /solr/collection/handler as
>> /solr/collection/select?qt=/handler.
>> In our code, we fix that request to the intended path. We’ve been running a
>> Tomcat metrics search
>> filter for three years.
>> 
>> Also, see:
>> 
>> https://issues.apache.org/jira/browse/SOLR-8785
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>> On Nov 14, 2016, at 3:25 PM, Erick Erickson  wrote:
>> 
>> What do people think about exposing a Collections API call (name TBD,
>> but the sense is PERFORMANCESTATS) that would simply issue the
>> admin/mbeans call to each replica of a collection and report them
>> back. This would give operations monitors the ability to see, say,
>> anomalous replicas that had poor average response times for the last 5
>> minutes and the like.
>> 
>> Seems like an easy enhancement that would make ops people's lives easier.
>> 
>> I'll raise a JIRA if there's interest, but sure won't make progress on
>> it until I clear my plate of some other JIRAs that I've let linger for
>> far too long.
>> 
>> Erick
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>> 
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 

Re: Collection API for performance monitoring?

[Erick Erickson]

Point taken, and thanks for the link. The stats I'm referring to in
this thread are available now, and would (I think) be a quick win. I
don't have a huge amount of investment in it though, more "why didn't
we think of this before?" followed by "maybe there's a very good
reason not to bother". This may be it since we now standardize on
Jetty. My question of course is whether this would be supported moving
forward to netty or whatever...

Best,
Erick

On Mon, Nov 14, 2016 at 3:44 PM, Walter Underwood  wrote:
> I’m not fond of polling for performance stats. I’d rather have the app
> report them.
>
> We could integrate existing Jetty monitoring:
>
> http://metrics.dropwizard.io/3.1.0/manual/jetty/
>
> From our experience with a similar approach, we might need some
> Solr-specific metric
> conflation. SolrJ sends a request to /solr/collection/handler as
> /solr/collection/select?qt=/handler.
> In our code, we fix that request to the intended path. We’ve been running a
> Tomcat metrics search
> filter for three years.
>
> Also, see:
>
> https://issues.apache.org/jira/browse/SOLR-8785
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>
> On Nov 14, 2016, at 3:25 PM, Erick Erickson  wrote:
>
> What do people think about exposing a Collections API call (name TBD,
> but the sense is PERFORMANCESTATS) that would simply issue the
> admin/mbeans call to each replica of a collection and report them
> back. This would give operations monitors the ability to see, say,
> anomalous replicas that had poor average response times for the last 5
> minutes and the like.
>
> Seems like an easy enhancement that would make ops people's lives easier.
>
> I'll raise a JIRA if there's interest, but sure won't make progress on
> it until I clear my plate of some other JIRAs that I've let linger for
> far too long.
>
> Erick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9764) Design a memory efficient DocSet if a query returns all docs

[Michael Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665419#comment-15665419
 ] 

Michael Sun commented on SOLR-9764:
---

Upload a prototype. For further optimization, the MatchAllDocSet can also 
override intersection() etc.

> Design a memory efficient DocSet if a query returns all docs
> 
>
> Key: SOLR-9764
> URL: https://issues.apache.org/jira/browse/SOLR-9764
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Sun
> Attachments: SOLR-9764.patch
>
>
> In some use cases, particularly use cases with time series data, using 
> collection alias and partitioning data into multiple small collections using 
> timestamp, a filter query can match all documents in a collection. Currently 
> BitDocSet is used which contains a large array of long integers with every 
> bits set to 1. After querying, the resulted DocSet saved in filter cache is 
> large and becomes one of the main memory consumers in these use cases.
> For example. suppose a Solr setup has 14 collections for data in last 14 
> days, each collection with one day of data. A filter query for last one week 
> data would result in at least six DocSet in filter cache which matches all 
> documents in six collections respectively.   
> This is to design a new DocSet that is memory efficient for such a use case.  
> The new DocSet removes the large array, reduces memory usage and GC pressure 
> without losing advantage of large filter cache.
> In particular, for use cases when using time series data, collection alias 
> and partition data into multiple small collections using timestamp, the gain 
> can be large.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665417#comment-15665417
 ] 

David Smiley commented on SOLR-9708:


Sure, put the proposed logic into the HighlightComponent. 

It's definitely a separate issue for the Standard & FVH to be fully 
configurable via params. 

> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9764) Design a memory efficient DocSet if a query returns all docs

[Michael Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Sun updated SOLR-9764:
--
Attachment: SOLR-9764.patch

> Design a memory efficient DocSet if a query returns all docs
> 
>
> Key: SOLR-9764
> URL: https://issues.apache.org/jira/browse/SOLR-9764
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Sun
> Attachments: SOLR-9764.patch
>
>
> In some use cases, particularly use cases with time series data, using 
> collection alias and partitioning data into multiple small collections using 
> timestamp, a filter query can match all documents in a collection. Currently 
> BitDocSet is used which contains a large array of long integers with every 
> bits set to 1. After querying, the resulted DocSet saved in filter cache is 
> large and becomes one of the main memory consumers in these use cases.
> For example. suppose a Solr setup has 14 collections for data in last 14 
> days, each collection with one day of data. A filter query for last one week 
> data would result in at least six DocSet in filter cache which matches all 
> documents in six collections respectively.   
> This is to design a new DocSet that is memory efficient for such a use case.  
> The new DocSet removes the large array, reduces memory usage and GC pressure 
> without losing advantage of large filter cache.
> In particular, for use cases when using time series data, collection alias 
> and partition data into multiple small collections using timestamp, the gain 
> can be large.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9764) Design a memory efficient DocSet if a query returns all docs

[Michael Sun (JIRA)]

Michael Sun created SOLR-9764:
-

 Summary: Design a memory efficient DocSet if a query returns all 
docs
 Key: SOLR-9764
 URL: https://issues.apache.org/jira/browse/SOLR-9764
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Michael Sun


In some use cases, particularly use cases with time series data, using 
collection alias and partitioning data into multiple small collections using 
timestamp, a filter query can match all documents in a collection. Currently 
BitDocSet is used which contains a large array of long integers with every bits 
set to 1. After querying, the resulted DocSet saved in filter cache is large 
and becomes one of the main memory consumers in these use cases.

For example. suppose a Solr setup has 14 collections for data in last 14 days, 
each collection with one day of data. A filter query for last one week data 
would result in at least six DocSet in filter cache which matches all documents 
in six collections respectively.   

This is to design a new DocSet that is memory efficient for such a use case.  
The new DocSet removes the large array, reduces memory usage and GC pressure 
without losing advantage of large filter cache.

In particular, for use cases when using time series data, collection alias and 
partition data into multiple small collections using timestamp, the gain can be 
large.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Collection API for performance monitoring?

[Walter Underwood]

I’m not fond of polling for performance stats. I’d rather have the app report 
them.

We could integrate existing Jetty monitoring:

http://metrics.dropwizard.io/3.1.0/manual/jetty/ 


From our experience with a similar approach, we might need some Solr-specific 
metric
conflation. SolrJ sends a request to /solr/collection/handler as 
/solr/collection/select?qt=/handler.
In our code, we fix that request to the intended path. We’ve been running a 
Tomcat metrics search
filter for three years.

Also, see:

https://issues.apache.org/jira/browse/SOLR-8785 


wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 14, 2016, at 3:25 PM, Erick Erickson  wrote:
> 
> What do people think about exposing a Collections API call (name TBD,
> but the sense is PERFORMANCESTATS) that would simply issue the
> admin/mbeans call to each replica of a collection and report them
> back. This would give operations monitors the ability to see, say,
> anomalous replicas that had poor average response times for the last 5
> minutes and the like.
> 
> Seems like an easy enhancement that would make ops people's lives easier.
> 
> I'll raise a JIRA if there's interest, but sure won't make progress on
> it until I clear my plate of some other JIRAs that I've let linger for
> far too long.
> 
> Erick
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 

[jira] [Created] (SOLR-9763) Remove the workaround implemented for HADOOP-12767

[Hrishikesh Gadre (JIRA)]

Hrishikesh Gadre created SOLR-9763:
--

 Summary: Remove the workaround implemented for HADOOP-12767
 Key: SOLR-9763
 URL: https://issues.apache.org/jira/browse/SOLR-9763
 Project: Solr
  Issue Type: Sub-task
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Hrishikesh Gadre
Priority: Minor


Here is the workaround that needs to be removed.
https://github.com/apache/lucene-solr/blob/c0b7edb5c858ce5f3e6308b9c32747c5e3729acc/solr/core/src/java/org/apache/solr/security/DelegationTokenKerberosFilter.java#L98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_102) - Build # 18284 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18284/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 67749 lines...]
-ecj-javadoc-lint-tests:
[mkdir] Created dir: /tmp/ecj922377887
 [ecj-lint] Compiling 95 source files to /tmp/ecj922377887
 [ecj-lint] invalid Class-Path header in manifest of jar file: 
/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: 
/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] --
 [ecj-lint] 1. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientMultiConstructorTest.java
 (at line 80)
 [ecj-lint] new CloudSolrClient(hosts, "foo");
 [ecj-lint] ^
 [ecj-lint] Resource leak: '' is never closed
 [ecj-lint] --
 [ecj-lint] --
 [ecj-lint] 2. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/io/graph/GraphExpressionTest.java
 (at line 893)
 [ecj-lint] throw new Exception(error);
 [ecj-lint] ^^^
 [ecj-lint] Resource leak: 'client' is not closed at this location
 [ecj-lint] --
 [ecj-lint] 3. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/io/graph/GraphExpressionTest.java
 (at line 905)
 [ecj-lint] throw new Exception(error);
 [ecj-lint] ^^^
 [ecj-lint] Resource leak: 'client' is not closed at this location
 [ecj-lint] --
 [ecj-lint] --
 [ecj-lint] 4. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExpessionTest.java
 (at line 77)
 [ecj-lint] stream = new 
CloudSolrStream(StreamExpressionParser.parse("search(collection1, q=*:*, 
fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_i asc\")"), factory);
 [ecj-lint] 
^^^
 [ecj-lint] Resource leak: 'stream' is never closed
 [ecj-lint] --
 [ecj-lint] 5. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExpessionTest.java
 (at line 99)
 [ecj-lint] stream = new 
SelectStream(StreamExpressionParser.parse("select(\"a_s as fieldA\", 
search(collection1, q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_i 
asc\"))"), factory);
 [ecj-lint] 
^^^
 [ecj-lint] Resource leak: 'stream' is never closed
 [ecj-lint] --
 [ecj-lint] 6. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExpessionTest.java
 (at line 116)
 [ecj-lint] stream = new 
DaemonStream(StreamExpressionParser.parse("daemon(search(collection1, q=*:*, 
fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_i asc\"), id=\"blah\", 
runInterval=\"1000\", queueSize=\"100\")"), factory);
 [ecj-lint] 
^^
 [ecj-lint] Resource leak: 'stream' is never closed
 [ecj-lint] --
 [ecj-lint] 7. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExpessionTest.java
 (at line 134)
 [ecj-lint] stream = new 
TopicStream(StreamExpressionParser.parse("topic(collection2, collection1, 
q=*:*, fl=\"id,a_s,a_i,a_f\", id=\"blah\", checkpointEvery=1000)"), factory);
 [ecj-lint] 
^^^
 [ecj-lint] Resource leak: 'stream' is never closed
 [ecj-lint] --
 [ecj-lint] 8. WARNING in 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExpessionTest.java
 (at line 150)
 [ecj-lint] stream = new 
StatsStream(StreamExpressionParser.parse("stats(collection1, q=*:*, 
fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_i asc\", sum(a_i), avg(a_i), count(*), 
min(a_i), max(a_i))"), factory);
 [ecj-lint] 

 [ecj-lint] Resource leak: 'stream' is never closed
 

Collection API for performance monitoring?

[Erick Erickson]

What do people think about exposing a Collections API call (name TBD,
but the sense is PERFORMANCESTATS) that would simply issue the
admin/mbeans call to each replica of a collection and report them
back. This would give operations monitors the ability to see, say,
anomalous replicas that had poor average response times for the last 5
minutes and the like.

Seems like an easy enhancement that would make ops people's lives easier.

I'll raise a JIRA if there's interest, but sure won't make progress on
it until I clear my plate of some other JIRAs that I've let linger for
far too long.

Erick

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9751) PreAnalyzedField can cause managed schema corruption

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665359#comment-15665359
 ] 

Steve Rowe commented on SOLR-9751:
--

Manual testing of the Schema API in the standalone Solr case shows the same 
problem as in the SolrCloud case, even though it didn't trigger system failure, 
so the problem is not confined to SolrCloud.

> PreAnalyzedField can cause managed schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9751) PreAnalyzedField can cause managed schema corruption

[Steve Rowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe updated SOLR-9751:
-
Summary: PreAnalyzedField can cause managed schema corruption  (was: 
PreanalyzedField can cause schema corruption)

> PreAnalyzedField can cause managed schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9751) PreanalyzedField can cause schema corruption

[Steve Rowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe updated SOLR-9751:
-
Attachment: SOLR-9751.patch

Patch with a fix.

The problem results from a combination of lenient schema parsing and the lack 
of a concept for non-user-specifiable index-time analyzers.

This patch adds a new interface {{HasImplicitIndexAnalyzer}}, implemented by 
{{PreAnalyzedField}}, and schema parsing and serialization now properly handle 
this case.  As a result, when a field type implements 
{{HasImplicitIndexAnalyzer}}, regardless of the original specified analyzer 
type, an analyzer (if any) will always be specified as a query-time analyzer, 
even if it was originally specified as a non-specific or index-time analyzer. 

I've also adding logged warnings for cases where analyzers are specified in the 
schema for field types that don't support analyzers (currently 
non-{{TextField}}-s).

I'll commit once all tests and precommit pass.

> PreanalyzedField can cause schema corruption
> 
>
> Key: SOLR-9751
> URL: https://issues.apache.org/jira/browse/SOLR-9751
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Affects Versions: 6.2, 6.3
>Reporter: liuyang
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-9751.patch, SOLR-9751.patch
>
>
> The exception as follows:
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> test_shard1_replica1: Can't load schema managed-schema: Plugin init failure 
> for [schema.xml] fieldType "preanalyzed": Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> managed-schema: Plugin init failure for [schema.xml] fieldType "preanalyzed": 
> Cannot load analyzer: 
> org.apache.solr.schema.PreAnalyzedField$PreAnalyzedAnalyzer
> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:600)
> at org.apache.solr.schema.IndexSchema.(IndexSchema.java:183)
> at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:172)
> at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
> at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
> at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
> at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
> ... 7 more
> Test procedure:
> 1.create collection using sample_techproducts_configs;
> 2.add field in Solr web view;
> 3.add field again in Solr web view.
> manage-schema is modifyed as follows:
> 
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9762) Remove the workaround implemented for HADOOP-13346

[Hrishikesh Gadre (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hrishikesh Gadre updated SOLR-9762:
---
Description: 
Here is the workaround that needs to be removed,
https://github.com/apache/lucene-solr/blob/c0b7edb5c858ce5f3e6308b9c32747c5e3729acc/solr/core/src/java/org/apache/solr/security/KerberosPlugin.java#L253


> Remove the workaround implemented for HADOOP-13346
> --
>
> Key: SOLR-9762
> URL: https://issues.apache.org/jira/browse/SOLR-9762
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
>Priority: Minor
>
> Here is the workaround that needs to be removed,
> https://github.com/apache/lucene-solr/blob/c0b7edb5c858ce5f3e6308b9c32747c5e3729acc/solr/core/src/java/org/apache/solr/security/KerberosPlugin.java#L253



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9762) Remove the workaround implemented for HADOOP-13346

[Hrishikesh Gadre (JIRA)]

Hrishikesh Gadre created SOLR-9762:
--

 Summary: Remove the workaround implemented for HADOOP-13346
 Key: SOLR-9762
 URL: https://issues.apache.org/jira/browse/SOLR-9762
 Project: Solr
  Issue Type: Sub-task
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Hrishikesh Gadre
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9761) Solr security related changes for Hadoop 3

[Hrishikesh Gadre (JIRA)]

Hrishikesh Gadre created SOLR-9761:
--

 Summary: Solr security related changes for Hadoop 3
 Key: SOLR-9761
 URL: https://issues.apache.org/jira/browse/SOLR-9761
 Project: Solr
  Issue Type: Task
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Hrishikesh Gadre


SOLR-9515 tracks the work required to update Solr codebase to work with Hadoop 
3. This jira is to track the updates required in the Solr security framework 
w.r.t Hadoop 3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7526) Improvements to UnifiedHighlighter OffsetStrategies

[Timothy M. Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665317#comment-15665317
 ] 

Timothy M. Rodriguez commented on LUCENE-7526:
--

Added the proposed changes.  I'm on the fence around the refactor for 
MultiValueTokenStream, I'd much prefer to get rid of it completely if we could. 
 But for now having some symmetry between the two impls seems worthwhile to me? 
 I'd like to punt on that one.

> Improvements to UnifiedHighlighter OffsetStrategies
> ---
>
> Key: LUCENE-7526
> URL: https://issues.apache.org/jira/browse/LUCENE-7526
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.4
>
>
> This ticket improves several of the UnifiedHighlighter FieldOffsetStrategies 
> by reducing reliance on creating or re-creating TokenStreams.
> The primary changes are as follows:
> * AnalysisOffsetStrategy - split into two offset strategies
>   ** MemoryIndexOffsetStrategy - the primary analysis mode that utilizes a 
> MemoryIndex for producing Offsets
>   ** TokenStreamOffsetStrategy - an offset strategy that avoids creating a 
> MemoryIndex.  Can only be used if the query distills down to terms and 
> automata.
> * TokenStream removal 
>   ** MemoryIndexOffsetStrategy - previously a TokenStream was created to fill 
> the memory index and then once consumed a new one was generated by 
> uninverting the MemoryIndex back into a TokenStream if there were automata 
> (wildcard/mtq queries) involved.  Now this is avoided, which should save 
> memory and avoid a second pass over the data.
>   ** TermVectorOffsetStrategy - this was refactored in a similar way to avoid 
> generating a TokenStream if automata are involved.
>   ** PostingsWithTermVectorsOffsetStrategy - similar refactoring
> * CompositePostingsEnum - aggregates several underlying PostingsEnums for 
> wildcard/mtq queries.  This should improve relevancy by providing unified 
> metrics for a wildcard across all it's term matches
> * Added a HighlightFlag for enabling the newly separated 
> TokenStreamOffsetStrategy since it can adversely affect passage relevancy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9166) Export handler returns zero for numeric fields that are not in the original doc

[Erick Erickson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson resolved SOLR-9166.
--
   Resolution: Fixed
Fix Version/s: 6.4
   trunk

Sorry for the pre-commit foo.

> Export handler returns zero for numeric fields that are not in the original 
> doc
> ---
>
> Key: SOLR-9166
> URL: https://issues.apache.org/jira/browse/SOLR-9166
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Erick Erickson
> Fix For: trunk, 6.4
>
> Attachments: SOLR-9166-6x.patch, SOLR-9166.patch, SOLR-9166.patch, 
> SOLR-9166.patch, SOLR-9166.patch
>
>
> From the dev list discussion:
> My original post.
> Zero is different from not
> existing. And let's claim that I want to process a stream and, say,
> facet on in integer field over the result set. There's no way on the
> client side to distinguish between a document that has a zero in the
> field and one that didn't have the field in the first place so I'll
> over-count the zero bucket.
> From Dennis Gove:
> Is this true for non-numeric fields as well? I agree that this seems like a 
> very bad thing.
> I can't imagine that a fix would cause a problem with Streaming Expressions, 
> ParallelSQL, or other given that the /select handler is not returning 0 for 
> these missing fields (the /select handler is the default handler for the 
> Streaming API so if nulls were a problem I imagine we'd have already seen 
> it). 
> That said, within Streaming Expressions there is a select(...) function which 
> supports a replace(...) operation which allows you to replace one value (or 
> null) with some other value. If a 0 were necessary one could use a 
> select(...) to replace null with 0 using an expression like this 
>select(, replace(fieldA, null, withValue=0)). 
> The end result of that would be that the field fieldA would never have a null 
> value and for all tuples where a null value existed it would be replaced with 
> 0.
> Details on the select function can be found at 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61330338#StreamingExpressions-select.
> And to answer Denis' question, null gets returned for string DocValues fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[Timothy M. Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665291#comment-15665291
 ] 

Timothy M. Rodriguez commented on SOLR-9708:


Thanks for catching those things. I've fixed them and pushed to the pr.

Regarding the hl.useUnifiedHighlighter I'm actually very in favor of that idea, 
but perhaps that logic would be better in the highlight component?  In that way 
the actual highlighters would be more like the facet params that help tweak 
with algorithm gets used.  I agree that we really shouldn't have to "configure" 
the highlighters.  Perhaps that should be a separate issue though more in line 
with the the other changes mentioned?

> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7456) PerField(DocValues|Postings)Format do not call the per-field merge methods

[Michael McCandless (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-7456:
---
Fix Version/s: 6.3
   master (7.0)

> PerField(DocValues|Postings)Format do not call the per-field merge methods
> --
>
> Key: LUCENE-7456
> URL: https://issues.apache.org/jira/browse/LUCENE-7456
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 6.2.1
>Reporter: Julien MASSENET
> Fix For: master (7.0), 6.3
>
> Attachments: LUCENE-7456-v2.patch, LUCENE-7456.patch
>
>
> While porting some old codec code from Lucene 4.3.1, I couldn't get the 
> per-field formats to call upon the per-field merge methods; the default merge 
> method was always being called.
> I think this is a side-effect of LUCENE-5894.
> Attached is a patch with a test that reproduces the error and an associated 
> fix that pass the unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9166) Export handler returns zero for numeric fields that are not in the original doc

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664998#comment-15664998
 ] 

ASF subversion and git services commented on SOLR-9166:
---

Commit c0b7edb5c858ce5f3e6308b9c32747c5e3729acc in lucene-solr's branch 
refs/heads/master from [~erickerickson]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c0b7edb ]

SOLR-9166: Export handler returns zero for numeric fields that are not in the 
original doc. Fixed precommit


> Export handler returns zero for numeric fields that are not in the original 
> doc
> ---
>
> Key: SOLR-9166
> URL: https://issues.apache.org/jira/browse/SOLR-9166
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Erick Erickson
> Attachments: SOLR-9166-6x.patch, SOLR-9166.patch, SOLR-9166.patch, 
> SOLR-9166.patch, SOLR-9166.patch
>
>
> From the dev list discussion:
> My original post.
> Zero is different from not
> existing. And let's claim that I want to process a stream and, say,
> facet on in integer field over the result set. There's no way on the
> client side to distinguish between a document that has a zero in the
> field and one that didn't have the field in the first place so I'll
> over-count the zero bucket.
> From Dennis Gove:
> Is this true for non-numeric fields as well? I agree that this seems like a 
> very bad thing.
> I can't imagine that a fix would cause a problem with Streaming Expressions, 
> ParallelSQL, or other given that the /select handler is not returning 0 for 
> these missing fields (the /select handler is the default handler for the 
> Streaming API so if nulls were a problem I imagine we'd have already seen 
> it). 
> That said, within Streaming Expressions there is a select(...) function which 
> supports a replace(...) operation which allows you to replace one value (or 
> null) with some other value. If a 0 were necessary one could use a 
> select(...) to replace null with 0 using an expression like this 
>select(, replace(fieldA, null, withValue=0)). 
> The end result of that would be that the field fieldA would never have a null 
> value and for all tuples where a null value existed it would be replaced with 
> 0.
> Details on the select function can be found at 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61330338#StreamingExpressions-select.
> And to answer Denis' question, null gets returned for string DocValues fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18283 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18283/
Java: 32bit/jdk-9-ea+140 -server -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 39167 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default 
locale]
[forbidden-apis]   in org.apache.solr.client.solrj.io.stream.StreamingTest 
(StreamingTest.java:989)
[forbidden-apis] Scanned 751 (and 725 related) class file(s) for forbidden API 
invocations (in 0.46s), 1 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:765: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:366: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:514: 
Check for forbidden API calls failed, see log.

Total time: 76 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6225) BasicDistributedZkTest.testDistribSearch failures

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664956#comment-15664956
 ] 

Cassandra Targett commented on SOLR-6225:
-

This is marked as a duplicate of SOLR-7774, which has been closed, and this has 
some commits itself. Can this be closed?

> BasicDistributedZkTest.testDistribSearch failures
> -
>
> Key: SOLR-6225
> URL: https://issues.apache.org/jira/browse/SOLR-6225
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud, update
>Affects Versions: 4.9
>Reporter: Shalin Shekhar Mangar
> Fix For: 4.10
>
>
> The org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch fails 
> often on jenkins:
> {code}
> 1 tests failed.
> REGRESSION:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch
> Error Message:
> commitWithin did not work expected:<68> but was:<67>
> Stack Trace:
> java.lang.AssertionError: commitWithin did not work expected:<68> but was:<67>
> at 
> __randomizedtesting.SeedInfo.seed([307BF2EDE49408D1:B19D7CF593CB68ED]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at 
> org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:357)
> at 
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4924) indices getting out of sync with SolrCloud

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664920#comment-15664920
 ] 

Cassandra Targett commented on SOLR-4924:
-

Have we ever been able to confirm this is fixed with SOLR-4260?

> indices getting out of sync with SolrCloud
> --
>
> Key: SOLR-4924
> URL: https://issues.apache.org/jira/browse/SOLR-4924
> Project: Solr
>  Issue Type: Bug
>  Components: replication (java), SolrCloud
>Affects Versions: 4.2
> Environment: Linux 2.6.18-308.16.1.el5 #1 SMP Tue Oct 2 22:01:43 EDT 
> 2012 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5.8 (Final)
> Solr 4.2.1
>Reporter: Ricardo Merizalde
>
> We are experiencing an issue in our production servers where the indices get 
> out of sync. Customers will see different results/result sorting depending of 
> the instance that serves the request.
> We currently have 2 instances with a single shard. This is our update handler 
> configuration
> 
>   
> 
> 60
> 
> 5000
> 
> false
>   
>   
> 
> 5000
>   
>   
> ${solr.data.dir:}
>   
> 
> When the indices get out of sync the follower replica ends up with a higher 
> version than the master. Optimizing the leader or reloading the follower core 
> doesn't not help. The only why to get the indices in sync is to restart the 
> server.
> This is an state example of the leader:
> version: 1102541
> numDocs: 214007
> maxDoc: 370861
> deletedDocs: 156854 
> While the follower core has the following state:
> version: 1109143
> numDocs: 213890
> maxDoc: 341585
> deletedDocs: 127695 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene 6.2.1 file format

[Michael McCandless]

Hi,

The codec file format docs were accidentally missing in 6.2.1, but
will be fixed for 6.4:
https://issues.apache.org/jira/browse/LUCENE-7532

Mike McCandless

http://blog.mikemccandless.com


On Mon, Nov 14, 2016 at 1:56 PM, adkumar  wrote:
> Hello all,
>
> I'm reading/learning Lucene and have a question that I have not been able to
> find any recent information on. I am looking for an in-depth description of
> the 6.2.1 Lucene file format, similar to this:
> http://hackerlabs.github.io/blog/2011/10/01/hacking-lucene-the-index-format/
>
> I have looked at the the documentation at:
> http://lucene.apache.org/core/6_3_0/core/org/apache/lucene/codecs/lucene62/package-summary.html#package.description
>
> Unfortunately I'm still a bit lost. Does anyone have suggestions on where to
> look next? There is plenty of information on older versions of Lucene, but
> I'd like to work with 6.x.
>
> Thanks
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Lucene-6-2-1-file-format-tp4305803.html
> Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9760) solr.cmd on Windows requires modify permissions in the current directory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664798#comment-15664798
 ] 

ASF GitHub Bot commented on SOLR-9760:
--

GitHub user afscrome opened a pull request:

https://github.com/apache/lucene-solr/pull/113

SOLR-9760 Avoid temporary files to determine java version

Avoid creating a temporary file so that solr does not require permissions 
in the current working directory.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/afscrome/lucene-solr patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #113


commit 5330ed23cfbb78650537a23d455ae92648b203dd
Author: Alex Crome 
Date:   2016-11-14T19:37:26Z

SOLR-9760 Avoid temporary files to determine java version

Avoid creating a temporary file so that solr does not require permissions 
in the current working directory.




> solr.cmd on Windows requires modify permissions in the current directory
> 
>
> Key: SOLR-9760
> URL: https://issues.apache.org/jira/browse/SOLR-9760
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Affects Versions: 6.3
> Environment: Windows
>Reporter: Alex Crome
>
> Currently starting solr fails if the user does not have permission to write 
> to the current directory.  This is caused by the resolve_java_vendor function 
> writing a temporary file to the current directory (javares). 
> {code}
> :resolve_java_vendor
> set "JAVA_VENDOR=Oracle"
> "%JAVA%" -version 2>&1 | findstr /i "IBM J9" > javares
> set /p JAVA_VENDOR_OUT= del javares
> if NOT "%JAVA_VENDOR_OUT%" == "" (
>   set "JAVA_VENDOR=IBM J9"
> )
> {code}
> Rather than writing this temporary file to disk, The exit code of findstr can 
> be used to determine if there is a match.  (0 == match, 1 == no match, 2 == 
> syntax error)
> {code}
> :resolve_java_vendor
> "%JAVA%" -version 2>&1 | findstr /i "IBM J9" > nul
> if %ERRORLEVEL% == 1 set "JAVA_VENDOR=Oracle" else set "JAVA_VENDOR=IBM J9"
> {code}
> By not writing this temp file, you can reduce the permissions solr needs.  As 
> a work around until this is fixed, you can start solr in a directory that has 
> the required permissions, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #113: SOLR-9760 Avoid temporary files to determine ...

[afscrome]

GitHub user afscrome opened a pull request:

https://github.com/apache/lucene-solr/pull/113

SOLR-9760 Avoid temporary files to determine java version

Avoid creating a temporary file so that solr does not require permissions 
in the current working directory.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/afscrome/lucene-solr patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #113


commit 5330ed23cfbb78650537a23d455ae92648b203dd
Author: Alex Crome 
Date:   2016-11-14T19:37:26Z

SOLR-9760 Avoid temporary files to determine java version

Avoid creating a temporary file so that solr does not require permissions 
in the current working directory.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9442) Add json.nl=arrnvp (array of NamedValuePair) style in JSONResponseWriter

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664719#comment-15664719
 ] 

Hoss Man commented on SOLR-9442:


I know i'm a little late to the party, but this new format seems overly 
cumbersome to parse -- for a given "NamedValuePair" the only constant is that 
_if_ the JSON "name" attribute exists, then it's value will be a String telling 
you the name of this Pair -- if it does not exist, then you know this pair has 
no name. But in order to know either the type or the value of the Pair you have 
to either know beforehand what to expect, or iterate -- either over the 
possibly type values ("int","str",etc...), or over the list of attribute keys 
in the Pair (ignoring "name").

It seems like a better structure (that would still maintain the original goal 
of parity with the amount of data returned in the XML format) would be..

{noformat}
NamedList("a"=1,"b"=2,null=3) 
  => [{"name":"a",   "type":"int","value":1},
  {"name":"b",   "type":"int","value":2},
  {"name":null,  "type":"int","value":3}]

NamedList("a"=1,"bar”=“foo",null=3.4f)
  => [{"name":"a",   "type":"int","value":1},
  {"name":"b",   "type":"str","value":"foo"},
  {"name":null,  "type":"float",  "value":3.4}]

NamedList("bar”=null,null=true)
  => [{"name":"bar", "type":"null",   "value":null},
  {"name":null,  "type":"bool",   "value":true}]
{noformat}

...the key improvement (in my mind) being:  The set of JSON attributes is fixed 
for all Pairs: "name", "type", and "value".  Clients can always look for those 
3 attributes to learn everything they can learn from the equivalent XML output.

[~cpoerschke] & [~jm100] - what do you think? better then what we have now?


> Add json.nl=arrnvp (array of NamedValuePair) style in JSONResponseWriter
> 
>
> Key: SOLR-9442
> URL: https://issues.apache.org/jira/browse/SOLR-9442
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Response Writers
>Reporter: Jonny Marks
>Assignee: Christine Poerschke
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9442.patch, SOLR-9442.patch, SOLR-9442.patch
>
>
> The JSONResponseWriter class currently supports several styles of NamedList 
> output format, documented on the wiki at http://wiki.apache.org/solr/SolJSON 
> and in the code at 
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/response/JSONResponseWriter.java#L71-L76.
> For example the 'arrmap' style:
> {code}NamedList("a"=1,"b"=2,null=3) => [{"a":1},{"b":2},3]
> NamedList("a"=1,"bar”=“foo",null=3.4f) => [{"a":1},{"bar”:”foo"},{3.4}]{code}
> This patch creates a new style ‘arrnvp’ which is an array of named value 
> pairs. For example:
> {code}NamedList("a"=1,"b"=2,null=3) => 
> [{"name":"a","int":1},{"name":"b","int":2},{"int":3}]
> NamedList("a"=1,"bar”=“foo",null=3.4f) => 
> [{"name":"a","int":1},{"name":"b","str":"foo"},{"float":3.4}]{code}
> This style maintains the type information of the values, similar to the xml 
> format:
> {code:xml}
>   1
>   foo
>   3.4
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664701#comment-15664701
 ] 

Julian Hyde commented on SOLR-8593:
---

CALCITE-1306 covers this. It's not standard SQL but could be enabled via an 
extension.

I disagree that "Solr will run this filter faster than Calcite". With query 
optimization, both queries will produce identical plans. This issue is not 
about performance. It is about syntactic sugar (not that there's anything wrong 
with that).

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Lucene 6.2.1 file format

[adkumar]

Hello all, 

I'm reading/learning Lucene and have a question that I have not been able to
find any recent information on. I am looking for an in-depth description of
the 6.2.1 Lucene file format, similar to this:
http://hackerlabs.github.io/blog/2011/10/01/hacking-lucene-the-index-format/

I have looked at the the documentation at: 
http://lucene.apache.org/core/6_3_0/core/org/apache/lucene/codecs/lucene62/package-summary.html#package.description

Unfortunately I'm still a bit lost. Does anyone have suggestions on where to
look next? There is plenty of information on older versions of Lucene, but
I'd like to work with 6.x.

Thanks



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Lucene-6-2-1-file-format-tp4305803.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664613#comment-15664613
 ] 

Cao Manh Dat edited comment on SOLR-8593 at 11/14/16 6:30 PM:
--

But we wanna to handle having clause without the function like ( Because Solr 
will run this filter faster than Calcite )
{code}
having field_i = 19
{code}
and left the other cases for Calcite to handle.

Is there any better ways to do this kind of filter?


was (Author: caomanhdat):
But we wanna to handle having clause without the function like
{code}
having field_i = 19
{code}
Is there any better ways to do this kind of filter?

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664613#comment-15664613
 ] 

Cao Manh Dat commented on SOLR-8593:


But we wanna to handle having clause without the function like
{code}
having field_i = 19
{code}
Is there any better ways to do this kind of filter?

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9760) solr.cmd on Windows requires modify permissions in the current directory

[Alex Crome (JIRA)]

Alex Crome created SOLR-9760:


 Summary: solr.cmd on Windows requires modify permissions in the 
current directory
 Key: SOLR-9760
 URL: https://issues.apache.org/jira/browse/SOLR-9760
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: Server
Affects Versions: 6.3
 Environment: Windows
Reporter: Alex Crome


Currently starting solr fails if the user does not have permission to write to 
the current directory.  This is caused by the resolve_java_vendor function 
writing a temporary file to the current directory (javares). 

{code}
:resolve_java_vendor
set "JAVA_VENDOR=Oracle"
"%JAVA%" -version 2>&1 | findstr /i "IBM J9" > javares
set /p JAVA_VENDOR_OUT=&1 | findstr /i "IBM J9" > nul
if %ERRORLEVEL% == 1 set "JAVA_VENDOR=Oracle" else set "JAVA_VENDOR=IBM J9"
{code}

By not writing this temp file, you can reduce the permissions solr needs.  As a 
work around until this is fixed, you can start solr in a directory that has the 
required permissions, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7543) Make changes-to-html target an offline operation

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664603#comment-15664603
 ] 

Steve Rowe commented on LUCENE-7543:


Thaks hoss, +1 to all your suggestions - I'd forgotten about the ability to put 
the doap.rdf files wherever we wanted.

{quote}
bq. 2. Put them in place in a cloned Lucene/Solr Git repo (probably at lucene/ 
and solr/).
I would suggest they might make more sense in dev-tools? just a gut feeling.
{quote}

Sure, maybe {{dev-tools/doap/lucene/doap.rdf}} and 
{{dev-tools/doap/solr/doap.rdf}} - alternatively I guess we could rename them 
to {{lucene-doap.rdf}} and {{solr-doap.rdf}} and put them directly in 
{{dev-tools/}}, but I prefer the former.

> Make changes-to-html target an offline operation
> 
>
> Key: LUCENE-7543
> URL: https://issues.apache.org/jira/browse/LUCENE-7543
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Steve Rowe
>
> Currently changes-to-html pulls release dates from JIRA, and so fails when 
> JIRA is inaccessible (e.g. from behind a firewall).
> SOLR-9711 advocates adding a build sysprop to ignore JIRA connection 
> failures, but I'd rather make the operation always offline.
> In an offline discussion, [~hossman] advocated moving Lucene's and Solr's 
> {{doap.rdf}} files, which contain all of the release dates that the 
> changes-to-html now pulls from JIRA, from the CMS Subversion repository 
> (downloadable from the website at http://lucene.apache.org/core/doap.rdf and 
> http://lucene.apache.org/solr/doap.rdf) to the Lucene/Solr git repository. If 
> we did that, then the process could be entirely offline if release dates were 
> taken from the local {{doap.rdf}} files instead of downloaded from JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664595#comment-15664595
 ] 

Julian Hyde commented on SOLR-8593:
---

You're making a mistake I see a lot of people making: trying to do complex 
semantic transformations on the AST (SqlNode). That's an anti-pattern, because 
SQL's complex rules for name-resolution make the AST very brittle. You should 
do those kinds of transformations on the relational algebra tree (RelNode).

In fact, Calcite will convert query into a {{Scan -> Filter -> Aggregate -> 
Filter -> Project}} logical plan (the first Filter is the WHERE clause, the 
second Filter is the HAVING clause), so I don't think you need to do any tricky 
processing looking for aliases.

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664576#comment-15664576
 ] 

Julian Hyde edited comment on SOLR-8593 at 11/14/16 6:11 PM:
-

Regarding the alias for "count(\*)". I guess one approach is to extend Calcite 
to allow a pluggable alias derivation (it has to be pluggable because you can't 
please everyone). Another approach is to leave the aliases as they are but 
generate field names for the JSON result set. Note that if you call 
SqlNode.getParserPosition() on each item in the select clause it will tell you 
the start and end point of that expression in the original SQL string, so you 
can extract the "count(\*)" using that information.

I don't think the the following should be valid, but under your proposed change 
it would be:

{code}
SELECT deptno
FROM (
  SELECT deptno, count(*)
  FROM emp
  GROUP BY deptno) AS t
WHERE t."count(*)" > 3
{code}

Note that "count(\*)" is not an expression; it is a reference to a "column" 
produced by the sub-query. In my opinion, using a textual expression is very 
confusing, and we should not do it. Derived alias of {{count(\*)}} should be 
something not easily guessable, which will encourage users to use an alias:

{code}
SELECT deptno
FROM (
  SELECT deptno, count(*) AS c
  FROM emp
  GROUP BY deptno) AS t
WHERE t.c > 3
{code}


was (Author: julianhyde):
Regarding the alias for "count(*)". I guess one approach is to extend Calcite 
to allow a pluggable alias derivation (it has to be pluggable because you can't 
please everyone). Another approach is to leave the aliases as they are but 
generate field names for the JSON result set. Note that if you call 
SqlNode.getParserPosition() on each item in the select clause it will tell you 
the start and end point of that expression in the original SQL string, so you 
can extract the "count(*)" using that information.

I don't think the the following should be valid, but under your proposed change 
it would be:

{code}
SELECT deptno
FROM (
  SELECT deptno, count(\*)
  FROM emp
  GROUP BY deptno) AS t
WHERE t."count(*)" > 3
{code}

Note that "count(\*)" is not an expression; it is a reference to a "column" 
produced by the sub-query. In my opinion, using a textual expression is very 
confusing, and we should not do it. Derived alias of {{count(\*)}} should be 
something not easily guessable, which will encourage users to use an alias:

{code}
SELECT deptno
FROM (
  SELECT deptno, count(\*) AS c
  FROM emp
  GROUP BY deptno) AS t
WHERE t.c > 3
{code}

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7543) Make changes-to-html target an offline operation

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664580#comment-15664580
 ] 

Hoss Man commented on LUCENE-7543:
--

Thanks for filing this jira steve, I'd forgotten about this conversation...

bq. So, what uses doap.rdf and how is it kept up to date? And if it is moved to 
GitHub, do we abandon the original location/processes?

The doap.rdf files are published publicly on the lucene website for consumption 
by any system (internal or external) that wants to know about the history of 
solr/lucene releases -- DOAP is an RDF standard for this purpose, much like 
FOAF.

Notably the doap.rdf files power projects.apache.org...

* https://projects.apache.org/doap.html
* https://projects.apache.org/committee.html?lucene
* https://projects.apache.org/project.html?lucene-core
* https://projects.apache.org/project.html?lucene-solr

In theory the DOAP files could be auto generated by some system like Jira, but 
it's a good idea to edit them manually for the express purpose of being able to 
correct mistakes (ie: they are living documents)

bq. An extra release step will be required if we move it to the Git repo, so 
that the files will continue to be present on the website, similarly to how 
javadocs are now copied to the website.

that would not be neccessary...

from an external perspective, the doap files just have to be available at some 
public / well-known URL.  traditionally that has been...

* http://lucene.apache.org/core/doap.rdf
* http://lucene.apache.org/solr/doap.rdf

...but there is no reason the well known & public URLs can't be things like 
"https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;f=solr/doap.rdf;a=blob_plain;hb=HEAD;
 that are served directly form source control (the projects.apache.org 
instructions actually suggest that approach) and we could add some .htaccess 
rules to the website to make the existing URLs redirect to new URLs served 
directly from GIT.

>From an internal perspective we just have to update this list to point 
>wherever we want...
https://svn.apache.org/repos/asf/comdev/projects.apache.org/data/projects.xml

bq. 2. Put them in place in a cloned Lucene/Solr Git repo (probably at lucene/ 
and solr/).

I would suggest they might make more sense in dev-tools?  just a gut feeling.

> Make changes-to-html target an offline operation
> 
>
> Key: LUCENE-7543
> URL: https://issues.apache.org/jira/browse/LUCENE-7543
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Steve Rowe
>
> Currently changes-to-html pulls release dates from JIRA, and so fails when 
> JIRA is inaccessible (e.g. from behind a firewall).
> SOLR-9711 advocates adding a build sysprop to ignore JIRA connection 
> failures, but I'd rather make the operation always offline.
> In an offline discussion, [~hossman] advocated moving Lucene's and Solr's 
> {{doap.rdf}} files, which contain all of the release dates that the 
> changes-to-html now pulls from JIRA, from the CMS Subversion repository 
> (downloadable from the website at http://lucene.apache.org/core/doap.rdf and 
> http://lucene.apache.org/solr/doap.rdf) to the Lucene/Solr git repository. If 
> we did that, then the process could be entirely offline if release dates were 
> taken from the local {{doap.rdf}} files instead of downloaded from JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664576#comment-15664576
 ] 

Julian Hyde commented on SOLR-8593:
---

Regarding the alias for "count(*)". I guess one approach is to extend Calcite 
to allow a pluggable alias derivation (it has to be pluggable because you can't 
please everyone). Another approach is to leave the aliases as they are but 
generate field names for the JSON result set. Note that if you call 
SqlNode.getParserPosition() on each item in the select clause it will tell you 
the start and end point of that expression in the original SQL string, so you 
can extract the "count(*)" using that information.

I don't think the the following should be valid, but under your proposed change 
it would be:

{code}
SELECT deptno
FROM (
  SELECT deptno, count(\*)
  FROM emp
  GROUP BY deptno) AS t
WHERE t."count(*)" > 3
{code}

Note that "count(\*)" is not an expression; it is a reference to a "column" 
produced by the sub-query. In my opinion, using a textual expression is very 
confusing, and we should not do it. Derived alias of {{count(\*)}} should be 
something not easily guessable, which will encourage users to use an alias:

{code}
SELECT deptno
FROM (
  SELECT deptno, count(\*) AS c
  FROM emp
  GROUP BY deptno) AS t
WHERE t.c > 3
{code}

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7495) Unexpected docvalues type NUMERIC when grouping by a int facet

[Matt Hilt (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664554#comment-15664554
 ] 

Matt Hilt commented on SOLR-7495:
-

I have been testing this patch out on Solr 6.x. It does correct the problem (at 
least on 6.0.1), but only when the field does NOT have docValues enabled. Is 
this expected? Can the existing patch be extended to support docValues too, or 
is that a completely separate implementation?


> Unexpected docvalues type NUMERIC when grouping by a int facet
> --
>
> Key: SOLR-7495
> URL: https://issues.apache.org/jira/browse/SOLR-7495
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.0, 5.1, 5.2, 5.3
>Reporter: Fabio Batista da Silva
> Attachments: SOLR-7495.patch, SOLR-7495.patch
>
>
> Hey All,
> After upgrading from solr 4.10 to 5.1 with solr could
> I'm getting a IllegalStateException when i try to facet a int field.
> IllegalStateException: unexpected docvalues type NUMERIC for field 'year' 
> (expected=SORTED). Use UninvertingReader or index with docvalues.
> schema.xml
> {code}
> 
> 
> 
> 
> 
> 
>  multiValued="false" required="true"/>
>  multiValued="false" required="true"/>
> 
> 
>  stored="true"/>
> 
> 
> 
>  />
>  sortMissingLast="true"/>
>  positionIncrementGap="0"/>
>  positionIncrementGap="0"/>
>  positionIncrementGap="0"/>
>  precisionStep="0" positionIncrementGap="0"/>
>  positionIncrementGap="0"/>
>  positionIncrementGap="100">
> 
> 
>  words="stopwords.txt" />
> 
>  maxGramSize="15"/>
> 
> 
> 
>  words="stopwords.txt" />
>  synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> 
> 
> 
>  positionIncrementGap="100">
> 
> 
>  words="stopwords.txt" />
> 
>  maxGramSize="15"/>
> 
> 
> 
>  words="stopwords.txt" />
>  synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> 
> 
> 
>  class="solr.SpatialRecursivePrefixTreeFieldType" geo="true" 
> distErrPct="0.025" maxDistErr="0.09" units="degrees" />
> 
> id
> name
> 
> 
> {code}
> query :
> {code}
> http://solr.dev:8983/solr/my_collection/select?wt=json=id=index_type:foobar=true=year_make_model=true=true=year
> {code}
> Exception :
> {code}
> ull:org.apache.solr.common.SolrException: Exception during facet.field: year
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:627)
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:566)
> at 
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)
> at 
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)
> at 
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> 

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664525#comment-15664525
 ] 

Cao Manh Dat commented on SOLR-8593:


Thanks [~risdenk], 

Yeah, I will definitely do that in the future.

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18282 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18282/
Java: 32bit/jdk-9-ea+140 -client -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 39151 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default 
locale]
[forbidden-apis]   in org.apache.solr.client.solrj.io.stream.StreamingTest 
(StreamingTest.java:989)
[forbidden-apis] Scanned 751 (and 725 related) class file(s) for forbidden API 
invocations (in 0.45s), 1 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:765: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:366: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:514: 
Check for forbidden API calls failed, see log.

Total time: 74 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-4049) Spellcheck NPE

[Cassandra Targett (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett closed SOLR-4049.
---
   Resolution: Won't Fix
Fix Version/s: (was: 6.0)
   (was: 4.9)

Closing based on no response to my comments from Sep 16.

> Spellcheck NPE
> --
>
> Key: SOLR-4049
> URL: https://issues.apache.org/jira/browse/SOLR-4049
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud, spellchecker
>Affects Versions: 4.0, 6.0
> Environment: 5.0.0.2012.11.07.11.39.14
>Reporter: Markus Jelsma
>
> In SolrCloud when we have some shard unavailable we get the followine NPE in 
> the SpellcheckComponent
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.solr.handler.component.SpellCheckComponent.finishStage(SpellCheckComponent.java:297)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:315)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1830)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:476)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
>   at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>   at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>   at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>   at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>   at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>   at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>   at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>   at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>   at 
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>   at 
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
>   at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>   at java.lang.Thread.run(Thread.java:662)
> {code}
> We can trigger this error by misconfiguring a node so it returns an HTTP 500.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9633) Limit FastLRUCache by RAM Usage

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar updated SOLR-9633:

Attachment: SOLR-9633.patch

Thanks Yonik and Michael for the reviews. This patch wrap all updates to 
ramBytes in the put() method with the check as requested. I'll commit shortly.

Michael, to your comment on this working only with Accountable, the example 
configurations specify this parameter only for filterCache so I feel it is 
unlikely that users will try to use FastLRUCache with maxRamMB for something 
else. But I'll make sure to include this point in the reference guide when 
documenting this feature.

> Limit FastLRUCache by RAM Usage
> ---
>
> Key: SOLR-9633
> URL: https://issues.apache.org/jira/browse/SOLR-9633
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9633.patch, SOLR-9633.patch
>
>
> SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on 
> memory usage. That helps with the query result cache but not with the filter 
> cache which defaults to FastLRUCache. This issue intends to add the same 
> feature to FastLRUCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Kevin Risden (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664433#comment-15664433
 ] 

Kevin Risden commented on SOLR-8593:


[~caomanhdat] - I merged your changes with the jira/solr-8593 branch. The pr 
104 was updated automatically.

In the future, I think you can open a PR against the existing jira/solr-8593 
branch. This would probably be easier than a patch to merge changes.

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch

> Support updates of numeric DocValues
> 
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
>  Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, 
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, 
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, 
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be 
> really nice to have Solr support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547986#comment-15547986
 ] 

Ishan Chattopadhyaya edited comment on SOLR-5944 at 11/14/16 5:03 PM:
--

Updated patch, brought up to master. Here are my replies inline (not all of 
them, but I'll keep editing this comment to provide all replies).


Ok -- it took a while, but here's my notes after reviewing the latest patch

{panel:title=DistributedUpdateProcessor}

* waitForDependentUpdates {color:green}FIXED{color}
** I know you & shalin went back and forth a bit on the wait call (ie: 
wait(100) with max retries vs wait(5000)) but i think the way things settled 
out {{bucket.wait(waitTimeout.timeLeft(TimeUnit.MILLISECONDS));}} would be 
better then a generic {{wait(5000)}}
*** consider the scenerio where: the dependent update is never going to come; a 
spurious notify/wake happens during the first "wait" call @ 4950ms; the 
lookupVersion call takes 45ms.  Now we've only got 5ms left on our original 
TimeOut, but we _could_ wind up "wait"ing another full 5s (total of 10s) unless 
we get another spurrious notify/wake inthe mean time.
** {{log.info("Fetched the update: " + missingUpdate);}} that's a really good 
candidate for templating since the AddUpdateCommand.toString() could be 
expensive if log.info winds up being a no-op (ie: {{log.info("Fetched the 
update: \{\}", missingUpdate);}})

* fetchMissingUpdateFromLeader
** In response to a previous question you said...{quote}
[FIXED. Initially, I wanted to fetch all missing updates, i.e. from what we 
have till what we want. Noble suggested that fetching only one at a time makes 
more sense.]
{quote} ... but from what i can tell skimming RTGC.processGetUpdates() it's 
still possible that multiple updates will be returned, notably in the case 
where: {{// Must return all delete-by-query commands that occur after the first 
add requested}}.  How is that possibility handled in the code paths that use 
fetchMissingUpdateFromLeader?
*** that seems like a scenerio that would be really easy to test for -- similar 
to how outOfOrderDeleteUpdatesIndividualReplicaTest works
** {{assert ((List) missingUpdates).size() == 1: "More than 1 update ...}}
*** based on my skimming of the code, an empty list is just as possible, so the 
assertion is missleading (ideally it should say how many updates it got, or 
maybe toString() the whole List ?)

{panel}


{panel:title=AtomicUpdateDocumentMerger}

* isSupportedFieldForInPlaceUpdate
** javadocs

* getFieldNamesFromIndex
** javadocs
** method name seems VERY missleading considering what it does 
{color:green}Changed it to getSearcherNonStoredDVs{color}

* isInPlaceUpdate
** javadocs should be clear what hapens to inPlaceUpdatedFields if result is 
false (even if answer is "undefined"
** based on usage, wouldn't it be simplier if instead of returning a boolean, 
this method just returned a (new) Set of inplace update fields found, and if 
the set is empty that means it's not an in place update? 
{color:green}FIXED{color}
** isn't getFieldNamesFromIndex kind of an expensive method to call on every 
AddUpdateCommand ?
*** couldn't this list of fields be created by the caller and re-used at least 
for the entire request (ie: when adding multiple docs) ? {color:green}The set 
returned is precomputed upon the opening of a searcher. The only cost I see is 
to create a new unmodifiableSet every time. I'd prefer to take up this 
optimization later, if needed.{color}
** {{if (indexFields.contains(fieldName) == false && 
schema.isDynamicField(fieldName))}}
*** why does it matter one way or the other if it's a dynamicField? 
{color:green}Changed the logic to check in the IW for presence of field. Added 
a comment: "// if dynamic field and this field doesn't exist, DV update can't 
work"{color}
** the special {{DELETED}} sentinal value still isn't being checked against the 
return value of {{getInputDocumentFromTlog}} {color:green}Not using 
getInputDocumentFromTlog call anymore{color}
** this method still seems like it could/should do "cheaper" validation (ie: 
not requiring SchemaField object creation, or tlog lookups) first.  (Ex: the 
set of supported atomic ops are checked after isSupportedFieldForInPlaceUpdate 
& a possible read from the tlog). {color:green}FIXED{color}
*** My suggested rewrite would be something like...{code}
Set candidateResults = new HashSet<>();
// first pass, check the things that are virtually free,
// and bail out early if anything is obviously not a valid in-place update
for (String fieldName : sdoc.getFieldNames()) {
  if (schema.getUniqueKeyField().getName().equals(fieldName)
  || fieldName.equals(DistributedUpdateProcessor.VERSION_FIELD)) {
continue;
  }
  Object fieldValue = sdoc.getField(fieldName).getValue();
  if (! (fieldValue instanceof Map) ) {
// not even an atomic update, definitely not an 

[jira] [Commented] (SOLR-9166) Export handler returns zero for numeric fields that are not in the original doc

[Shalin Shekhar Mangar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664404#comment-15664404
 ] 

Shalin Shekhar Mangar commented on SOLR-9166:
-

This has broken precommit. Please fix.

> Export handler returns zero for numeric fields that are not in the original 
> doc
> ---
>
> Key: SOLR-9166
> URL: https://issues.apache.org/jira/browse/SOLR-9166
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Erick Erickson
> Attachments: SOLR-9166-6x.patch, SOLR-9166.patch, SOLR-9166.patch, 
> SOLR-9166.patch, SOLR-9166.patch
>
>
> From the dev list discussion:
> My original post.
> Zero is different from not
> existing. And let's claim that I want to process a stream and, say,
> facet on in integer field over the result set. There's no way on the
> client side to distinguish between a document that has a zero in the
> field and one that didn't have the field in the first place so I'll
> over-count the zero bucket.
> From Dennis Gove:
> Is this true for non-numeric fields as well? I agree that this seems like a 
> very bad thing.
> I can't imagine that a fix would cause a problem with Streaming Expressions, 
> ParallelSQL, or other given that the /select handler is not returning 0 for 
> these missing fields (the /select handler is the default handler for the 
> Streaming API so if nulls were a problem I imagine we'd have already seen 
> it). 
> That said, within Streaming Expressions there is a select(...) function which 
> supports a replace(...) operation which allows you to replace one value (or 
> null) with some other value. If a 0 were necessary one could use a 
> select(...) to replace null with 0 using an expression like this 
>select(, replace(fieldA, null, withValue=0)). 
> The end result of that would be that the field fieldA would never have a null 
> value and for all tuples where a null value existed it would be replaced with 
> 0.
> Details on the select function can be found at 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61330338#StreamingExpressions-select.
> And to answer Denis' question, null gets returned for string DocValues fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7543) Make changes-to-html target an offline operation

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664343#comment-15664343
 ] 

Steve Rowe commented on LUCENE-7543:


bq. [~steve_rowe], is there anything I could help with this task? If you share 
the details of the planned implementation, I could create a patch.

Yes, that would be great, thanks!.  Here's what I was going to do (but haven't 
really started #3 yet):

# Download the doap.rdf files
# Put them in place in a cloned Lucene/Solr Git repo (probably at {{lucene/}} 
and {{solr/}}).
# Edit these two files to contain the same dates as JIRA has - you can look at 
how current changes-to-html pulls release info from JIRA and do the same thing 
to get the raw data.  If there are conflicts, I think JIRA is probably more 
authoritative.
# Figure out how to parse the doap.rdf files, rather than the JIRA version 
output.  (I'm being vague because I haven't looked at details of what's 
involved here.)
# Use doap.rdf release dates rather than those from JIRA.
# Stop downloading version data from JIRA.


> Make changes-to-html target an offline operation
> 
>
> Key: LUCENE-7543
> URL: https://issues.apache.org/jira/browse/LUCENE-7543
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Steve Rowe
>
> Currently changes-to-html pulls release dates from JIRA, and so fails when 
> JIRA is inaccessible (e.g. from behind a firewall).
> SOLR-9711 advocates adding a build sysprop to ignore JIRA connection 
> failures, but I'd rather make the operation always offline.
> In an offline discussion, [~hossman] advocated moving Lucene's and Solr's 
> {{doap.rdf}} files, which contain all of the release dates that the 
> changes-to-html now pulls from JIRA, from the CMS Subversion repository 
> (downloadable from the website at http://lucene.apache.org/core/doap.rdf and 
> http://lucene.apache.org/solr/doap.rdf) to the Lucene/Solr git repository. If 
> we did that, then the process could be entirely offline if release dates were 
> taken from the local {{doap.rdf}} files instead of downloaded from JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1153 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1153/

7 tests failed.
FAILED:  
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
at 
__randomizedtesting.SeedInfo.seed([AF34898F071538B4:71BEE7F23CB75A47]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:184)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:141)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:136)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
at 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR(LeaderInitiatedRecoveryOnShardRestartTest.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 

[jira] [Commented] (LUCENE-7543) Make changes-to-html target an offline operation

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664311#comment-15664311
 ] 

Steve Rowe commented on LUCENE-7543:


bq. So, what uses doap.rdf and how is it kept up to date?

I'm not sure what uses it, but keeping it up-to-date is part of the ReleaseToDo.

bq. And if it is moved to GitHub, do we abandon the original location/processes?

It won't be moved to GitHub of course, but rather to the Apache Git repo.

These files are currently hosted in the Lucene/Solr CMS-based website 
repository.  An extra release step will be required if we move it to the Git 
repo, so that the files will continue to be present on the website, similarly 
to how javadocs are now copied to the website.  (I don't think keeping two 
copies of these files, one in Git and one in the CMS repo, would be a good 
alternative.)

> Make changes-to-html target an offline operation
> 
>
> Key: LUCENE-7543
> URL: https://issues.apache.org/jira/browse/LUCENE-7543
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Steve Rowe
>
> Currently changes-to-html pulls release dates from JIRA, and so fails when 
> JIRA is inaccessible (e.g. from behind a firewall).
> SOLR-9711 advocates adding a build sysprop to ignore JIRA connection 
> failures, but I'd rather make the operation always offline.
> In an offline discussion, [~hossman] advocated moving Lucene's and Solr's 
> {{doap.rdf}} files, which contain all of the release dates that the 
> changes-to-html now pulls from JIRA, from the CMS Subversion repository 
> (downloadable from the website at http://lucene.apache.org/core/doap.rdf and 
> http://lucene.apache.org/solr/doap.rdf) to the Lucene/Solr git repository. If 
> we did that, then the process could be entirely offline if release dates were 
> taken from the local {{doap.rdf}} files instead of downloaded from JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8332) factor HttpShardHandler[Factory]'s url shuffling out into a ReplicaListTransformer class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664307#comment-15664307
 ] 

ASF GitHub Bot commented on SOLR-8332:
--

Github user cpoerschke commented on the issue:

https://github.com/apache/lucene-solr/pull/102
  
Closing out after committing.


> factor HttpShardHandler[Factory]'s url shuffling out into a 
> ReplicaListTransformer class
> 
>
> Key: SOLR-8332
> URL: https://issues.apache.org/jira/browse/SOLR-8332
> Project: Solr
>  Issue Type: Wish
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Fix For: 6.x, master (7.0)
>
> Attachments: SOLR-8332.patch, SOLR-8332.patch, SOLR-8332.patch, 
> SOLR-8332.patch
>
>
> Proposed patch against trunk to follow. No change in behaviour intended. This 
> would be as a step towards SOLR-6730.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664306#comment-15664306
 ] 

Ishan Chattopadhyaya commented on SOLR-5944:


Replied to review comments inline here:
https://issues.apache.org/jira/browse/SOLR-5944?focusedCommentId=15547986=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15547986

> Support updates of numeric DocValues
> 
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
>  Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, 
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, 
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, 
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, 
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be 
> really nice to have Solr support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #102: SOLR-8332: factor HttpShardHandler[Factory]'s url sh...

[cpoerschke]

Github user cpoerschke commented on the issue:

https://github.com/apache/lucene-solr/pull/102
  
Closing out after committing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8332) factor HttpShardHandler[Factory]'s url shuffling out into a ReplicaListTransformer class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664305#comment-15664305
 ] 

ASF GitHub Bot commented on SOLR-8332:
--

Github user cpoerschke closed the pull request at:

https://github.com/apache/lucene-solr/pull/102


> factor HttpShardHandler[Factory]'s url shuffling out into a 
> ReplicaListTransformer class
> 
>
> Key: SOLR-8332
> URL: https://issues.apache.org/jira/browse/SOLR-8332
> Project: Solr
>  Issue Type: Wish
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Fix For: 6.x, master (7.0)
>
> Attachments: SOLR-8332.patch, SOLR-8332.patch, SOLR-8332.patch, 
> SOLR-8332.patch
>
>
> Proposed patch against trunk to follow. No change in behaviour intended. This 
> would be as a step towards SOLR-6730.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #102: SOLR-8332: factor HttpShardHandler[Factory]'s...

[cpoerschke]

Github user cpoerschke closed the pull request at:

https://github.com/apache/lucene-solr/pull/102


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch

> Support updates of numeric DocValues
> 
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
>  Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, 
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, 
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, 
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, 
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be 
> really nice to have Solr support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547986#comment-15547986
 ] 

Ishan Chattopadhyaya edited comment on SOLR-5944 at 11/14/16 4:13 PM:
--

Updated patch, brought up to master. Here are my replies inline (not all of 
them, but I'll keep editing this comment to provide all replies).


Ok -- it took a while, but here's my notes after reviewing the latest patch

{panel:title=DistributedUpdateProcessor}

* waitForDependentUpdates {color:green}FIXED{color}
** I know you & shalin went back and forth a bit on the wait call (ie: 
wait(100) with max retries vs wait(5000)) but i think the way things settled 
out {{bucket.wait(waitTimeout.timeLeft(TimeUnit.MILLISECONDS));}} would be 
better then a generic {{wait(5000)}}
*** consider the scenerio where: the dependent update is never going to come; a 
spurious notify/wake happens during the first "wait" call @ 4950ms; the 
lookupVersion call takes 45ms.  Now we've only got 5ms left on our original 
TimeOut, but we _could_ wind up "wait"ing another full 5s (total of 10s) unless 
we get another spurrious notify/wake inthe mean time.
** {{log.info("Fetched the update: " + missingUpdate);}} that's a really good 
candidate for templating since the AddUpdateCommand.toString() could be 
expensive if log.info winds up being a no-op (ie: {{log.info("Fetched the 
update: \{\}", missingUpdate);}})

* fetchMissingUpdateFromLeader
** In response to a previous question you said...{quote}
[FIXED. Initially, I wanted to fetch all missing updates, i.e. from what we 
have till what we want. Noble suggested that fetching only one at a time makes 
more sense.]
{quote} ... but from what i can tell skimming RTGC.processGetUpdates() it's 
still possible that multiple updates will be returned, notably in the case 
where: {{// Must return all delete-by-query commands that occur after the first 
add requested}}.  How is that possibility handled in the code paths that use 
fetchMissingUpdateFromLeader?
*** that seems like a scenerio that would be really easy to test for -- similar 
to how outOfOrderDeleteUpdatesIndividualReplicaTest works
** {{assert ((List) missingUpdates).size() == 1: "More than 1 update ...}}
*** based on my skimming of the code, an empty list is just as possible, so the 
assertion is missleading (ideally it should say how many updates it got, or 
maybe toString() the whole List ?)

{panel}


{panel:title=AtomicUpdateDocumentMerger}

* isSupportedFieldForInPlaceUpdate
** javadocs

* getFieldNamesFromIndex
** javadocs
** method name seems VERY missleading considering what it does 
{color:green}Changed it to getSearcherNonStoredDVs{color}

* isInPlaceUpdate
** javadocs should be clear what hapens to inPlaceUpdatedFields if result is 
false (even if answer is "undefined"
** based on usage, wouldn't it be simplier if instead of returning a boolean, 
this method just returned a (new) Set of inplace update fields found, and if 
the set is empty that means it's not an in place update? 
{color:green}FIXED{color}
** isn't getFieldNamesFromIndex kind of an expensive method to call on every 
AddUpdateCommand ?
*** couldn't this list of fields be created by the caller and re-used at least 
for the entire request (ie: when adding multiple docs) ? {color:green}The set 
returned is precomputed upon the opening of a searcher. The only cost I see is 
to create a new unmodifiableSet every time. I'd prefer to take up this 
optimization later, if needed.{color}
** {{if (indexFields.contains(fieldName) == false && 
schema.isDynamicField(fieldName))}}
*** why does it matter one way or the other if it's a dynamicField? 
{color:green}Changed the logic to check in the IW for presence of field. Added 
a comment: "// if dynamic field and this field doesn't exist, DV update can't 
work"{color}
** the special {{DELETED}} sentinal value still isn't being checked against the 
return value of {{getInputDocumentFromTlog}} {color:green}Not using 
getInputDocumentFromTlog call anymore{color}
** this method still seems like it could/should do "cheaper" validation (ie: 
not requiring SchemaField object creation, or tlog lookups) first.  (Ex: the 
set of supported atomic ops are checked after isSupportedFieldForInPlaceUpdate 
& a possible read from the tlog). {color:green}FIXED{color}
*** My suggested rewrite would be something like...{code}
Set candidateResults = new HashSet<>();
// first pass, check the things that are virtually free,
// and bail out early if anything is obviously not a valid in-place update
for (String fieldName : sdoc.getFieldNames()) {
  if (schema.getUniqueKeyField().getName().equals(fieldName)
  || fieldName.equals(DistributedUpdateProcessor.VERSION_FIELD)) {
continue;
  }
  Object fieldValue = sdoc.getField(fieldName).getValue();
  if (! (fieldValue instanceof Map) ) {
// not even an atomic update, definitely not an 

[jira] [Created] (SOLR-9759) Admin UI should post streaming expressions

[Gus Heck (JIRA)]

Gus Heck created SOLR-9759:
--

 Summary: Admin UI should post streaming expressions
 Key: SOLR-9759
 URL: https://issues.apache.org/jira/browse/SOLR-9759
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: UI
Affects Versions: 6.2.1
Reporter: Gus Heck


Haven't had the chance to test this in 6.3, but in 6.2.1 I just ran into 
request entity too large, when I pasted an expression into the admin ui to 
begin debugging it... 

Furthermore, the UI gives no indication of any error at all, leading one to 
sit, waiting for the response. Firefox JavaScript console shows a 413 response 
and this:

11:01:11.095 Error: JSON.parse: unexpected character at line 1 column 1 of the 
JSON data
$scope.doStream/<@http://localhost:8984/solr/js/angular/controllers/stream.js:48:24
v/http://localhost:8984/solr/libs/angular-resource.min.js:33:133
processQueue@http://localhost:8984/solr/libs/angular.js:13193:27
scheduleProcessQueue/<@http://localhost:8984/solr/libs/angular.js:13209:27
$RootScopeProvider/this.$gethttp://localhost:8984/solr/libs/angular.js:14406:16
$RootScopeProvider/this.$gethttp://localhost:8984/solr/libs/angular.js:14222:15
$RootScopeProvider/this.$gethttp://localhost:8984/solr/libs/angular.js:14511:13
done@http://localhost:8984/solr/libs/angular.js:9669:36
completeRequest@http://localhost:8984/solr/libs/angular.js:9859:7
requestLoaded@http://localhost:8984/solr/libs/angular.js:9800:9
1angular.js:11617:18
consoleLog/<()angular.js:11617
$ExceptionHandlerProvider/this.$get

[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_102) - Build # 2179 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/2179/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; 
[core_node2:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:42530/rbw/m","node_name":"127.0.0.1:42530_rbw%2Fm","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/30)={   
"replicationFactor":"3",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node1":{   "state":"down",   
"base_url":"http://127.0.0.1:36172/rbw/m;,   
"core":"c8n_1x3_lf_shard1_replica1",   
"node_name":"127.0.0.1:36172_rbw%2Fm"}, "core_node2":{   
"core":"c8n_1x3_lf_shard1_replica2",   
"base_url":"http://127.0.0.1:42530/rbw/m;,   
"node_name":"127.0.0.1:42530_rbw%2Fm",   "state":"active",   
"leader":"true"}, "core_node3":{   
"core":"c8n_1x3_lf_shard1_replica3",   
"base_url":"http://127.0.0.1:46430/rbw/m;,   
"node_name":"127.0.0.1:46430_rbw%2Fm",   "state":"down",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 
1; 
[core_node2:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:42530/rbw/m","node_name":"127.0.0.1:42530_rbw%2Fm","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/30)={
  "replicationFactor":"3",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node1":{
  "state":"down",
  "base_url":"http://127.0.0.1:36172/rbw/m;,
  "core":"c8n_1x3_lf_shard1_replica1",
  "node_name":"127.0.0.1:36172_rbw%2Fm"},
"core_node2":{
  "core":"c8n_1x3_lf_shard1_replica2",
  "base_url":"http://127.0.0.1:42530/rbw/m;,
  "node_name":"127.0.0.1:42530_rbw%2Fm",
  "state":"active",
  "leader":"true"},
"core_node3":{
  "core":"c8n_1x3_lf_shard1_replica3",
  "base_url":"http://127.0.0.1:46430/rbw/m;,
  "node_name":"127.0.0.1:46430_rbw%2Fm",
  "state":"down",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
at 
__randomizedtesting.SeedInfo.seed([2C881B8257E1D26F:A4DC2458F91DBF97]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:168)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 

Re: VOTE: Apache Solr Ref Guide for Solr 6.3, RC3

[Cassandra Targett]

I think we can call this vote passed. I'll start the release process.

On Sat, Nov 12, 2016 at 8:42 PM, Tomás Fernández Löbbe
 wrote:
> LGTM. +1
>
> On Thu, Nov 10, 2016 at 4:41 PM, Steve Rowe  wrote:
>>
>> +1
>>
>> --
>> Steve
>> www.lucidworks.com
>>
>> > On Nov 9, 2016, at 1:22 PM, Cassandra Targett 
>> > wrote:
>> >
>> > Please VOTE for the 3rd release candidate of the Apache Solr Reference
>> > Guide for 6.3.
>> >
>> > I paged through all 738 pages looking for text that runs off the edge
>> > of the page, and didn't see any. I usually only spot check pages I
>> > remember were edited, but this time went through the whole thing.
>> >
>> > Artifacts are available from:
>> >
>> > https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-6.3-RC3/
>> >
>> > $ more apache-solr-ref-guide-6.3-RC3/apache-solr-ref-guide-6.3.pdf.sha1
>> > 103b831a8f8a1aa2cff2fa20b5c0025b2f075ae0  apache-solr-ref-guide-6.3.pdf
>> >
>> > Here's my +1.
>> >
>> > Thanks,
>> > Cassandra
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Deleted] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-5944:
---
Comment: was deleted

(was: Some minor changes:
1. Split the UpdateLogTest to smaller tests, more readable.
2. Added a javadoc comment to DUP.
3. Combined the testReplay1, 2, 3 into a single test.)

> Support updates of numeric DocValues
> 
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
>  Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, 
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, 
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, 
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, 
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be 
> really nice to have Solr support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch

Some minor changes:
1. Split the UpdateLogTest to smaller tests, more readable.
2. Added a javadoc comment to DUP.
3. Combined the testReplay1, 2, 3 into a single test.

> Support updates of numeric DocValues
> 
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
>  Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, 
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, 
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, 
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, 
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be 
> really nice to have Solr support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch

Some minor changes:
1. Split the UpdateLogTest to smaller tests, more readable.
2. Added a javadoc comment to DUP.
3. Combined the testReplay1, 2, 3 into a single test.

> Support updates of numeric DocValues
> 
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
>  Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, 
> SOLR-5944.patch, 
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, 
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, 
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, 
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, 
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be 
> really nice to have Solr support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9166) Export handler returns zero for numeric fields that are not in the original doc

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664186#comment-15664186
 ] 

ASF subversion and git services commented on SOLR-9166:
---

Commit 375dd323a729035e2c957ecb49a6b88135e3edfd in lucene-solr's branch 
refs/heads/branch_6x from [~erickerickson]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=375dd32 ]

SOLR-9166: Export handler returns zero for numeric fields that are not in the 
original doc


> Export handler returns zero for numeric fields that are not in the original 
> doc
> ---
>
> Key: SOLR-9166
> URL: https://issues.apache.org/jira/browse/SOLR-9166
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Erick Erickson
> Attachments: SOLR-9166-6x.patch, SOLR-9166.patch, SOLR-9166.patch, 
> SOLR-9166.patch, SOLR-9166.patch
>
>
> From the dev list discussion:
> My original post.
> Zero is different from not
> existing. And let's claim that I want to process a stream and, say,
> facet on in integer field over the result set. There's no way on the
> client side to distinguish between a document that has a zero in the
> field and one that didn't have the field in the first place so I'll
> over-count the zero bucket.
> From Dennis Gove:
> Is this true for non-numeric fields as well? I agree that this seems like a 
> very bad thing.
> I can't imagine that a fix would cause a problem with Streaming Expressions, 
> ParallelSQL, or other given that the /select handler is not returning 0 for 
> these missing fields (the /select handler is the default handler for the 
> Streaming API so if nulls were a problem I imagine we'd have already seen 
> it). 
> That said, within Streaming Expressions there is a select(...) function which 
> supports a replace(...) operation which allows you to replace one value (or 
> null) with some other value. If a 0 were necessary one could use a 
> select(...) to replace null with 0 using an expression like this 
>select(, replace(fieldA, null, withValue=0)). 
> The end result of that would be that the field fieldA would never have a null 
> value and for all tuples where a null value existed it would be replaced with 
> 0.
> Details on the select function can be found at 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61330338#StreamingExpressions-select.
> And to answer Denis' question, null gets returned for string DocValues fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9166) Export handler returns zero for numeric fields that are not in the original doc

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664175#comment-15664175
 ] 

ASF subversion and git services commented on SOLR-9166:
---

Commit 4a31b29cb031a10d25de01e25d1d9e5b1a4a7787 in lucene-solr's branch 
refs/heads/master from [~erickerickson]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4a31b29 ]

SOLR-9166: Export handler returns zero for numeric fields that are not in the 
original doc


> Export handler returns zero for numeric fields that are not in the original 
> doc
> ---
>
> Key: SOLR-9166
> URL: https://issues.apache.org/jira/browse/SOLR-9166
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Erick Erickson
> Attachments: SOLR-9166-6x.patch, SOLR-9166.patch, SOLR-9166.patch, 
> SOLR-9166.patch, SOLR-9166.patch
>
>
> From the dev list discussion:
> My original post.
> Zero is different from not
> existing. And let's claim that I want to process a stream and, say,
> facet on in integer field over the result set. There's no way on the
> client side to distinguish between a document that has a zero in the
> field and one that didn't have the field in the first place so I'll
> over-count the zero bucket.
> From Dennis Gove:
> Is this true for non-numeric fields as well? I agree that this seems like a 
> very bad thing.
> I can't imagine that a fix would cause a problem with Streaming Expressions, 
> ParallelSQL, or other given that the /select handler is not returning 0 for 
> these missing fields (the /select handler is the default handler for the 
> Streaming API so if nulls were a problem I imagine we'd have already seen 
> it). 
> That said, within Streaming Expressions there is a select(...) function which 
> supports a replace(...) operation which allows you to replace one value (or 
> null) with some other value. If a 0 were necessary one could use a 
> select(...) to replace null with 0 using an expression like this 
>select(, replace(fieldA, null, withValue=0)). 
> The end result of that would be that the field fieldA would never have a null 
> value and for all tuples where a null value existed it would be replaced with 
> 0.
> Details on the select function can be found at 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61330338#StreamingExpressions-select.
> And to answer Denis' question, null gets returned for string DocValues fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul updated SOLR-9512:
-
Attachment: SOLR-9512.patch

no tests yet

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
> Attachments: SOLR-9512.patch, SOLR-9512.patch
>
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_102) - Build # 18281 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18281/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
at 
__randomizedtesting.SeedInfo.seed([AF6F634B63B039FA:581C8D13A558961C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1329)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11762 lines...]
   [junit4] Suite: 

[jira] [Updated] (SOLR-9758) refactor preferLocalShards implementation

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated SOLR-9758:
--
Attachment: SOLR-9758.patch

> refactor preferLocalShards implementation
> -
>
> Key: SOLR-9758
> URL: https://issues.apache.org/jira/browse/SOLR-9758
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-9758.patch
>
>
> This ticket proposes to refactor the existing preferLocalShards 
> implementation (from SOLR-6832 and SOLR-8298) based upon the recent (in 
> SOLR-8332) ReplicaListTransformer addition.
> The intention of the refactor is to encapsulate the local shard url selection 
> logic within the HttpShardHandlerFactory.getReplicaListTransformer method (it 
> is currently spread across the public HttpShardHandler.prepDistributed and 
> the private HttpShardHandler.getURLs method) and to thus remove the 
> ResponseBuilder.preferredHostAddress field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org