[jira] [Commented] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625774#comment-16625774 ] Aaron LaBella commented on SOLR-12789: -- Hi Alexandre, thank for you the additional detail and background. While I understand the goal here, I don't agree with how the end result was achieved. I think the real "issue" here was that the examples and documentation are stale. Likewise, UIMA core can (and should) be upgraded to the latest 2.10.2, and the additional unnecessary dependencies should absolutely be removed from the dist. I'm attaching a simple patch (*SOLR-12789-4.patch*) that does just this. I would like to propose that we re-instate the contrib/uima project and apply my patch instead. I think this is a fair compromise since 6 Java classes doesn't quite compromise as "dead weight", especially if those 6 classes provide direct end-user value. While I would certain agree, UIMA has a steep learning curve, there are folks out there that are using it, and removing it entirely from the Solr dist is likely to do a disservice to those folks who are in-fact doing text analytics using it. All that being said, I think the only thing that really remains is a clean-up of the documentation and examples. I'm happy to do that over the next couple weeks if we agree to this strategy. Thanks so much. > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Major > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch, SOLR-12789-4.patch > > > I've been sitting on this patch for over 2 years (and likewise it's been > running IN production for the same) ... finally got around to contributing it > back to the community. This change prepares the UIMAUpdateRequestProcessor > to allow subclasses to have additional control over how the analysis engine > is selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. I don't necessarily need > it to be back-ported, having in 7.4.1 would suffice. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12789: - Attachment: SOLR-12789-4.patch > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Major > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch, SOLR-12789-4.patch > > > I've been sitting on this patch for over 2 years (and likewise it's been > running IN production for the same) ... finally got around to contributing it > back to the community. This change prepares the UIMAUpdateRequestProcessor > to allow subclasses to have additional control over how the analysis engine > is selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. I don't necessarily need > it to be back-ported, having in 7.4.1 would suffice. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11694) Remove extremely outdated UIMA contrib module
[ https://issues.apache.org/jira/browse/SOLR-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623739#comment-16623739 ] Aaron LaBella commented on SOLR-11694: -- I'm confused... since when does a feature require the "committers" of a project to actually use it. I thought the point of open source and a product / features overall was to provide value to the actual "users" of it. And for what it's worth, I've personally provided a few patches to te Solr UIMA project over the past couple years. I think there is a misconception to it "rotting" away. There was nothing to upgrade or maintain because the feature works exactly as it was designed and written to work. Again, I've never heard of a feature being completely removed like this when you have real users out there that are dependent upon it. > Remove extremely outdated UIMA contrib module > - > > Key: SOLR-11694 > URL: https://issues.apache.org/jira/browse/SOLR-11694 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Reporter: Cassandra Targett >Assignee: Alexandre Rafalovitch >Priority: Major > Fix For: 7.5, master (8.0) > > Attachments: SOLR-11694.patch > > > A user on the [solr-user mailing list back in > June|https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201706.mbox/%3CCANsk%2BC_PvZJ38AQ2VfzKRYSQn6c8b33kGvaXxR3qNS3GQ4VUKA%40mail.gmail.com%3E] > brought up the fact that IBM has bought Alchemy and keys are no longer > available to use Solr's UIMA contrib. > Someone recently made a [similar > comment|https://lucene.apache.org/solr/guide/7_1/uima-integration.html#comment_7174] > to the Solr Ref Guide page and asking for a patch. > I know next to nothing about UIMA, but figured it's worth an issue to > determine what to do here. I think folks are saying it's no longer usable? Or > maybe only usable by people who already have keys (which will possibly expire > at some point)? > Anyone have an idea what needs to be done here? It seems we should have some > kind of answer, but if it's no longer usable perhaps we should retire the > contrib. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623732#comment-16623732 ] Aaron LaBella commented on SOLR-12789: -- Thanks Alexandre ... I added comments in SOLR-11694 (as you already saw). I'll need to discuss with my IBM colleagues and see what we want to do going forward. Like I said, this is a feature we've built production applications on top of and removing it is just plain "not cool". There were other ways it could've been cleaned up and maintained without removing it. As a whole, I'm really disappointed in this decision from the Solr community, as I never heard of pulling out a function that works absolutely fine. The reason that I would say its not "maintained" is that there's nothing wrong with the current feature. If anything, it could be upgraded to a later UIMAJ but even that is easily worked around. I really see this as a big step backwards for Solr with regards to text analytics. > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Major > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for over 2 years (and likewise it's been > running IN production for the same) ... finally got around to contributing it > back to the community. This change prepares the UIMAUpdateRequestProcessor > to allow subclasses to have additional control over how the analysis engine > is selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. I don't necessarily need > it to be back-ported, having in 7.4.1 would suffice. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11694) Remove extremely outdated UIMA contrib module
[ https://issues.apache.org/jira/browse/SOLR-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622839#comment-16622839 ] Aaron LaBella commented on SOLR-11694: -- Apace UIMA 3.0 is in alpha state, and it'll probably be a long time before a.) its an actual 3.0 release and b.) anyone out there actually adopts it. 2.10.2 is the latest and its from February of 2018. I'm really not sure why we're pulling a feature out of Solr that has to be used out there by several people. What harm does it have to keep it there? We're saving 7MB of space?? I don't actively follow the Solr mailing list and I'm not sure just because no one responded in ~6 months that you can safely assume that the entire community doesn't use UIMA. I feel strongly that this is not the right decision here. If you want to clean up, I'd agree, remove the AlchemyAPI, OpenCalais, and Tagger. Upgrade uima-core 2.3.1 to 2.10.2 and keep the rest of the Solr/UIMA classes – they work just fine. There's no reason to pull this contrib project. > Remove extremely outdated UIMA contrib module > - > > Key: SOLR-11694 > URL: https://issues.apache.org/jira/browse/SOLR-11694 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Reporter: Cassandra Targett >Assignee: Alexandre Rafalovitch >Priority: Major > Fix For: 7.5, master (8.0) > > Attachments: SOLR-11694.patch > > > A user on the [solr-user mailing list back in > June|https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201706.mbox/%3CCANsk%2BC_PvZJ38AQ2VfzKRYSQn6c8b33kGvaXxR3qNS3GQ4VUKA%40mail.gmail.com%3E] > brought up the fact that IBM has bought Alchemy and keys are no longer > available to use Solr's UIMA contrib. > Someone recently made a [similar > comment|https://lucene.apache.org/solr/guide/7_1/uima-integration.html#comment_7174] > to the Solr Ref Guide page and asking for a patch. > I know next to nothing about UIMA, but figured it's worth an issue to > determine what to do here. I think folks are saying it's no longer usable? Or > maybe only usable by people who already have keys (which will possibly expire > at some point)? > Anyone have an idea what needs to be done here? It seems we should have some > kind of answer, but if it's no longer usable perhaps we should retire the > contrib. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-11694) Remove extremely outdated UIMA contrib module
[ https://issues.apache.org/jira/browse/SOLR-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622539#comment-16622539 ] Aaron LaBella edited comment on SOLR-11694 at 9/20/18 7:08 PM: --- Hi Team, _*How do we undo this commit/decision?*_ We use UIMA and SOLR +*every day*+ in production, and this decision will effectively freeze us on the last working Solr / UIMA version of 7.4.0. Surely there are other folks out there doing text analytics with even older versions of UIMA and wish to use the latest and greatest solr? Aso, even though SOLR/LUCENE "ship" with UIMA 2.3.1, I've been using UIMA 2.8.1 for 2+ years by simply dropping in the more up-to-date JAR in the core-specific "lib" directory. I'd have to look through all the lucene stuff again, but, I'm quite certain the lucene support for UIMA is still active as well, no? was (Author: aaronlab): Hi Team, How do we undo this commit/decision? We use UIMA and SOLR +*every day*+ in production, and this decision will effectively freeze us on the last working Solr / UIMA version of 7.4.0. Surely there are other folks out there doing text analytics with even older versions of UIMA and wish to use the latest and greatest solr? ALso, even though SOLR/LUCENE "ship" with UIMA 2.3.1, I've been using UIMA 2.8.1 for 2+ years by simply dropping in the more up-to-date JAR in the core-specific "lib" directory. I'd have to look through all the lucene stuff again, but, I'm quite certain the lucene support for UIMA is still active as well, no? > Remove extremely outdated UIMA contrib module > - > > Key: SOLR-11694 > URL: https://issues.apache.org/jira/browse/SOLR-11694 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Reporter: Cassandra Targett >Assignee: Alexandre Rafalovitch >Priority: Major > Fix For: 7.5, master (8.0) > > Attachments: SOLR-11694.patch > > > A user on the [solr-user mailing list back in > June|https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201706.mbox/%3CCANsk%2BC_PvZJ38AQ2VfzKRYSQn6c8b33kGvaXxR3qNS3GQ4VUKA%40mail.gmail.com%3E] > brought up the fact that IBM has bought Alchemy and keys are no longer > available to use Solr's UIMA contrib. > Someone recently made a [similar > comment|https://lucene.apache.org/solr/guide/7_1/uima-integration.html#comment_7174] > to the Solr Ref Guide page and asking for a patch. > I know next to nothing about UIMA, but figured it's worth an issue to > determine what to do here. I think folks are saying it's no longer usable? Or > maybe only usable by people who already have keys (which will possibly expire > at some point)? > Anyone have an idea what needs to be done here? It seems we should have some > kind of answer, but if it's no longer usable perhaps we should retire the > contrib. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-11694) Remove extremely outdated UIMA contrib module
[ https://issues.apache.org/jira/browse/SOLR-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622539#comment-16622539 ] Aaron LaBella edited comment on SOLR-11694 at 9/20/18 7:08 PM: --- Hi Team, How do we undo this commit/decision? We use UIMA and SOLR +*every day*+ in production, and this decision will effectively freeze us on the last working Solr / UIMA version of 7.4.0. Surely there are other folks out there doing text analytics with even older versions of UIMA and wish to use the latest and greatest solr? ALso, even though SOLR/LUCENE "ship" with UIMA 2.3.1, I've been using UIMA 2.8.1 for 2+ years by simply dropping in the more up-to-date JAR in the core-specific "lib" directory. I'd have to look through all the lucene stuff again, but, I'm quite certain the lucene support for UIMA is still active as well, no? was (Author: aaronlab): Hi Team, How do we undo this commit/decision? We use UIMA and SOLR +*every day*+ in production, and this decision will effectively freeze us on the last working Solr / UIMA version of 7.4.0. Surely there are other folks out there doing text analytics with even older versions of UIMA and wish to use the latest and greatest solr? > Remove extremely outdated UIMA contrib module > - > > Key: SOLR-11694 > URL: https://issues.apache.org/jira/browse/SOLR-11694 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Reporter: Cassandra Targett >Assignee: Alexandre Rafalovitch >Priority: Major > Fix For: 7.5, master (8.0) > > Attachments: SOLR-11694.patch > > > A user on the [solr-user mailing list back in > June|https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201706.mbox/%3CCANsk%2BC_PvZJ38AQ2VfzKRYSQn6c8b33kGvaXxR3qNS3GQ4VUKA%40mail.gmail.com%3E] > brought up the fact that IBM has bought Alchemy and keys are no longer > available to use Solr's UIMA contrib. > Someone recently made a [similar > comment|https://lucene.apache.org/solr/guide/7_1/uima-integration.html#comment_7174] > to the Solr Ref Guide page and asking for a patch. > I know next to nothing about UIMA, but figured it's worth an issue to > determine what to do here. I think folks are saying it's no longer usable? Or > maybe only usable by people who already have keys (which will possibly expire > at some point)? > Anyone have an idea what needs to be done here? It seems we should have some > kind of answer, but if it's no longer usable perhaps we should retire the > contrib. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11694) Remove extremely outdated UIMA contrib module
[ https://issues.apache.org/jira/browse/SOLR-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622539#comment-16622539 ] Aaron LaBella commented on SOLR-11694: -- Hi Team, How do we undo this commit/decision? We use UIMA and SOLR +*every day*+ in production, and this decision will effectively freeze us on the last working Solr / UIMA version of 7.4.0. Surely there are other folks out there doing text analytics with even older versions of UIMA and wish to use the latest and greatest solr? > Remove extremely outdated UIMA contrib module > - > > Key: SOLR-11694 > URL: https://issues.apache.org/jira/browse/SOLR-11694 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Reporter: Cassandra Targett >Assignee: Alexandre Rafalovitch >Priority: Major > Fix For: 7.5, master (8.0) > > Attachments: SOLR-11694.patch > > > A user on the [solr-user mailing list back in > June|https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201706.mbox/%3CCANsk%2BC_PvZJ38AQ2VfzKRYSQn6c8b33kGvaXxR3qNS3GQ4VUKA%40mail.gmail.com%3E] > brought up the fact that IBM has bought Alchemy and keys are no longer > available to use Solr's UIMA contrib. > Someone recently made a [similar > comment|https://lucene.apache.org/solr/guide/7_1/uima-integration.html#comment_7174] > to the Solr Ref Guide page and asking for a patch. > I know next to nothing about UIMA, but figured it's worth an issue to > determine what to do here. I think folks are saying it's no longer usable? Or > maybe only usable by people who already have keys (which will possibly expire > at some point)? > Anyone have an idea what needs to be done here? It seems we should have some > kind of answer, but if it's no longer usable perhaps we should retire the > contrib. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12789: - Priority: Major (was: Blocker) > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Major > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for over 2 years (and likewise it's been > running IN production for the same) ... finally got around to contributing it > back to the community. This change prepares the UIMAUpdateRequestProcessor > to allow subclasses to have additional control over how the analysis engine > is selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. I don't necessarily need > it to be back-ported, having in 7.4.1 would suffice. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12789: - Description: I've been sitting on this patch for over 2 years (and likewise it's been running IN production for the same) ... finally got around to contributing it back to the community. This change prepares the UIMAUpdateRequestProcessor to allow subclasses to have additional control over how the analysis engine is selected. In my case, I wrote a sub-class that allows for *dynamic* detection of the UIMA analysis engine based on the document fields. ie: a field in the document can be used to select different UIMA configurations and rules. Can someone please commit this as soon as possible. I don't necessarily need it to be back-ported, having in 7.4.1 would suffice. Thanks! was: I've been sitting on this patch for over 2 years (and likewise it's been running IN production for the same) ... finally got around to contributing it back to the community. This change prepares the UIMAUpdateRequestProcessor to allow subclasses to have additional control over how the analysis engine is selected. In my case, I wrote a sub-class that allows for *dynamic* detection of the UIMA analysis engine based on the document fields. ie: a field in the document can be used to select different UIMA configurations and rules. Can someone please commit this as soon as possible. Thanks! > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for over 2 years (and likewise it's been > running IN production for the same) ... finally got around to contributing it > back to the community. This change prepares the UIMAUpdateRequestProcessor > to allow subclasses to have additional control over how the analysis engine > is selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. I don't necessarily need > it to be back-ported, having in 7.4.1 would suffice. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12790) move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses override. expose other fields as protected
[ https://issues.apache.org/jira/browse/SOLR-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12790: - Description: moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to allow subclasses to override. expose other fields as protected. This allows sub-classes to apply different logic other than round-robin, when using the LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* uses the first / primary server unless it is unavailable, and then it uses the secondary server(s). Can someone please commit this as soon as possible. I don't necessarily need it to be back-ported, having in 7.4.1 would suffice. Thanks! was:moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to allow subclasses to override. expose other fields as protected. This allows sub-classes to apply different logic other than round-robin, when using the LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* uses the first / primary server unless it is unavailable, and then it uses the secondary server(s). > move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses > override. expose other fields as protected > > > Key: SOLR-12790 > URL: https://issues.apache.org/jira/browse/SOLR-12790 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Affects Versions: 5.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12790-1.patch > > > moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to > allow subclasses to override. expose other fields as protected. This allows > sub-classes to apply different logic other than round-robin, when using the > LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* > uses the first / primary server unless it is unavailable, and then it uses > the secondary server(s). > > Can someone please commit this as soon as possible. I don't necessarily need > it to be back-ported, having in 7.4.1 would suffice. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12790) move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses override. expose other fields as protected
[ https://issues.apache.org/jira/browse/SOLR-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12790: - Description: moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to allow subclasses to override. expose other fields as protected. This allows sub-classes to apply different logic other than round-robin, when using the LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* uses the first / primary server unless it is unavailable, and then it uses the secondary server(s). (was: moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to allow subclasses override. expose other fields as protected. This allows sub-classes to apply different logic other than round-robin, when using the LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* uses the first / primary server unless it is unavailable, and then it uses the secondary server(s).) > move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses > override. expose other fields as protected > > > Key: SOLR-12790 > URL: https://issues.apache.org/jira/browse/SOLR-12790 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Affects Versions: 5.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12790-1.patch > > > moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to > allow subclasses to override. expose other fields as protected. This allows > sub-classes to apply different logic other than round-robin, when using the > LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* > uses the first / primary server unless it is unavailable, and then it uses > the secondary server(s). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12790) move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses override. expose other fields as protected
[ https://issues.apache.org/jira/browse/SOLR-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12790: - Attachment: SOLR-12790-1.patch > move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses > override. expose other fields as protected > > > Key: SOLR-12790 > URL: https://issues.apache.org/jira/browse/SOLR-12790 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Affects Versions: 5.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12790-1.patch > > > moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to > allow subclasses override. expose other fields as protected. This allows > sub-classes to apply different logic other than round-robin, when using the > LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* > uses the first / primary server unless it is unavailable, and then it uses > the secondary server(s). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12790) move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses override. expose other fields as protected
Aaron LaBella created SOLR-12790: Summary: move ServerWrapper logic in LBHttpSolrClient to a method to allow subclasses override. expose other fields as protected Key: SOLR-12790 URL: https://issues.apache.org/jira/browse/SOLR-12790 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: SolrJ Affects Versions: 5.0 Reporter: Aaron LaBella Attachments: SOLR-12790-1.patch moved ServerWrapper logic in LBHttpSolrClient.request(...) to a method to allow subclasses override. expose other fields as protected. This allows sub-classes to apply different logic other than round-robin, when using the LBHttpSolrClient class. In my case, we wish to use an algorithm that *only* uses the first / primary server unless it is unavailable, and then it uses the secondary server(s). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12789: - Description: I've been sitting on this patch for over 2 years (and likewise it's been running IN production for the same) ... finally got around to contributing it back to the community. This change prepares the UIMAUpdateRequestProcessor to allow subclasses to have additional control over how the analysis engine is selected. In my case, I wrote a sub-class that allows for *dynamic* detection of the UIMA analysis engine based on the document fields. ie: a field in the document can be used to select different UIMA configurations and rules. Can someone please commit this as soon as possible. Thanks! was: I've been sitting on this patch for 2 years (and likewise it's been running production for the same) ... finally got around to contributing it back to the community. This change prepares the UIMAUpdateRequestProcessor to allow subclasses to have additional control over hhow the analysis engine is selected. In my case, I wrote a sub-class that allows for *dynamic* detection of the UIMA analysis engine based on the document fields. ie: a field in the document can be used to select different UIMA configurations and rules. Can someone please commit this as soon as possible. Thanks! > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for over 2 years (and likewise it's been > running IN production for the same) ... finally got around to contributing it > back to the community. This change prepares the UIMAUpdateRequestProcessor > to allow subclasses to have additional control over how the analysis engine > is selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622309#comment-16622309 ] Aaron LaBella commented on SOLR-12789: -- There is also a patch in here to add proper support for UIMA feature values which are arrays to map to multivalued fields > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for 2 years (and likewise it's been running > production for the same) ... finally got around to contributing it back to > the community. This change prepares the UIMAUpdateRequestProcessor to allow > subclasses to have additional control over hhow the analysis engine is > selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12789: - Attachment: SOLR-12789-3.patch > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for 2 years (and likewise it's been running > production for the same) ... finally got around to contributing it back to > the community. This change prepares the UIMAUpdateRequestProcessor to allow > subclasses to have additional control over hhow the analysis engine is > selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12789: - Attachment: SOLR-12789-1.patch > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for 2 years (and likewise it's been running > production for the same) ... finally got around to contributing it back to > the community. This change prepares the UIMAUpdateRequestProcessor to allow > subclasses to have additional control over hhow the analysis engine is > selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
[ https://issues.apache.org/jira/browse/SOLR-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-12789: - Attachment: SOLR-12789-2.patch > UIMA enhancements to allow for dynamic AE detection > --- > > Key: SOLR-12789 > URL: https://issues.apache.org/jira/browse/SOLR-12789 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - UIMA >Affects Versions: 6.0 >Reporter: Aaron LaBella >Priority: Blocker > Labels: ready-to-commit > Attachments: SOLR-12789-1.patch, SOLR-12789-2.patch, > SOLR-12789-3.patch > > > I've been sitting on this patch for 2 years (and likewise it's been running > production for the same) ... finally got around to contributing it back to > the community. This change prepares the UIMAUpdateRequestProcessor to allow > subclasses to have additional control over hhow the analysis engine is > selected. In my case, I wrote a sub-class that allows for *dynamic* > detection of the UIMA analysis engine based on the document fields. ie: a > field in the document can be used to select different UIMA configurations and > rules. > > Can someone please commit this as soon as possible. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12789) UIMA enhancements to allow for dynamic AE detection
Aaron LaBella created SOLR-12789: Summary: UIMA enhancements to allow for dynamic AE detection Key: SOLR-12789 URL: https://issues.apache.org/jira/browse/SOLR-12789 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: contrib - UIMA Affects Versions: 6.0 Reporter: Aaron LaBella I've been sitting on this patch for 2 years (and likewise it's been running production for the same) ... finally got around to contributing it back to the community. This change prepares the UIMAUpdateRequestProcessor to allow subclasses to have additional control over hhow the analysis engine is selected. In my case, I wrote a sub-class that allows for *dynamic* detection of the UIMA analysis engine based on the document fields. ie: a field in the document can be used to select different UIMA configurations and rules. Can someone please commit this as soon as possible. Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8548) CorePropertiesLocator does NOT follow symlinks anymore
[ https://issues.apache.org/jira/browse/SOLR-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101819#comment-15101819 ] Aaron LaBella commented on SOLR-8548: - Alan, no particular reason other than I think it's silly to have the code check an unlimited number of directories/depth for the core.properties file. I think 2 or 3 at most would be a reasonable limit, not sure why anyone would want to bury their solr cores anymore beyond that. Feel free to change that particular setting to whatever you feel is right. Thanks. > CorePropertiesLocator does NOT follow symlinks anymore > -- > > Key: SOLR-8548 > URL: https://issues.apache.org/jira/browse/SOLR-8548 > Project: Solr > Issue Type: Bug >Affects Versions: 5.4 >Reporter: Aaron LaBella >Assignee: Alan Woodward >Priority: Blocker > Attachments: SOLR-8548.patch > > > Looks like CorePropertiesLocator was switched over to use the new java.nio > stuff... and the simplified version of the walkFileTree method does NOT > follow symlinks by default. This is a critical feature that needs to be > supported for anyone who has their solr core(s) defined elsewhere, and set up > their solr home directory to point to those cores via symlinks. It's > important to realize that this is the behavior that has always existed > pre-5.4.0 by the nature of using the java.io package to listFiles. > I'm attaching a PATCH with the appropriate fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8548) CorePropertiesLocator does NOT follow symlinks anymore
[ https://issues.apache.org/jira/browse/SOLR-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-8548: Description: Looks like CorePropertiesLocator was switched over to use the new java.nio stuff... and the simplified version of the walkFileTree method does NOT follow symlinks by default. This is a critical feature that needs to be supported for anyone who has their solr core(s) defined elsewhere, and set up their solr home directory to point to those cores via symlinks. It's important to realize that this is the behavior that has always existed pre-5.4.0 by the nature of using the java.io package to listFiles. I'm attaching a PATCH with the appropriate fix. was: Looks like CorePropertiesLocator was switched over to use the new java.nio stuff... and the simplified version of the walkFileTree method does NOT follow symlinks by default. This is a critical feature that needs to be supported for anyone who has their solr core defined elsewhere, and set up their solr home directory to point to those cores via symlinks. It's important to realize that this is the behavior that as always existed pre-5.4.0 by the nature of using the java.io package to listFiles. I'm attaching a PATCH with the appropriate fix. > CorePropertiesLocator does NOT follow symlinks anymore > -- > > Key: SOLR-8548 > URL: https://issues.apache.org/jira/browse/SOLR-8548 > Project: Solr > Issue Type: Bug >Affects Versions: 5.4 >Reporter: Aaron LaBella >Priority: Blocker > Attachments: SOLR-8548.patch > > > Looks like CorePropertiesLocator was switched over to use the new java.nio > stuff... and the simplified version of the walkFileTree method does NOT > follow symlinks by default. This is a critical feature that needs to be > supported for anyone who has their solr core(s) defined elsewhere, and set up > their solr home directory to point to those cores via symlinks. It's > important to realize that this is the behavior that has always existed > pre-5.4.0 by the nature of using the java.io package to listFiles. > I'm attaching a PATCH with the appropriate fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8548) CorePropertiesLocator does NOT follow symlinks anymore
[ https://issues.apache.org/jira/browse/SOLR-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-8548: Attachment: SOLR-8548.patch patch to fix issue, please accept asap. > CorePropertiesLocator does NOT follow symlinks anymore > -- > > Key: SOLR-8548 > URL: https://issues.apache.org/jira/browse/SOLR-8548 > Project: Solr > Issue Type: Bug >Affects Versions: 5.4 >Reporter: Aaron LaBella >Priority: Blocker > Attachments: SOLR-8548.patch > > > Looks like CorePropertiesLocator was switched over to use the new java.nio > stuff... and the simplified version of the walkFileTree method does NOT > follow symlinks by default. This is a critical feature that needs to be > supported for anyone who has their solr core defined elsewhere, and set up > their solr home directory to point to those cores via symlinks. It's > important to realize that this is the behavior that as always existed > pre-5.4.0 by the nature of using the java.io package to listFiles. > I'm attaching a PATCH with the appropriate fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-8548) CorePropertiesLocator does NOT follow symlinks anymore
Aaron LaBella created SOLR-8548: --- Summary: CorePropertiesLocator does NOT follow symlinks anymore Key: SOLR-8548 URL: https://issues.apache.org/jira/browse/SOLR-8548 Project: Solr Issue Type: Bug Affects Versions: 5.4 Reporter: Aaron LaBella Priority: Blocker Looks like CorePropertiesLocator was switched over to use the new java.nio stuff... and the simplified version of the walkFileTree method does NOT follow symlinks by default. This is a critical feature that needs to be supported for anyone who has their solr core defined elsewhere, and set up their solr home directory to point to those cores via symlinks. It's important to realize that this is the behavior that as always existed pre-5.4.0 by the nature of using the java.io package to listFiles. I'm attaching a PATCH with the appropriate fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7885) Add support for loading HTTP resources
[
https://issues.apache.org/jira/browse/SOLR-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060905#comment-15060905
]
Aaron LaBella commented on SOLR-7885:
-
BTW, trying to call command=reload-config=... on the DIH doesn't
work -- since, if anything, it just updates the in memory config -- it doesn't
persist the config to the local file system, which was half the point of the
patch.
> Add support for loading HTTP resources
> --
>
> Key: SOLR-7885
> URL: https://issues.apache.org/jira/browse/SOLR-7885
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler, SolrJ
>Affects Versions: 5.3
>Reporter: Aaron LaBella
> Attachments: SOLR-7885-1.patch, SOLR-7885-2.patch
>
>
> I have a need to be able to load data import handler configuration files from
> an HTTP server instead of the local file system. So, I modified
> {code}org.apache.solr.core.SolrResourceLoader{code} and some of the
> respective dataimport files in {code}org.apache.solr.handler.dataimport{code}
> to be able to support doing this.
> {code}solrconfig.xml{code} now has the option to define a parameter:
> *configRemote*, and if defined (and it's an HTTP(s) URL), it'll attempt to
> load the resource. If successfully, it'll also persist the resource to the
> local file system so that it is available on a solr server restart per chance
> that the remote resource is currently unavailable.
> Lastly, to be consistent with the pattern that already exists in
> SolrResourceLoader, this feature is *disabled* by default, and requires the
> setting of an additional JVM property:
> {code}-Dsolr.allow.http.resourceloading=true{code}.
> Please review and let me know if there is anything else that needs to be done
> in order for this patch to make the next release. As far as I can tell, it's
> fully tested and ready to go.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7885) Add support for loading HTTP resources
[
https://issues.apache.org/jira/browse/SOLR-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060901#comment-15060901
]
Aaron LaBella commented on SOLR-7885:
-
I'm closing this issue as I literally just took my patch and moved it into a
custom RequestHandler instead.
Thanks.
> Add support for loading HTTP resources
> --
>
> Key: SOLR-7885
> URL: https://issues.apache.org/jira/browse/SOLR-7885
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler, SolrJ
>Affects Versions: 5.3
>Reporter: Aaron LaBella
> Attachments: SOLR-7885-1.patch, SOLR-7885-2.patch
>
>
> I have a need to be able to load data import handler configuration files from
> an HTTP server instead of the local file system. So, I modified
> {code}org.apache.solr.core.SolrResourceLoader{code} and some of the
> respective dataimport files in {code}org.apache.solr.handler.dataimport{code}
> to be able to support doing this.
> {code}solrconfig.xml{code} now has the option to define a parameter:
> *configRemote*, and if defined (and it's an HTTP(s) URL), it'll attempt to
> load the resource. If successfully, it'll also persist the resource to the
> local file system so that it is available on a solr server restart per chance
> that the remote resource is currently unavailable.
> Lastly, to be consistent with the pattern that already exists in
> SolrResourceLoader, this feature is *disabled* by default, and requires the
> setting of an additional JVM property:
> {code}-Dsolr.allow.http.resourceloading=true{code}.
> Please review and let me know if there is anything else that needs to be done
> in order for this patch to make the next release. As far as I can tell, it's
> fully tested and ready to go.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-7885) Add support for loading HTTP resources
[
https://issues.apache.org/jira/browse/SOLR-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella closed SOLR-7885.
---
Resolution: Not A Problem
> Add support for loading HTTP resources
> --
>
> Key: SOLR-7885
> URL: https://issues.apache.org/jira/browse/SOLR-7885
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler, SolrJ
>Affects Versions: 5.3
>Reporter: Aaron LaBella
> Attachments: SOLR-7885-1.patch, SOLR-7885-2.patch
>
>
> I have a need to be able to load data import handler configuration files from
> an HTTP server instead of the local file system. So, I modified
> {code}org.apache.solr.core.SolrResourceLoader{code} and some of the
> respective dataimport files in {code}org.apache.solr.handler.dataimport{code}
> to be able to support doing this.
> {code}solrconfig.xml{code} now has the option to define a parameter:
> *configRemote*, and if defined (and it's an HTTP(s) URL), it'll attempt to
> load the resource. If successfully, it'll also persist the resource to the
> local file system so that it is available on a solr server restart per chance
> that the remote resource is currently unavailable.
> Lastly, to be consistent with the pattern that already exists in
> SolrResourceLoader, this feature is *disabled* by default, and requires the
> setting of an additional JVM property:
> {code}-Dsolr.allow.http.resourceloading=true{code}.
> Please review and let me know if there is anything else that needs to be done
> in order for this patch to make the next release. As far as I can tell, it's
> fully tested and ready to go.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7885) Add support for loading HTTP resources
[
https://issues.apache.org/jira/browse/SOLR-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045068#comment-15045068
]
Aaron LaBella commented on SOLR-7885:
-
Sure. Our DIH configuration files are stored, protected and maintained on a
*separate* HTTP server with an admin/editing interface. This allows us to give
users an actual front-end editor, make changes to the DIH, and have those
changes reloaded in SOLR automatically -- *without* having to explicitly reload
SOLR in any form (either at runtime or with a server restart).
I guess I didn't try reload-config because a.) I'm not really seeing any great
documentation on this feature and b.) I'm confused if it actually *persists* an
uploaded and/or reloaded configuration back to the local file system. If not,
then this is a problem because it means that if the SOLR server restarts it'll
re-load the old configurations.
I suppose you could argue this as a "back door" to SOLR, but, it's also
something that is disabled by default and users would have to consciously
enable using -Dsolr.allow.http.resourceloading, assuming they are willing to
accept the risk.
> Add support for loading HTTP resources
> --
>
> Key: SOLR-7885
> URL: https://issues.apache.org/jira/browse/SOLR-7885
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler, SolrJ
>Affects Versions: 5.3
>Reporter: Aaron LaBella
> Attachments: SOLR-7885-1.patch, SOLR-7885-2.patch
>
>
> I have a need to be able to load data import handler configuration files from
> an HTTP server instead of the local file system. So, I modified
> {code}org.apache.solr.core.SolrResourceLoader{code} and some of the
> respective dataimport files in {code}org.apache.solr.handler.dataimport{code}
> to be able to support doing this.
> {code}solrconfig.xml{code} now has the option to define a parameter:
> *configRemote*, and if defined (and it's an HTTP(s) URL), it'll attempt to
> load the resource. If successfully, it'll also persist the resource to the
> local file system so that it is available on a solr server restart per chance
> that the remote resource is currently unavailable.
> Lastly, to be consistent with the pattern that already exists in
> SolrResourceLoader, this feature is *disabled* by default, and requires the
> setting of an additional JVM property:
> {code}-Dsolr.allow.http.resourceloading=true{code}.
> Please review and let me know if there is anything else that needs to be done
> in order for this patch to make the next release. As far as I can tell, it's
> fully tested and ready to go.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7885) Add support for loading HTTP resources
[
https://issues.apache.org/jira/browse/SOLR-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713468#comment-14713468
]
Aaron LaBella commented on SOLR-7885:
-
Can someone take a look at this patch and commit if approved?
Thanks.
Add support for loading HTTP resources
--
Key: SOLR-7885
URL: https://issues.apache.org/jira/browse/SOLR-7885
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler, SolrJ
Affects Versions: 5.3
Reporter: Aaron LaBella
Attachments: SOLR-7885-1.patch, SOLR-7885-2.patch
I have a need to be able to load data import handler configuration files from
an HTTP server instead of the local file system. So, I modified
{code}org.apache.solr.core.SolrResourceLoader{code} and some of the
respective dataimport files in {code}org.apache.solr.handler.dataimport{code}
to be able to support doing this.
{code}solrconfig.xml{code} now has the option to define a parameter:
*configRemote*, and if defined (and it's an HTTP(s) URL), it'll attempt to
load the resource. If successfully, it'll also persist the resource to the
local file system so that it is available on a solr server restart per chance
that the remote resource is currently unavailable.
Lastly, to be consistent with the pattern that already exists in
SolrResourceLoader, this feature is *disabled* by default, and requires the
setting of an additional JVM property:
{code}-Dsolr.allow.http.resourceloading=true{code}.
Please review and let me know if there is anything else that needs to be done
in order for this patch to make the next release. As far as I can tell, it's
fully tested and ready to go.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7760) Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration
[
https://issues.apache.org/jira/browse/SOLR-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680114#comment-14680114
]
Aaron LaBella commented on SOLR-7760:
-
Excellent, thanks so much guys! I added another patch/enhancement last week
too:
https://issues.apache.org/jira/browse/SOLR-7885
Are you able to take a look at that one and review/commit that too?
Thanks.
Fix method and field visibility for UIMAUpdateRequestProcessor and
SolrUIMAConfiguration
Key: SOLR-7760
URL: https://issues.apache.org/jira/browse/SOLR-7760
Project: Solr
Issue Type: Improvement
Components: contrib - UIMA
Affects Versions: 5x
Reporter: Aaron LaBella
Assignee: Tommaso Teofili
Priority: Critical
Fix For: Trunk, 5.4
Attachments: SOLR-7760.patch
The methods in
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java}}
are not public and they need to be for other code to be able to make use of
the configuration data, ie: mapped fields. Likewise,
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java}}
does not have an accessor for the SolrUIMAConfiguration object
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7885) Add support for loading HTTP resources
Aaron LaBella created SOLR-7885:
---
Summary: Add support for loading HTTP resources
Key: SOLR-7885
URL: https://issues.apache.org/jira/browse/SOLR-7885
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler, SolrJ
Affects Versions: 5.3
Reporter: Aaron LaBella
I have a need to be able to load data import handler configuration files from
an HTTP server instead of the local file system. So, I modified
{code}org.apache.solr.core.SolrResourceLoader{code} and some of the respective
dataimport files in {code}org.apache.solr.handler.dataimport{code} to be able
to support doing this.
{code}solrconfig.xml{code} now has the option to define a parameter:
*configRemote*, and if defined (and it's an HTTP(s) URL), it'll attempt to load
the resource. If successfully, it'll also persist the resource to the local
file system so that it is available on a solr server restart per chance that
the remote resource is currently unavailable.
Lastly, to be consistent with the pattern that already exists in
SolrResourceLoader, this feature is *disabled* by default, and requires the
setting of an additional JVM property:
{code}-Dsolr.allow.http.resourceloading=true{code}.
Please review and let me know if there is anything else that needs to be done
in order for this patch to make the next release. As far as I can tell, it's
fully tested and ready to go.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7760) Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration
[
https://issues.apache.org/jira/browse/SOLR-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660714#comment-14660714
]
Aaron LaBella commented on SOLR-7760:
-
Is someone able to commit this patch soon? It'd be great to get in the next
release of solr.
Thanks!
Fix method and field visibility for UIMAUpdateRequestProcessor and
SolrUIMAConfiguration
Key: SOLR-7760
URL: https://issues.apache.org/jira/browse/SOLR-7760
Project: Solr
Issue Type: Improvement
Components: contrib - UIMA
Affects Versions: 5x
Reporter: Aaron LaBella
Priority: Critical
Fix For: 5.3
Attachments: SOLR-7760.patch
The methods in
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java}}
are not public and they need to be for other code to be able to make use of
the configuration data, ie: mapped fields. Likewise,
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java}}
does not have an accessor for the SolrUIMAConfiguration object
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7885) Add support for loading HTTP resources
[
https://issues.apache.org/jira/browse/SOLR-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-7885:
Attachment: SOLR-7885-2.patch
SOLR-7885-1.patch
I split the patch into 2 parts, one for solr-core and one for dataimporthandler
Add support for loading HTTP resources
--
Key: SOLR-7885
URL: https://issues.apache.org/jira/browse/SOLR-7885
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler, SolrJ
Affects Versions: 5.3
Reporter: Aaron LaBella
Attachments: SOLR-7885-1.patch, SOLR-7885-2.patch
I have a need to be able to load data import handler configuration files from
an HTTP server instead of the local file system. So, I modified
{code}org.apache.solr.core.SolrResourceLoader{code} and some of the
respective dataimport files in {code}org.apache.solr.handler.dataimport{code}
to be able to support doing this.
{code}solrconfig.xml{code} now has the option to define a parameter:
*configRemote*, and if defined (and it's an HTTP(s) URL), it'll attempt to
load the resource. If successfully, it'll also persist the resource to the
local file system so that it is available on a solr server restart per chance
that the remote resource is currently unavailable.
Lastly, to be consistent with the pattern that already exists in
SolrResourceLoader, this feature is *disabled* by default, and requires the
setting of an additional JVM property:
{code}-Dsolr.allow.http.resourceloading=true{code}.
Please review and let me know if there is anything else that needs to be done
in order for this patch to make the next release. As far as I can tell, it's
fully tested and ready to go.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7760) Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration
[
https://issues.apache.org/jira/browse/SOLR-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643287#comment-14643287
]
Aaron LaBella commented on SOLR-7760:
-
Yes, I wrote a UimaSearchHandler that extends SearchHandler to do adhoc uima
analysis. It is configured to reference the same updateRequestProcessorChain
that is used during index time but in order to generically know what fields to
return I need public access to the SolrUIMAConfiguration and fieldMappings. I
don't have time to write a testcase, but, here's some sample code that uses it:
{code:java}
UpdateRequestProcessorChain chain =
req.getCore().getUpdateProcessingChain(uimaChain);
ListUpdateRequestProcessorFactory processors =
chain.getProcessors();
UpdateRequestProcessorFactory factory = processors.get(0);
UpdateRequestProcessor processor = factory.getInstance(req, null,
null);
SolrUIMAConfiguration conf =
((UIMAUpdateRequestProcessor)processor).getConfiguration();
MapString, MapString, MapField map =
conf.getTypesFeaturesFieldsMapping();
for(Map.EntryString,MapString,MapField entry : map.entrySet())
{
String annotationClass = entry.getKey();
annotationClasses.add(annotationClass);
}
{code}
Fix method and field visibility for UIMAUpdateRequestProcessor and
SolrUIMAConfiguration
Key: SOLR-7760
URL: https://issues.apache.org/jira/browse/SOLR-7760
Project: Solr
Issue Type: Improvement
Components: contrib - UIMA
Affects Versions: 5x
Reporter: Aaron LaBella
Priority: Critical
Fix For: 5.3
Attachments: SOLR-7760.patch
The methods in
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java}}
are not public and they need to be for other code to be able to make use of
the configuration data, ie: mapped fields. Likewise,
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java}}
does not have an accessor for the SolrUIMAConfiguration object
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7760) Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration
[
https://issues.apache.org/jira/browse/SOLR-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-7760:
Description: The methods in
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java}}
are not public and they need to be for other code to be able to make use of
the configuration data, ie: mapped fields. Likewise,
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java}}
does not have an accessor for the SolrUIMAConfiguration object (was: The
methods in
solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java
are not public and they need to be for other code to be able to make use of
the configuration data, ie: mapped fields. Likewise,
solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java
does not have an accessor for the SolrUIMAConfiguration object)
Fix method and field visibility for UIMAUpdateRequestProcessor and
SolrUIMAConfiguration
Key: SOLR-7760
URL: https://issues.apache.org/jira/browse/SOLR-7760
Project: Solr
Issue Type: Improvement
Components: contrib - UIMA
Affects Versions: 5x
Reporter: Aaron LaBella
Priority: Critical
Fix For: 5.3
Attachments: SOLR-7760.patch
The methods in
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java}}
are not public and they need to be for other code to be able to make use of
the configuration data, ie: mapped fields. Likewise,
{{solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java}}
does not have an accessor for the SolrUIMAConfiguration object
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7760) Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration
[ https://issues.apache.org/jira/browse/SOLR-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615457#comment-14615457 ] Aaron LaBella commented on SOLR-7760: - Can someone with write access to Solr repository please review, apply and commit the patch above? I'd like to see this in the next version of Solr. Thanks. Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration Key: SOLR-7760 URL: https://issues.apache.org/jira/browse/SOLR-7760 Project: Solr Issue Type: Improvement Components: contrib - UIMA Affects Versions: 5x Reporter: Aaron LaBella Priority: Critical Fix For: 5.3 Attachments: SOLR-7760.patch The methods in solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java are not public and they need to be for other code to be able to make use of the configuration data, ie: mapped fields. Likewise, solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java does not have an accessor for the SolrUIMAConfiguration object -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7760) Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration
[ https://issues.apache.org/jira/browse/SOLR-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-7760: Attachment: SOLR-7760.patch patch per issue description Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration Key: SOLR-7760 URL: https://issues.apache.org/jira/browse/SOLR-7760 Project: Solr Issue Type: Improvement Components: contrib - UIMA Affects Versions: 5x Reporter: Aaron LaBella Priority: Critical Fix For: 5.3 Attachments: SOLR-7760.patch The methods in solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java are not public and they need to be for other code to be able to make use of the configuration data, ie: mapped fields. Likewise, solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java does not have an accessor for the SolrUIMAConfiguration object -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7760) Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration
Aaron LaBella created SOLR-7760: --- Summary: Fix method and field visibility for UIMAUpdateRequestProcessor and SolrUIMAConfiguration Key: SOLR-7760 URL: https://issues.apache.org/jira/browse/SOLR-7760 Project: Solr Issue Type: Improvement Components: contrib - UIMA Affects Versions: 5x Reporter: Aaron LaBella Priority: Critical Fix For: 5.3 The methods in solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java are not public and they need to be for other code to be able to make use of the configuration data, ie: mapped fields. Likewise, solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java does not have an accessor for the SolrUIMAConfiguration object -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6194) Allow access to DataImporter and DIHConfiguration
[
https://issues.apache.org/jira/browse/SOLR-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-6194:
Attachment: SOLR-6194.patch
another small patch
Allow access to DataImporter and DIHConfiguration
-
Key: SOLR-6194
URL: https://issues.apache.org/jira/browse/SOLR-6194
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 4.10
Reporter: Aaron LaBella
Assignee: Shalin Shekhar Mangar
Fix For: 4.10
Attachments: SOLR-6194.patch, SOLR-6194.patch
Original Estimate: 2h
Remaining Estimate: 2h
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully
resolved) that are executed for an entity within the data import
configuration. This makes it much easier to debug the dih, rather than
turning on debug/verbose flags and digging through the raw response.
Additionally, it gives me a service that I can then go take the queries
from and run them.
Here's a snippet of Java code that I can now execute now that I have access
to the DIHConfiguration:
{code:title=Snippet.java|borderStyle=solid}
/**
* @return a map of all the queries for each entity in the given config
*/
protected MapString,String getEntityQueries(DIHConfiguration config,
MapString,Object params)
{
MapString,String queries = new LinkedHashMap();
if (config != null config.getEntities() != null)
{
//make a new variable resolve
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
//for each entity
for (Entity e : config.getEntities())
{
//get the query and resolve it
if (e.getAllAttributes().containsKey(SqlEntityProcessor.QUERY))
{
String query = e.getAllAttributes().get(SqlEntityProcessor.QUERY);
query = query.replaceAll(\\s+, ).trim();
String resolved = vr.replaceTokens(query);
resolved = resolved.replaceAll(\\s+, ).trim();
queries.put(e.getName(),resolved);
queries.put(e.getName()+_raw,query);
}
}
}
return queries;
}
{code}
I'm attaching a patch that I would appreciate someone have a look for
consideration. It's fully tested -- please let me know if there is something
else I need to do/test.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (SOLR-6194) Allow access to DataImporter and DIHConfiguration
[
https://issues.apache.org/jira/browse/SOLR-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella reopened SOLR-6194:
-
Thanks shalin! I'm attaching one more small patch that opens up a couple
other methods. Now with these fixes I have the ability to actually run the
entity queries from my data import handler for easier debug/inspection,
*without* actually having to run an import. Can you please review and commit
this small patch as well?
Aaron
Allow access to DataImporter and DIHConfiguration
-
Key: SOLR-6194
URL: https://issues.apache.org/jira/browse/SOLR-6194
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 4.10
Reporter: Aaron LaBella
Assignee: Shalin Shekhar Mangar
Fix For: 4.10
Attachments: SOLR-6194.patch, SOLR-6194.patch
Original Estimate: 2h
Remaining Estimate: 2h
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully
resolved) that are executed for an entity within the data import
configuration. This makes it much easier to debug the dih, rather than
turning on debug/verbose flags and digging through the raw response.
Additionally, it gives me a service that I can then go take the queries
from and run them.
Here's a snippet of Java code that I can now execute now that I have access
to the DIHConfiguration:
{code:title=Snippet.java|borderStyle=solid}
/**
* @return a map of all the queries for each entity in the given config
*/
protected MapString,String getEntityQueries(DIHConfiguration config,
MapString,Object params)
{
MapString,String queries = new LinkedHashMap();
if (config != null config.getEntities() != null)
{
//make a new variable resolve
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
//for each entity
for (Entity e : config.getEntities())
{
//get the query and resolve it
if (e.getAllAttributes().containsKey(SqlEntityProcessor.QUERY))
{
String query = e.getAllAttributes().get(SqlEntityProcessor.QUERY);
query = query.replaceAll(\\s+, ).trim();
String resolved = vr.replaceTokens(query);
resolved = resolved.replaceAll(\\s+, ).trim();
queries.put(e.getName(),resolved);
queries.put(e.getName()+_raw,query);
}
}
}
return queries;
}
{code}
I'm attaching a patch that I would appreciate someone have a look for
consideration. It's fully tested -- please let me know if there is something
else I need to do/test.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6194) Allow access to DataImporter and DIHConfiguration
[
https://issues.apache.org/jira/browse/SOLR-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046083#comment-14046083
]
Aaron LaBella commented on SOLR-6194:
-
Shalin,
Good feedback. Here's the static utility class I wrote:
{code:title=Snippet.java|borderStyle=solid}
//...
final public class SolrQueryUtils
{
static public void runEntityQueries(
DIHConfiguration config, SolrCore core, DataImporter importer,
Entity entity, String entityName,
SolrQueryRequest req, SolrQueryResponse rsp,
MapString,Object params) throws Exception
{
CollectionMapString,Object rows = new LinkedList();
EntityProcessorWrapper epw = null;
try
{
if (entityName == null || entity.getName().equals(entityName))
{
epw = SolrQueryUtils.getEntityProcessorWrapperFrom(
core, importer, entity, req, rsp, params
);
if (epw != null)
{
MapString, Object row = epw.nextRow();
//just for sanity, TODO: add a better way to do this
final int MAX = 1;
int index = 0;
while (row != null)
{
if (index++ MAX) { rows.add(row); }
row = epw.nextRow();
}
}
}
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
if (epw != null)
{
epw.getDatasource().close();
epw.close();
epw.destroy();
}
}
if (entityName == null || entity.getName().equals(entityName))
{
rsp.add(entity.getName()+_results,rows);
}
//potentially recurse if there are children
CollectionEntity children = entity.getChildren();
if (children != null)
{
//for each child entity
for (Entity child : children)
{
SolrQueryUtils.runEntityQueries(config,core,importer,child,entityName,req,rsp,params);
}
}
}
static public EntityProcessorWrapper getEntityProcessorWrapperFrom(
SolrCore core, DataImporter importer, Entity entity,
SolrQueryRequest req, SolrQueryResponse rsp,
MapString,Object params) throws Exception
{
RequestInfo reqinfo = new RequestInfo(req,params,null);
DocBuilder docBuilder = importer.getDocBuilder(null,reqinfo);
EntityProcessorWrapper epw = docBuilder.getEntityProcessorWrapper(entity);
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
Context ctx = new ContextImpl(
epw, //EntityProcessorWrapper epw
vr, //VariableResolver resolver
null, //DataSource ds
Context.FULL_DUMP,//String currProcess
reqinfo.getRawParams(), //MapString, Object global
null, //ContextImpl parentContext
docBuilder//DocBuilder docBuilder
);
DataSource? ds = importer.getDataSourceInstance(
entity, entity.getDataSourceName(), ctx
);
SolrQueryUtils.initDataSource(importer.getConfig(),core,entity,ds,ctx);
epw.setDatasource(ds);
epw.init(ctx);
epw.setInitalized(true);
return epw;
}
static public void initDataSource(
DIHConfiguration config, SolrCore core, Entity entity, DataSource? ds,
Context ctx
)
{
//add all the properties from the core descriptor
CoreDescriptor descriptor = core.getCoreDescriptor();
MapString,Object globals = new LinkedHashMap();
globals =
SolrQueryUtils.addProperties(descriptor.getPersistableStandardProperties(),
globals);
globals =
SolrQueryUtils.addProperties(descriptor.getPersistableUserProperties(),
globals);
//sort the keys for easier debugging
globals = new TreeMap(globals);
//make a new variable for the datasource properties resolver
VariableResolver globalResolver = new VariableResolver(globals);
MapString,String dsProps =
config.getDataSources().get(entity.getDataSourceName());
MapString,String dsPropsResolved = new LinkedHashMap();
for(Map.EntryString,String entry : dsProps.entrySet())
{
dsPropsResolved.put(entry.getKey(),globalResolver.replaceTokens(entry.getValue()));
}
Properties dsProperties = new Properties();
dsProperties.putAll(dsPropsResolved);
ds.init(ctx,dsProperties);
}
static public MapString,Object addProperties(Properties p,
MapString,Object map)
{
if (p != null map != null)
{
Enumeration? enumer = p.propertyNames();
while(enumer.hasMoreElements())
{
String key = enumer.nextElement().toString();
String value = p.getProperty(key);
map.put(key,value);
}
}
return map;
}
}
{code}
And, I have a custom data import handler that extends DataImportHandler. I
added support for calling the data import handler with command='runquery',
which ends up looking up the entities from the config, and calling the
[jira] [Commented] (SOLR-6194) Allow access to DataImporter and DIHConfiguration
[
https://issues.apache.org/jira/browse/SOLR-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046087#comment-14046087
]
Aaron LaBella commented on SOLR-6194:
-
NOTE: as you can see above ... actually getting the EntityProcessorWrapper, and
initializing the datasource, seems like it is much more code than I would've
hoped/expected for. I think the solr data import handler framework is an
incredible piece of software, it's just unfortunate that the API's aren't a
little more open and flexible to allow for extensibility and use-cases beyond
just getting data into solr. I could envision using the code base to define
adhoc queries that support things like variable resolution, evaluators,
transformers, etc -- it's very powerful stuff ;-)
Thanks.
Allow access to DataImporter and DIHConfiguration
-
Key: SOLR-6194
URL: https://issues.apache.org/jira/browse/SOLR-6194
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 4.10
Reporter: Aaron LaBella
Assignee: Shalin Shekhar Mangar
Fix For: 4.10
Attachments: SOLR-6194.patch, SOLR-6194.patch
Original Estimate: 2h
Remaining Estimate: 2h
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully
resolved) that are executed for an entity within the data import
configuration. This makes it much easier to debug the dih, rather than
turning on debug/verbose flags and digging through the raw response.
Additionally, it gives me a service that I can then go take the queries
from and run them.
Here's a snippet of Java code that I can now execute now that I have access
to the DIHConfiguration:
{code:title=Snippet.java|borderStyle=solid}
/**
* @return a map of all the queries for each entity in the given config
*/
protected MapString,String getEntityQueries(DIHConfiguration config,
MapString,Object params)
{
MapString,String queries = new LinkedHashMap();
if (config != null config.getEntities() != null)
{
//make a new variable resolve
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
//for each entity
for (Entity e : config.getEntities())
{
//get the query and resolve it
if (e.getAllAttributes().containsKey(SqlEntityProcessor.QUERY))
{
String query = e.getAllAttributes().get(SqlEntityProcessor.QUERY);
query = query.replaceAll(\\s+, ).trim();
String resolved = vr.replaceTokens(query);
resolved = resolved.replaceAll(\\s+, ).trim();
queries.put(e.getName(),resolved);
queries.put(e.getName()+_raw,query);
}
}
}
return queries;
}
{code}
I'm attaching a patch that I would appreciate someone have a look for
consideration. It's fully tested -- please let me know if there is something
else I need to do/test.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6194) Allow access to DataImporter and DIHConfiguration
Aaron LaBella created SOLR-6194:
---
Summary: Allow access to DataImporter and DIHConfiguration
Key: SOLR-6194
URL: https://issues.apache.org/jira/browse/SOLR-6194
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 4.10
Reporter: Aaron LaBella
Fix For: 4.10
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully resolved)
that are executed for an entity within the data import configuration. This
makes it much easier to debug the dih, rather than turning on debug/verbose
flags and digging through the raw response. Additionally, it gives me a
service that I can then go take the queries from and run them.
Here's a snippet of Java code that I can now execute now that I have access to
the DIHConfiguration:
/**
* @return a map of all the queries for each entity in the given config
*/
protected MapString,String getEntityQueries(DIHConfiguration config,
MapString,Object params)
{
MapString,String queries = new LinkedHashMap();
if (config != null config.getEntities() != null)
{
//make a new variable resolve
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
//for each entity
for (Entity e : config.getEntities())
{
//get the query and resolve it
if (e.getAllAttributes().containsKey(SqlEntityProcessor.QUERY))
{
String query = e.getAllAttributes().get(SqlEntityProcessor.QUERY);
query = query.replaceAll(\\s+, ).trim();
String resolved = vr.replaceTokens(query);
resolved = resolved.replaceAll(\\s+, ).trim();
queries.put(e.getName(),resolved);
queries.put(e.getName()+_raw,query);
}
}
}
return queries;
}
I'm attaching a patch that I would appreciate someone have a look for
consideration. It's fully tested -- please let me know if there is something
else I need to do/test.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6194) Allow access to DataImporter and DIHConfiguration
[
https://issues.apache.org/jira/browse/SOLR-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-6194:
Attachment: SOLR-6194.patch
Allow access to DataImporter and DIHConfiguration
-
Key: SOLR-6194
URL: https://issues.apache.org/jira/browse/SOLR-6194
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 4.10
Reporter: Aaron LaBella
Fix For: 4.10
Attachments: SOLR-6194.patch
Original Estimate: 2h
Remaining Estimate: 2h
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully
resolved) that are executed for an entity within the data import
configuration. This makes it much easier to debug the dih, rather than
turning on debug/verbose flags and digging through the raw response.
Additionally, it gives me a service that I can then go take the queries
from and run them.
Here's a snippet of Java code that I can now execute now that I have access
to the DIHConfiguration:
{code:title=Snippet.java|borderStyle=solid}
/**
* @return a map of all the queries for each entity in the given config
*/
protected MapString,String getEntityQueries(DIHConfiguration config,
MapString,Object params)
{
MapString,String queries = new LinkedHashMap();
if (config != null config.getEntities() != null)
{
//make a new variable resolve
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
//for each entity
for (Entity e : config.getEntities())
{
//get the query and resolve it
if (e.getAllAttributes().containsKey(SqlEntityProcessor.QUERY))
{
String query = e.getAllAttributes().get(SqlEntityProcessor.QUERY);
query = query.replaceAll(\\s+, ).trim();
String resolved = vr.replaceTokens(query);
resolved = resolved.replaceAll(\\s+, ).trim();
queries.put(e.getName(),resolved);
queries.put(e.getName()+_raw,query);
}
}
}
return queries;
}
{code}
I'm attaching a patch that I would appreciate someone have a look for
consideration. It's fully tested -- please let me know if there is something
else I need to do/test.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6194) Allow access to DataImporter and DIHConfiguration
[
https://issues.apache.org/jira/browse/SOLR-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-6194:
Description:
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully resolved)
that are executed for an entity within the data import configuration. This
makes it much easier to debug the dih, rather than turning on debug/verbose
flags and digging through the raw response. Additionally, it gives me a
service that I can then go take the queries from and run them.
Here's a snippet of Java code that I can now execute now that I have access to
the DIHConfiguration:
{code:title=Snippet.java|borderStyle=solid}
/**
* @return a map of all the queries for each entity in the given config
*/
protected MapString,String getEntityQueries(DIHConfiguration config,
MapString,Object params)
{
MapString,String queries = new LinkedHashMap();
if (config != null config.getEntities() != null)
{
//make a new variable resolve
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
//for each entity
for (Entity e : config.getEntities())
{
//get the query and resolve it
if (e.getAllAttributes().containsKey(SqlEntityProcessor.QUERY))
{
String query = e.getAllAttributes().get(SqlEntityProcessor.QUERY);
query = query.replaceAll(\\s+, ).trim();
String resolved = vr.replaceTokens(query);
resolved = resolved.replaceAll(\\s+, ).trim();
queries.put(e.getName(),resolved);
queries.put(e.getName()+_raw,query);
}
}
}
return queries;
}
{code}
I'm attaching a patch that I would appreciate someone have a look for
consideration. It's fully tested -- please let me know if there is something
else I need to do/test.
was:
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully resolved)
that are executed for an entity within the data import configuration. This
makes it much easier to debug the dih, rather than turning on debug/verbose
flags and digging through the raw response. Additionally, it gives me a
service that I can then go take the queries from and run them.
Here's a snippet of Java code that I can now execute now that I have access to
the DIHConfiguration:
/**
* @return a map of all the queries for each entity in the given config
*/
protected MapString,String getEntityQueries(DIHConfiguration config,
MapString,Object params)
{
MapString,String queries = new LinkedHashMap();
if (config != null config.getEntities() != null)
{
//make a new variable resolve
VariableResolver vr = new VariableResolver();
vr.addNamespace(dataimporter.request,params);
//for each entity
for (Entity e : config.getEntities())
{
//get the query and resolve it
if (e.getAllAttributes().containsKey(SqlEntityProcessor.QUERY))
{
String query = e.getAllAttributes().get(SqlEntityProcessor.QUERY);
query = query.replaceAll(\\s+, ).trim();
String resolved = vr.replaceTokens(query);
resolved = resolved.replaceAll(\\s+, ).trim();
queries.put(e.getName(),resolved);
queries.put(e.getName()+_raw,query);
}
}
}
return queries;
}
I'm attaching a patch that I would appreciate someone have a look for
consideration. It's fully tested -- please let me know if there is something
else I need to do/test.
Allow access to DataImporter and DIHConfiguration
-
Key: SOLR-6194
URL: https://issues.apache.org/jira/browse/SOLR-6194
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 4.10
Reporter: Aaron LaBella
Fix For: 4.10
Attachments: SOLR-6194.patch
Original Estimate: 2h
Remaining Estimate: 2h
I'd like to change the visibility and access to a couple of the internal
classes of DataImportHandler, specifically DataImporter and DIHConfiguration.
My reasoning is that I've added the ability for a new data import handler
command called *getquery* that will return the exact queries (fully
resolved) that are executed for an entity within the data import
configuration. This makes it much easier to debug the dih, rather
[jira] [Commented] (SOLR-6129) DateFormatTransformer doesn't resolve dateTimeFormat
[ https://issues.apache.org/jira/browse/SOLR-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14035676#comment-14035676 ] Aaron LaBella commented on SOLR-6129: - excellent. thanks!! DateFormatTransformer doesn't resolve dateTimeFormat Key: SOLR-6129 URL: https://issues.apache.org/jira/browse/SOLR-6129 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.8.1 Reporter: Aaron LaBella Assignee: Shalin Shekhar Mangar Fix For: 4.9, 5.0 Attachments: SOLR-6129.patch, SOLR-6129.patch Original Estimate: 1h Remaining Estimate: 1h The DateFormatTransformer doesn't use a VariableResolver to resolve the format, which could potentially be passed in via a dataimport request parameter. The attached patch is tested and working. Please include in 4.9. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6129) DateFormatTransformer doesn't resolve dateTimeFormat
[ https://issues.apache.org/jira/browse/SOLR-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033942#comment-14033942 ] Aaron LaBella commented on SOLR-6129: - Anyone out there? This patch is ready to go -- just needs someone to approve/accept/commit. Thanks. DateFormatTransformer doesn't resolve dateTimeFormat Key: SOLR-6129 URL: https://issues.apache.org/jira/browse/SOLR-6129 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.8.1 Reporter: Aaron LaBella Fix For: 4.9 Attachments: SOLR-6129.patch Original Estimate: 1h Remaining Estimate: 1h The DateFormatTransformer doesn't use a VariableResolver to resolve the format, which could potentially be passed in via a dataimport request parameter. The attached patch is tested and working. Please include in 4.9. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6018) Solr DataImportHandler not finding dynamic fields
[
https://issues.apache.org/jira/browse/SOLR-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019924#comment-14019924
]
Aaron LaBella commented on SOLR-6018:
-
Any comments/updates on this issue?
Solr DataImportHandler not finding dynamic fields
-
Key: SOLR-6018
URL: https://issues.apache.org/jira/browse/SOLR-6018
Project: Solr
Issue Type: Bug
Components: contrib - DataImportHandler
Affects Versions: 4.7
Reporter: Aaron LaBella
Fix For: 4.9
Attachments: 0001-dih-config-check-for-dynamic-fields.patch
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match logic in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_S type=string indexed=true stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6129) DateFormatTransformer doesn't resolve dateTimeFormat
[ https://issues.apache.org/jira/browse/SOLR-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018763#comment-14018763 ] Aaron LaBella commented on SOLR-6129: - Can someone have a look at this patch and hopefully commit it? I'm looking to get it in for the 4.9 release. Thanks. DateFormatTransformer doesn't resolve dateTimeFormat Key: SOLR-6129 URL: https://issues.apache.org/jira/browse/SOLR-6129 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.8.1 Reporter: Aaron LaBella Fix For: 4.9 Attachments: SOLR-6129.patch Original Estimate: 1h Remaining Estimate: 1h The DateFormatTransformer doesn't use a VariableResolver to resolve the format, which could potentially be passed in via a dataimport request parameter. The attached patch is tested and working. Please include in 4.9. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6129) DateFormatTransformer doesn't resolve dateTimeFormat
Aaron LaBella created SOLR-6129: --- Summary: DateFormatTransformer doesn't resolve dateTimeFormat Key: SOLR-6129 URL: https://issues.apache.org/jira/browse/SOLR-6129 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.8.1 Reporter: Aaron LaBella Fix For: 4.9 The DateFormatTransformer doesn't use a VariableResolver to resolve the format, which could potentially be passed in via a dataimport request parameter. The attached patch is tested and working. Please include in 4.9. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6129) DateFormatTransformer doesn't resolve dateTimeFormat
[ https://issues.apache.org/jira/browse/SOLR-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-6129: Attachment: SOLR-6129.patch DateFormatTransformer doesn't resolve dateTimeFormat Key: SOLR-6129 URL: https://issues.apache.org/jira/browse/SOLR-6129 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.8.1 Reporter: Aaron LaBella Fix For: 4.9 Attachments: SOLR-6129.patch Original Estimate: 1h Remaining Estimate: 1h The DateFormatTransformer doesn't use a VariableResolver to resolve the format, which could potentially be passed in via a dataimport request parameter. The attached patch is tested and working. Please include in 4.9. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6013) Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility
[ https://issues.apache.org/jira/browse/SOLR-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992748#comment-13992748 ] Aaron LaBella commented on SOLR-6013: - Shalin, that's fine I suppose (sorry I didn't notice you changed them to public/final). I'm just wondering though, wouldn't it make sense to access the bean properties using traditional getter methods instead of accessing them directly? Just curious as to the reasoning of not providing the getters. In either case, I'm fine with whatever you decide and re-closing this issue. Thanks. Aaron Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility -- Key: SOLR-6013 URL: https://issues.apache.org/jira/browse/SOLR-6013 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Assignee: Shalin Shekhar Mangar Fix For: 4.9, 5.0 Attachments: 0001-add-getters-for-datemathparser.patch, 0001-change-method-access-to-protected.patch, 0001-change-method-variable-visibility-and-refactor-for-extensibility.patch, SOLR-6013.patch Original Estimate: 1h Remaining Estimate: 1h This is similar to issue 5981, the Evaluator class is declared as abstract, yet the parseParams method is package private? Surely this is an oversight, as I wouldn't expect everyone writing their own evaluators to have to deal with parsing the parameters. Similarly, I needed to refactor DateFormatEvaluator because I need to do some custom date math/parsing and it wasn't written in a way that I can extend it. Please review/apply my attached patch to the next version of Solr, ie: 4.8 or 4.9 if I must wait. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (SOLR-6013) Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility
[
https://issues.apache.org/jira/browse/SOLR-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella reopened SOLR-6013:
-
Thanks Shalin, but, you forgot two of the public 'getter' methods that I added
in the last patch to Evaluator.VariableWrapper:
public String getVarName() {
return this.varName;
}
public VariableResolver getVariableResolver() {
return this.vr;
}
Can you fix/commit/close?
Thanks.
Aaron
Fix method visibility of Evaluator, refactor DateFormatEvaluator for
extensibility
--
Key: SOLR-6013
URL: https://issues.apache.org/jira/browse/SOLR-6013
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 4.7
Reporter: Aaron LaBella
Assignee: Shalin Shekhar Mangar
Fix For: 4.9, 5.0
Attachments: 0001-add-getters-for-datemathparser.patch,
0001-change-method-access-to-protected.patch,
0001-change-method-variable-visibility-and-refactor-for-extensibility.patch,
SOLR-6013.patch
Original Estimate: 1h
Remaining Estimate: 1h
This is similar to issue 5981, the Evaluator class is declared as abstract,
yet the parseParams method is package private? Surely this is an oversight,
as I wouldn't expect everyone writing their own evaluators to have to deal
with parsing the parameters.
Similarly, I needed to refactor DateFormatEvaluator because I need to do some
custom date math/parsing and it wasn't written in a way that I can extend it.
Please review/apply my attached patch to the next version of Solr, ie: 4.8 or
4.9 if I must wait.
Thanks!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3671) DIH doesn't use its own interface + writerImpl has no information about the request
[ https://issues.apache.org/jira/browse/SOLR-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989627#comment-13989627 ] Aaron LaBella commented on SOLR-3671: - James, I downloaded the patch and refactored my code/test case and everything works fine. Can you go ahead and commit that to branch_4x? Your patch cleans up the code to use the DIHWriter interface, which is better anyhow. Thanks. DIH doesn't use its own interface + writerImpl has no information about the request --- Key: SOLR-3671 URL: https://issues.apache.org/jira/browse/SOLR-3671 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0-ALPHA, 4.0-BETA Reporter: Roman Chyla Assignee: James Dyer Priority: Minor Attachments: SOLR-3671.patch, SOLR-3671.patch The use case: I would like to extend DIH by providing a new writer, I have tried everything but can't accomplish it without either a) duplicating whole DIHandler or b) java reflection tricks. Almost everything inside DIH is private and the mechanism to instantiate a new writer based on the 'writerImpl' mechanism seems lacking important functionality It doesn't give the new class a chance to get information about the request, update processor. Also, the writer is instantiated twice (when 'writerImpl' is there), which is really unnecessary. As a solution, the existing DIHandler.getSolrWriter() should instantiate the appropriate writer and send it to DocBuilder (it is already doing that for SolrWriter). And DocBuilder doesn't need to create a second (duplicate) writer -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
[ https://issues.apache.org/jira/browse/SOLR-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989625#comment-13989625 ] Aaron LaBella commented on SOLR-5981: - James, I downloaded the patch from SOLR-3671 and refactored my code/test case and everything works fine. Can you go ahead and commit that to branch_4x? Your patch cleans up the code to use the DIHWriter interface, which is better anyhow. Thanks. Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) --- Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Assignee: Shawn Heisey Priority: Minor Fix For: 4.9, 5.0 Attachments: SOLR-5981.patch Original Estimate: 1h Remaining Estimate: 1h I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6013) Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility
[ https://issues.apache.org/jira/browse/SOLR-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982999#comment-13982999 ] Aaron LaBella commented on SOLR-6013: - NOTE: the patches can be applied from oldest to newest Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility -- Key: SOLR-6013 URL: https://issues.apache.org/jira/browse/SOLR-6013 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Fix For: 4.9 Attachments: 0001-add-getters-for-datemathparser.patch, 0001-change-method-access-to-protected.patch, 0001-change-method-variable-visibility-and-refactor-for-extensibility.patch Original Estimate: 1h Remaining Estimate: 1h This is similar to issue 5981, the Evaluator class is declared as abstract, yet the parseParams method is package private? Surely this is an oversight, as I wouldn't expect everyone writing their own evaluators to have to deal with parsing the parameters. Similarly, I needed to refactor DateFormatEvaluator because I need to do some custom date math/parsing and it wasn't written in a way that I can extend it. Please review/apply my attached patch to the next version of Solr, ie: 4.8 or 4.9 if I must wait. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6013) Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility
[ https://issues.apache.org/jira/browse/SOLR-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-6013: Attachment: 0001-change-method-access-to-protected.patch Thanks Shalin, I'm attaching another patch to change the method accessors to protected (instead of public) and marked the methods as lucene.experimental. Let me know if there's anything else. Otherwise, can you, or someone else commit/push these patches into the 4.x branch so it makes the next release? Thanks Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility -- Key: SOLR-6013 URL: https://issues.apache.org/jira/browse/SOLR-6013 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Fix For: 4.9 Attachments: 0001-add-getters-for-datemathparser.patch, 0001-change-method-access-to-protected.patch, 0001-change-method-variable-visibility-and-refactor-for-extensibility.patch Original Estimate: 1h Remaining Estimate: 1h This is similar to issue 5981, the Evaluator class is declared as abstract, yet the parseParams method is package private? Surely this is an oversight, as I wouldn't expect everyone writing their own evaluators to have to deal with parsing the parameters. Similarly, I needed to refactor DateFormatEvaluator because I need to do some custom date math/parsing and it wasn't written in a way that I can extend it. Please review/apply my attached patch to the next version of Solr, ie: 4.8 or 4.9 if I must wait. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
[ https://issues.apache.org/jira/browse/SOLR-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981186#comment-13981186 ] Aaron LaBella commented on SOLR-5981: - Sure ... I did a proof of concept to use the DataImportHandler framework to import into mongodb. I think the architecture and functionality that DIH supports is fantastic (ie: evaluators, transformers, etc.), and the only import that mongodb supports (as far as I know) is a csv. So, I took advantage of the solr code base here to do everything that the DIH does, ie: connect to a DB and get data, just instead of dumping the results into the solr index, I actually create mongodb documents. Actually, my proof of concept supports two modes: insert and copy -- the former just inserts into mongodb and skips solr, the second will insert documents into both. Turns out someone else had a similar idea, but, they re-wrote half the solr dih framework: http://code.google.com/p/sql-to-nosql-importer/ My solution only requires a small extension... I'm happy to share it with the solr community if anyone else wants it. I think using mongodb as the document store and solr to index just the fields of the document you want to search on has the most potential for serious scalability. Let me know if you have any additional questions/thoughts/comments. Thanks. Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) --- Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Assignee: Shawn Heisey Priority: Minor Fix For: 4.9, 5.0 Attachments: SOLR-5981.patch Original Estimate: 1h Remaining Estimate: 1h I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6018) Solr DataImportHandler not finding dynamic fields
Aaron LaBella created SOLR-6018: --- Summary: Solr DataImportHandler not finding dynamic fields Key: SOLR-6018 URL: https://issues.apache.org/jira/browse/SOLR-6018 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Fix For: 4.9 There is an issue with org.apache.solr.handler.dataimport.DocBuilder.addFields around ~line 643. The logic currently says see if you can find the field from the schema, ie: SchemaField sf = schema.getFieldOrNull(key); and, if not found, go ask DIHConfiguration to find it: -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6018) Solr DataImportHandler not finding dynamic fields
[ https://issues.apache.org/jira/browse/SOLR-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-6018: Description: There is an issue with org.apache.solr.handler.dataimport.DocBuilder.addFields around ~line 643. The logic currently says see if you can find the field from the schema, ie: SchemaField sf = schema.getFieldOrNull(key); and, if not found, go ask DIHConfiguration to find it, ie: sf = config.getSchemaField(key); The latter call takes into account case-insensitivity, which is a big deal since some databases, ie: DB2, upper case all the resulting column names. In order to not modify solr-core (ie: the match login in IndexSchema), I'm attaching a patch that makes DIHConfiguration apply the same case-insensitive logic to the DynamicFields. Without this patch, dynamic fields will not be added to the index unless you declare them like this: dynamicField name=*_Stype=string indexed=true stored=true / (note the capital S) which is in-consistent with what I believe to be solr schema conventions to have all the schema fields as lower-case. Thanks. was: There is an issue with org.apache.solr.handler.dataimport.DocBuilder.addFields around ~line 643. The logic currently says see if you can find the field from the schema, ie: SchemaField sf = schema.getFieldOrNull(key); and, if not found, go ask DIHConfiguration to find it: Solr DataImportHandler not finding dynamic fields - Key: SOLR-6018 URL: https://issues.apache.org/jira/browse/SOLR-6018 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Fix For: 4.9 There is an issue with org.apache.solr.handler.dataimport.DocBuilder.addFields around ~line 643. The logic currently says see if you can find the field from the schema, ie: SchemaField sf = schema.getFieldOrNull(key); and, if not found, go ask DIHConfiguration to find it, ie: sf = config.getSchemaField(key); The latter call takes into account case-insensitivity, which is a big deal since some databases, ie: DB2, upper case all the resulting column names. In order to not modify solr-core (ie: the match login in IndexSchema), I'm attaching a patch that makes DIHConfiguration apply the same case-insensitive logic to the DynamicFields. Without this patch, dynamic fields will not be added to the index unless you declare them like this: dynamicField name=*_Stype=string indexed=true stored=true / (note the capital S) which is in-consistent with what I believe to be solr schema conventions to have all the schema fields as lower-case. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6018) Solr DataImportHandler not finding dynamic fields
[
https://issues.apache.org/jira/browse/SOLR-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-6018:
Description:
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_Stype=string indexed=true stored=true
/
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
was:
There is an issue with org.apache.solr.handler.dataimport.DocBuilder.addFields
around ~line 643. The logic currently says see if you can find the field from
the schema, ie:
SchemaField sf = schema.getFieldOrNull(key);
and, if not found, go ask DIHConfiguration to find it, ie:
sf = config.getSchemaField(key);
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
dynamicField name=*_Stype=string indexed=true stored=true
/
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
Solr DataImportHandler not finding dynamic fields
-
Key: SOLR-6018
URL: https://issues.apache.org/jira/browse/SOLR-6018
Project: Solr
Issue Type: Bug
Components: contrib - DataImportHandler
Affects Versions: 4.7
Reporter: Aaron LaBella
Fix For: 4.9
Attachments: 0001-dih-config-check-for-dynamic-fields.patch
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_Stype=string indexed=true
stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6018) Solr DataImportHandler not finding dynamic fields
[
https://issues.apache.org/jira/browse/SOLR-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-6018:
Description:
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_S type=string indexed=true stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
was:
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_Stype=string indexed=true stored=true
/
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
Solr DataImportHandler not finding dynamic fields
-
Key: SOLR-6018
URL: https://issues.apache.org/jira/browse/SOLR-6018
Project: Solr
Issue Type: Bug
Components: contrib - DataImportHandler
Affects Versions: 4.7
Reporter: Aaron LaBella
Fix For: 4.9
Attachments: 0001-dih-config-check-for-dynamic-fields.patch
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_S type=string indexed=true stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6018) Solr DataImportHandler not finding dynamic fields
[
https://issues.apache.org/jira/browse/SOLR-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-6018:
Attachment: 0001-dih-config-check-for-dynamic-fields.patch
Solr DataImportHandler not finding dynamic fields
-
Key: SOLR-6018
URL: https://issues.apache.org/jira/browse/SOLR-6018
Project: Solr
Issue Type: Bug
Components: contrib - DataImportHandler
Affects Versions: 4.7
Reporter: Aaron LaBella
Fix For: 4.9
Attachments: 0001-dih-config-check-for-dynamic-fields.patch
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_Stype=string indexed=true
stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
[ https://issues.apache.org/jira/browse/SOLR-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981251#comment-13981251 ] Aaron LaBella commented on SOLR-5981: - James, good point. But, I think the natural place to start extending things is by extending the DataImportHandler so that you can take advantage of a request handler, ie: requestHandler name=/dih class=org.apache.solr.handler.dataimport.DataImportHandler Otherwise, you have to start doing all that work yourself. Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) --- Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Assignee: Shawn Heisey Priority: Minor Fix For: 4.9, 5.0 Attachments: SOLR-5981.patch Original Estimate: 1h Remaining Estimate: 1h I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6018) Solr DataImportHandler not finding dynamic fields
[
https://issues.apache.org/jira/browse/SOLR-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron LaBella updated SOLR-6018:
Description:
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match logic in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_S type=string indexed=true stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
was:
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match login in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_S type=string indexed=true stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
Solr DataImportHandler not finding dynamic fields
-
Key: SOLR-6018
URL: https://issues.apache.org/jira/browse/SOLR-6018
Project: Solr
Issue Type: Bug
Components: contrib - DataImportHandler
Affects Versions: 4.7
Reporter: Aaron LaBella
Fix For: 4.9
Attachments: 0001-dih-config-check-for-dynamic-fields.patch
There is an issue with
*org.apache.solr.handler.dataimport.DocBuilder:addFields* around ~line 643.
The logic currently says see if you can find the field from the schema, ie:
{code:title=DocBuilder.java|borderStyle=solid}
SchemaField sf = schema.getFieldOrNull(key);
{code}
and, if not found, go ask DIHConfiguration to find it, ie:
{code:title=DocBuilder.java|borderStyle=solid}
sf = config.getSchemaField(key);
{code}
The latter call takes into account case-insensitivity, which is a big deal
since some databases, ie: DB2, upper case all the resulting column names. In
order to not modify solr-core (ie: the match logic in IndexSchema), I'm
attaching a patch that makes DIHConfiguration apply the same case-insensitive
logic to the DynamicFields.
Without this patch, dynamic fields will not be added to the index unless you
declare them like this:
{code:xml}
dynamicField name=*_S type=string indexed=true stored=true /
{code}
(note the capital S)
which is in-consistent with what I believe to be solr schema conventions to
have all the schema fields as lower-case.
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6013) Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility
[ https://issues.apache.org/jira/browse/SOLR-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-6013: Attachment: 0001-change-method-variable-visibility-and-refactor-for-extensibility.patch attaching the patch for review/comments. Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility -- Key: SOLR-6013 URL: https://issues.apache.org/jira/browse/SOLR-6013 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Fix For: 4.8 Attachments: 0001-change-method-variable-visibility-and-refactor-for-extensibility.patch Original Estimate: 1h Remaining Estimate: 1h This is similar to issue 5981, the Evaluator class is declared as abstract, yet the parseParams method is package private? Surely this is an oversight, as I wouldn't expect everyone writing their own evaluators to have to deal with parsing the parameters. Similarly, I needed to refactor DateFormatEvaluator because I need to do some custom date math/parsing and it wasn't written in a way that I can extend it. Please review/apply my attached patch to the next version of Solr, ie: 4.8 or 4.9 if I must wait. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6013) Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility
Aaron LaBella created SOLR-6013: --- Summary: Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility Key: SOLR-6013 URL: https://issues.apache.org/jira/browse/SOLR-6013 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Fix For: 4.8 Attachments: 0001-change-method-variable-visibility-and-refactor-for-extensibility.patch This is similar to issue 5981, the Evaluator class is declared as abstract, yet the parseParams method is package private? Surely this is an oversight, as I wouldn't expect everyone writing their own evaluators to have to deal with parsing the parameters. Similarly, I needed to refactor DateFormatEvaluator because I need to do some custom date math/parsing and it wasn't written in a way that I can extend it. Please review/apply my attached patch to the next version of Solr, ie: 4.8 or 4.9 if I must wait. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
[ https://issues.apache.org/jira/browse/SOLR-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980061#comment-13980061 ] Aaron LaBella commented on SOLR-5981: - I'm not seeing this fix in the git mirror of lucene-solr? I'm also wondering why it was moved from 4.8 into 4.9, I thought it was ready to go? Thanks. Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) --- Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Assignee: Shawn Heisey Priority: Minor Fix For: 4.9, 5.0 Attachments: SOLR-5981.patch Original Estimate: 1h Remaining Estimate: 1h I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6013) Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility
[ https://issues.apache.org/jira/browse/SOLR-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron LaBella updated SOLR-6013: Attachment: 0001-add-getters-for-datemathparser.patch one more small patch after fully testing the changes for extensibility Fix method visibility of Evaluator, refactor DateFormatEvaluator for extensibility -- Key: SOLR-6013 URL: https://issues.apache.org/jira/browse/SOLR-6013 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.7 Reporter: Aaron LaBella Fix For: 4.8 Attachments: 0001-add-getters-for-datemathparser.patch, 0001-change-method-variable-visibility-and-refactor-for-extensibility.patch Original Estimate: 1h Remaining Estimate: 1h This is similar to issue 5981, the Evaluator class is declared as abstract, yet the parseParams method is package private? Surely this is an oversight, as I wouldn't expect everyone writing their own evaluators to have to deal with parsing the parameters. Similarly, I needed to refactor DateFormatEvaluator because I need to do some custom date math/parsing and it wasn't written in a way that I can extend it. Please review/apply my attached patch to the next version of Solr, ie: 4.8 or 4.9 if I must wait. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
[ https://issues.apache.org/jira/browse/SOLR-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980508#comment-13980508 ] Aaron LaBella commented on SOLR-5981: - Shawn, No problem, makes sense ... when you get a chance, can you go ahead and commit the patch? Thanks! Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) --- Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Assignee: Shawn Heisey Priority: Minor Fix For: 4.9, 5.0 Attachments: SOLR-5981.patch Original Estimate: 1h Remaining Estimate: 1h I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
[ https://issues.apache.org/jira/browse/SOLR-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967816#comment-13967816 ] Aaron LaBella commented on SOLR-5981: - Erick, Thanks -- will do. I probably would've done that but my SVN skills aren't that great. I accidentally built from trunk first, and then realized I should've built against a branch. Then, I tried to run git svn clone ... but that seemed to take forever as well. Just curious -- are there any plans to migrate lucene/solr to a git repository? +1 for git from me ;-) Thanks. Aaron Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) --- Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Assignee: Shawn Heisey Priority: Minor Fix For: 4.8, 5.0 Attachments: SOLR-5981.patch Original Estimate: 1h Remaining Estimate: 1h I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
[ https://issues.apache.org/jira/browse/SOLR-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967471#comment-13967471 ] Aaron LaBella commented on SOLR-5981: - @Shawn, Thanks! That was super fast. The patch looks good to me. And, for what it's worth, I had actually done a test before posting this issue in the svn trunk by changing the getSolrWriter method scope to public and it worked just fine, so, I'm sure changing everything to protected will be equally as helpful. Thanks again, I think this is a great change to extend the usage of DataImportHandler. I vote yes to the patch. Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) --- Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Improvement Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Assignee: Shawn Heisey Priority: Minor Fix For: 4.8, 5.0 Attachments: SOLR-5981.patch Original Estimate: 1h Remaining Estimate: 1h I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5981) Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected)
Aaron LaBella created SOLR-5981: --- Summary: Please change method visibility of getSolrWriter in DataImportHandler to public (or at least protected) Key: SOLR-5981 URL: https://issues.apache.org/jira/browse/SOLR-5981 Project: Solr Issue Type: Bug Components: contrib - DataImportHandler Affects Versions: 4.0 Environment: Linux 3.13.9-200.fc20.x86_64 Solr 4.6.0 Reporter: Aaron LaBella Priority: Blocker Fix For: 4.6 I've been using the org.apache.solr.handler.dataimport.DataImportHandler for a bit and it's an excellent model and architecture. I'd like to extend the usage of it to plugin my own DIHWriter, but, the code doesn't allow for it. Please change ~line 227 in the DataImportHander class to be: public SolrWriter getSolrWriter instead of: private SolrWriter getSolrWriter or, at a minimum, protected, so that I can extend DataImportHandler and override this method. Thank you *sincerely* in advance for the quick turn-around on this. If the change can be made in 4.6.0 and upstream, that'd be ideal. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
