[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734555#comment-16734555 ] Lucene/Solr QA commented on LUCENE-8601: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} LUCENE-8601 does not apply to master. Rebase required? Wrong Branch? See https://wiki.apache.org/lucene-java/HowToContribute#Contributing_your_work for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | LUCENE-8601 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12953747/7x_LUCENE-8601.06.patch | | Console output | https://builds.apache.org/job/PreCommit-LUCENE-Build/147/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Fix For: master (8.0), 7.7 > > Attachments: 7x_LUCENE-8601.06.patch, LUCENE-8601.01.patch, > LUCENE-8601.02.patch, LUCENE-8601.03.patch, LUCENE-8601.04.patch, > LUCENE-8601.05.patch, LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734391#comment-16734391 ] ASF subversion and git services commented on LUCENE-8601: - Commit e8bfa67716368ff3ad670a63d4a8935c6a989cac in lucene-solr's branch refs/heads/branch_7x from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e8bfa67 ] LUCENE-8601: attributes added to IndexableFieldType during indexing will now be preserved in the index and accessible at search time via FieldInfo attributes > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: 7x_LUCENE-8601.06.patch, LUCENE-8601.01.patch, > LUCENE-8601.02.patch, LUCENE-8601.03.patch, LUCENE-8601.04.patch, > LUCENE-8601.05.patch, LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734103#comment-16734103 ] Michael McCandless commented on LUCENE-8601: Thanks [~muralikpbhat] – I'll review and push to 7.x! > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: 7x_LUCENE-8601.06.patch, LUCENE-8601.01.patch, > LUCENE-8601.02.patch, LUCENE-8601.03.patch, LUCENE-8601.04.patch, > LUCENE-8601.05.patch, LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734046#comment-16734046 ] Murali Krishna P commented on LUCENE-8601: -- Thanks [~mikemccand]. Uploaded a separate patch for 7x branch [^7x_LUCENE-8601.06.patch] Regarding modifying the attribute, will open another issue as you suggested. We need to discuss how we should treat the scenarios if some attributes don't present in the subsequent update. Most of the use cases will have the attributes defined at the beginning itself and unlikely to change. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: 7x_LUCENE-8601.06.patch, LUCENE-8601.01.patch, > LUCENE-8601.02.patch, LUCENE-8601.03.patch, LUCENE-8601.04.patch, > LUCENE-8601.05.patch, LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733641#comment-16733641 ] Michael McCandless commented on LUCENE-8601: Hi [~muralikpbhat], I pushed the change to master, thanks! But the {{git cherry-pick}} back to 7.x was not clean – could you fixup the patch to apply to 7.x as well? Also, the test case uses FieldInfos API that was never back-ported to 7.x ({{getMergedFieldInfos}}). Also, staring at the code shortly after I pushed I noticed that the field type's attributes will be saved into FieldInfo the first time that field is seen for a given segment, but subsequent times it looks like we will fail to copy the attributes again? Can you also add a test case exposing this bug, and then fixing it? We can do that on a follow-on issue ... thanks! > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733635#comment-16733635 ] ASF subversion and git services commented on LUCENE-8601: - Commit 63dfba4c7d81a019a4008777beace0d391987ceb in lucene-solr's branch refs/heads/master from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=63dfba4 ] LUCENE-8601: attributes added to IndexableFieldType during indexing will now be preserved in the index and accessible at search time via FieldInfo attributes > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731696#comment-16731696 ] Michael McCandless commented on LUCENE-8601: Thanks I will review and push soon! > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731626#comment-16731626 ] Lucene/Solr QA commented on LUCENE-8601: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 43s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 47s{color} | {color:green} core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 46m 52s{color} | {color:green} core in the patch passed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 72m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | LUCENE-8601 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12953408/LUCENE-8601.06.patch | | Optional Tests | compile javac unit ratsources checkforbiddenapis validatesourcepatterns | | uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | ant | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh | | git revision | master / 752989f | | ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 | | Default Java | 1.8.0_191 | | Test Results | https://builds.apache.org/job/PreCommit-LUCENE-Build/146/testReport/ | | modules | C: lucene/core solr/core U: . | | Console output | https://builds.apache.org/job/PreCommit-LUCENE-Build/146/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731518#comment-16731518 ] Murali Krishna P commented on LUCENE-8601: -- Thanks, Added java docs for ignoreCurrentFormat, changed to 'if' from ternary, removed null initialization for class members and referenced attributes in DefaultIndexingChain in the new [^LUCENE-8601.06.patch]. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.06.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731288#comment-16731288 ] Michael McCandless commented on LUCENE-8601: Ahh OK thanks [~muralikpbhat]; that makes sense, so let's leave the assertion out. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730866#comment-16730866 ] Murali Krishna P commented on LUCENE-8601: -- Thanks Mike, regarding the assertion, i indeed first attempted to assert for equals, but the unit tests fail. The segment suffix supposed to change after merge and also there is backward compatibility test for merging from an old format where you have to write a new format. So, the assertion did not make sense and hence removed it. If you agree, I will make other changes in the next patch. o > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730210#comment-16730210 ] Michael McCandless commented on LUCENE-8601: Thanks [~muralikpbhat]! Maybe add javadocs about {{ignoreCurrentFormat}} parameter? Can you use multi-line {{if}} statement instead of ternary operator? I think the changes to {{PerFieldPostingsFormat}} are OK, except instead of removing the check that there was no format there and blindly overwriting it, can you change that to check that either it wasn't there (what it checks now) or, if it is there, that the value for the attributes match what that postings format wants to write? No need to initialize class members with {{= null}}; that's already the default for java. In {{DefaultIndexingChain}} can you use a local variable for the {{fieldType.getAttributes()}} in the two places where you reference it? > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730028#comment-16730028 ] Murali Krishna P commented on LUCENE-8601: -- [^LUCENE-8601.05.patch] which retains the attributes during the merge and modifies the PerField formats ignore them, has passed the Lucene/Solr QA above. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729119#comment-16729119 ] Lucene/Solr QA commented on LUCENE-8601: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 7s{color} | {color:green} core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 41m 0s{color} | {color:green} core in the patch passed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 14s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | LUCENE-8601 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12953073/LUCENE-8601.05.patch | | Optional Tests | compile javac unit ratsources checkforbiddenapis validatesourcepatterns | | uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | ant | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh | | git revision | master / 106d300 | | ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 | | Default Java | 1.8.0_191 | | Test Results | https://builds.apache.org/job/PreCommit-LUCENE-Build/142/testReport/ | | modules | C: lucene/core solr/core U: . | | Console output | https://builds.apache.org/job/PreCommit-LUCENE-Build/142/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.05.patch, > LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728862#comment-16728862 ] Murali Krishna P commented on LUCENE-8601: -- You are right, [~mikemccand]. The FieldInfo has an existing bug where the attributes are lost during merging. However, fixing it seems tricky as the existing classes relies on that bug (or is it by design?). PerField formats uses the attributes for storing format and suffix and they expect it to be null and generate new values during merge. Eventhough the attributes are mainly useful during index time, it might be good to persist it even after merge. I will make an attempt to change these PerField formats to overwrite the attributes, but these has explicit asserts right now to ensure the attributes are null and I am concerned I might be breaking something. Please let me know if you see any concerns with that approach. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725297#comment-16725297 ] Michael McCandless commented on LUCENE-8601: Hmm, I'm concerned that segment merging may not preserve the attributes. [~muralikpbhat] could you please add a test case that forces merging? E.g. index one document with attributes, commit (so it writes a segment), index another without attributes, commit, and confirm attributes survived? Can you also update the javadocs to state that if you try to index conflicting attributes the behavior is undefined (i.e. which attribute wins is undefined). > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725251#comment-16725251 ] Michael McCandless commented on LUCENE-8601: Thanks [~muralikpbhat]; new patch looks good ... I'll push shortly. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724142#comment-16724142 ] Lucene/Solr QA commented on LUCENE-8601: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 55s{color} | {color:green} core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 57s{color} | {color:green} core in the patch passed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 61m 5s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | LUCENE-8601 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12952123/LUCENE-8601.04.patch | | Optional Tests | compile javac unit ratsources checkforbiddenapis validatesourcepatterns | | uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | ant | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh | | git revision | master / 3be5b59 | | ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 | | Default Java | 1.8.0_191 | | Test Results | https://builds.apache.org/job/PreCommit-LUCENE-Build/140/testReport/ | | modules | C: lucene/core solr/core U: . | | Console output | https://builds.apache.org/job/PreCommit-LUCENE-Build/140/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723651#comment-16723651 ] Murali Krishna P commented on LUCENE-8601: -- Good catch [~jpountz], fixed it and modified the test: [^LUCENE-8601.04.patch] > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723036#comment-16723036 ] Adrien Grand commented on LUCENE-8601: -- Should the copy constructor of FieldType clone the attributes map? > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723009#comment-16723009 ] Michael McCandless commented on LUCENE-8601: Thanks [~muralikpbhat]; new patch looks great. I'll push soon. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722961#comment-16722961 ] Murali Krishna P commented on LUCENE-8601: -- Done that change [^LUCENE-8601.03.patch], thank you! > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.03.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722893#comment-16722893 ] Michael McCandless commented on LUCENE-8601: Thanks [~muralikpbhat]; can we make the {{FieldType}} attributes be {{null}} to start, and lazily create the {{HashMap}} the first time someone calls {{setAttribute}}? This way people who never use attributes won't pay any added cost (except the {{null}} check). > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722670#comment-16722670 ] Murali Krishna P commented on LUCENE-8601: -- Thanks [~mikemccand] for reviewing. Added the following changes. ([^LUCENE-8601.02.patch]) # Changed semantics to allow null # SchemaField returns null now. # Added @{{lucene.experimental}} to FieldType.putAttributes(), the IndexFieldType class itself was experimental already. # Added the java-docs for thread safety. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, > LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721504#comment-16721504 ] Michael McCandless commented on LUCENE-8601: Patch looks great [~muralikpbhat]; impressive how tiny the change is! For Solr's {{SchemaField}} can you instead return a {{Collections.emptyMap}}, maybe statically initialized/stored somewhere? Can we change the semantics to allow a {{null}} value to mean there are no attributes? I'm a little worried about having users who do not set any attributes having to pay the (tiny) cost of creating empty {{HashMap}} and iterating over it. Can you also add {{@lucene.experimental}} in the javadocs for the new methods, since this is a new API we should reserve the right to possibly change it again soon. Can you also explain in the javadocs that there is no thread safety here, i.e. user must not add attributes while other threads are indexing documents with that field type. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719896#comment-16719896 ] Murali Krishna P commented on LUCENE-8601: -- Lucene precommit Jenkins seems to be waiting for executor for long and not scheduling new jobs(including my second patch):[https://builds.apache.org/job/PreCommit-LUCENE-Build/] > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718913#comment-16718913 ] Murali Krishna P commented on LUCENE-8601: -- Fixed the failing test and solr. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.01.patch, LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType
[ https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716998#comment-16716998 ] Lucene/Solr QA commented on LUCENE-8601: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 30s{color} | {color:red} core in the patch failed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 17m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | lucene.index.TestFieldInfos | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | LUCENE-8601 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12951357/LUCENE-8601.patch | | Optional Tests | compile javac unit ratsources checkforbiddenapis validatesourcepatterns | | uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | ant | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh | | git revision | master / 40502b9 | | ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/PreCommit-LUCENE-Build/135/artifact/out/patch-unit-lucene_core.txt | | Test Results | https://builds.apache.org/job/PreCommit-LUCENE-Build/135/testReport/ | | modules | C: lucene/core U: lucene/core | | Console output | https://builds.apache.org/job/PreCommit-LUCENE-Build/135/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Adding attributes to IndexFieldType > --- > > Key: LUCENE-8601 > URL: https://issues.apache.org/jira/browse/LUCENE-8601 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: 7.5 >Reporter: Murali Krishna P >Priority: Major > Attachments: LUCENE-8601.patch > > > Today, we can write a custom Field using custom IndexFieldType, but when the > DefaultIndexingChain converts [IndexFieldType to > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662], > only few key informations such as indexing options and doc value type are > retained. The [Codec gets the > FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90], > but not the type details. > > FieldInfo has support for ['attributes'| > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47] > and it would be great if we can add 'attributes' to IndexFieldType also and > copy it to FieldInfo's 'attribute'. > > This would allow someone to write a custom codec (extending docvalueformat > for example) for only the 'special field' that he wants and delegate the rest > of the fields to the default codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org