[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2017-10-17 Thread nkeet shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208235#comment-16208235
 ] 

nkeet shah commented on LUCENE-7538:


Erick, we are not using SOLR purely as a search engine. we are using it 
slightly in a non-traditional manner, and our user case of 1GB file are very 
valid. Thanks Michael for the pointer.

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Fix For: 6.4, 7.0
>
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2017-10-12 Thread Erick Erickson (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202707#comment-16202707
 ] 

Erick Erickson commented on LUCENE-7538:


There are probably some a-priori limits, somewhere  someplace lots of data has 
to be kept. The document has to be indexed in memory-resident segments 
(essentially) before being flushed. I question whether it's reasonable to even 
try to index documents this large.

>From a user's perspective, what's the value in indexing such a huge document? 
>Assuming it's text, it's quite likely that it'll be found for almost every 
>search... and have such a low score that it won't be seen by the users. It 
>can't be displayed in a browser, it's too big. Even transmitting it to the 
>client is prohibitive.

So I'd ask a little different question: "What's the largest document it makes 
sense to index?". Personally I think it's far, far less than a gigabyte


> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Fix For: 6.4, 7.0
>
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2017-10-12 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202705#comment-16202705
 ] 

Michael McCandless commented on LUCENE-7538:


@nkeet see {{IndexWriter.MAX_STORED_STRING_LENGTH}} for the limit; you'll have 
to either not store that huge file, or break it up across several documents.

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Fix For: 6.4, 7.0
>
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2017-10-12 Thread nkeet (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202681#comment-16202681
 ] 

nkeet commented on LUCENE-7538:
---

Thanks for having this fixed. I have been using SOLR 6.1.0 for just about a 
year now, and i recently stated seeing this issue. On ivestigation, i found 
that, at times i have files which are 1GB huge. Now i do unsterstand that this 
fix will prevent the IndexWriter from closing. but how do you index such a huge 
file, are we saying that there is a limit to the file size that can be 
indexed?, if yes what is the limit?

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Fix For: 6.4, 7.0
>
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-10 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655055#comment-15655055
 ] 

Michael McCandless commented on LUCENE-7538:


Thank you for taking the time to report it [~steve_chen_veeva]!

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Fix For: master (7.0), 6.4
>
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-10 Thread Steve Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654658#comment-15654658
 ] 

Steve Chen commented on LUCENE-7538:


[~mikemccand] Thank you for getting this fixed so quickly!

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Fix For: master (7.0), 6.4
>
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-10 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653745#comment-15653745
 ] 

ASF subversion and git services commented on LUCENE-7538:
-

Commit be47009ce765f75661f3eda4878b4bb14a9688a1 in lucene-solr's branch 
refs/heads/branch_6x from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=be47009 ]

LUCENE-7538: throw IllegalArgumentException if you attempt to store a 
too-massive text field


> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-10 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653739#comment-15653739
 ] 

ASF subversion and git services commented on LUCENE-7538:
-

Commit 2902727a1570544869271cf177ed299fdef6863f in lucene-solr's branch 
refs/heads/master from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2902727 ]

LUCENE-7538: throw IllegalArgumentException if you attempt to store a 
too-massive text field


> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Attachments: LUCENE-7538.patch, LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-04 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637193#comment-15637193
 ] 

Michael McCandless commented on LUCENE-7538:


bq.  I think the code piece is in DefaultIndexingChain::processField method.

Indeed, that's a great place ... I'll add the check there.

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Attachments: LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-04 Thread Steve Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637022#comment-15637022
 ] 

Steve Chen commented on LUCENE-7538:


[~mikemccand] I think the code piece is in DefaultIndexingChain::processField 
method.
{noformat}
// Add stored fields:
if (fieldType.stored()) {
  if (fp == null) {
fp = getOrAddField(fieldName, fieldType, false);
  }
  if (fieldType.stored()) {
try {
  storedFieldsWriter.writeField(fp.fieldInfo, field);
} catch (Throwable th) {
  throw AbortingException.wrap(th);
}
  }
{noformat}
It wraps up all the Throwable from writeField method call into an 
AbortingException.

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Attachments: LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
>

Re: [jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-04 Thread David Smiley 
Erick,

I'm working with multiple users/clients that have documents at this size
(though I don't know "Eric B").  So it happens, and sometimes the bigger
the document, the more genuinely relevant it tends to be (I've seen this).
So it's a challenge.  Sometimes big docs can be decomposed and then you
have to fend with collapsing/grouping/joining etc. but sometimes those
mechanisms won't quite work for your use-case, particularly if you're
*already* using them; although they can sometimes be layered (i.e. multiple
collapses).  But then there's perf issues, with, say, high cardinality
fields CollapseQParser.  Just an example.  So IMO we (Lucene/Solr hackers)
can't really dismiss the want/desire to index huge docs out of hand,
although I agree it's good to try and push-back on the requirement because
perhaps it can be avoided in some cases.

Cheers,
~ David

On Fri, Nov 4, 2016 at 11:34 AM Erick Erickson 
wrote:

I agree that more graceful handling is desirable, but I also have
to ask what _use_ indexing a single 800M document is.

If it's textual data it'll generate a hit very frequently, and will
likely be way down in the relevance list. If it's numerics, it'll
also likely be hit a lot. Will the users even see it? If a user did
find it, would they be able to display it? They'd have to wait
for the 800M doc to be transmitted just for starters whether
you get the real doc from Solr or some other system-of-record.

Where I'm going here is wondering whether it makes sense
in your application to have your own cutoff and not even send
it to be indexed. I've seen situations where it _is_ required
mind you, it's just a question worth asking.

Best,
Erick

On Fri, Nov 4, 2016 at 7:50 AM, Michael McCandless (JIRA)
 wrote:
>
> [
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636551#comment-15636551
]
>
> Michael McCandless commented on LUCENE-7538:
> 
>
> Unfortunately, yes, {{IndexWriter}} will still close itself if you try to
index a too-massive text field.
>
> All my patch does is catch the int overflow, but the resulting exception
isn't that much better :)
>
> {noformat}
>[junit4]   2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexWriter -Dtests.method=testMassiveField
-Dtests.seed=251DAB4FB5DAD334 -Dtests.locale=uk-UA
-Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
>[junit4] ERROR   37.4s | TestIndexWriter.testMassiveField <<<
>[junit4]> Throwable #1:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>[junit4]>at
__randomizedtesting.SeedInfo.seed([251DAB4FB5DAD334:E1314D20F8A334C3]:0)
>[junit4]>at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:740)
>[junit4]>at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:754)
>[junit4]>at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1558)
>[junit4]>at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
>[junit4]>at
org.apache.lucene.index.TestIndexWriter.testMassiveField(TestIndexWriter.java:2791)
>[junit4]>at java.lang.Thread.run(Thread.java:745)
>[junit4]> Caused by: java.lang.ArithmeticException: integer
overflow
>[junit4]>at java.lang.Math.multiplyExact(Math.java:867)
>[junit4]>at
org.apache.lucene.util.UnicodeUtil.maxUTF8Length(UnicodeUtil.java:618)
>[junit4]>at
org.apache.lucene.codecs.compressing.GrowableByteArrayDataOutput.writeString(GrowableByteArrayDataOutput.java:67)
>[junit4]>at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.writeField(CompressingStoredFieldsWriter.java:292)
>[junit4]>at
org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsWriter.writeField(AssertingStoredFieldsFormat.java:143)
>[junit4]>at
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:434)
>[junit4]>at
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373)
>[junit4]>at
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231)
>[junit4]>at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
>[junit4]>at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562)
>[junit4]>at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
>[junit4]>at
org.apache.lucene.index.TestIndexWriter.testMassiveField(TestIndexWriter.java:2784)
>[junit4]>... 36 more
> {noformat}
>
> I'll think about how to catch it more cleanly up front w/o closing the

Re: [jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-04 Thread Erick Erickson 
I agree that more graceful handling is desirable, but I also have
to ask what _use_ indexing a single 800M document is.

If it's textual data it'll generate a hit very frequently, and will
likely be way down in the relevance list. If it's numerics, it'll
also likely be hit a lot. Will the users even see it? If a user did
find it, would they be able to display it? They'd have to wait
for the 800M doc to be transmitted just for starters whether
you get the real doc from Solr or some other system-of-record.

Where I'm going here is wondering whether it makes sense
in your application to have your own cutoff and not even send
it to be indexed. I've seen situations where it _is_ required
mind you, it's just a question worth asking.

Best,
Erick

On Fri, Nov 4, 2016 at 7:50 AM, Michael McCandless (JIRA)
 wrote:
>
> [ 
> https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636551#comment-15636551
>  ]
>
> Michael McCandless commented on LUCENE-7538:
> 
>
> Unfortunately, yes, {{IndexWriter}} will still close itself if you try to 
> index a too-massive text field.
>
> All my patch does is catch the int overflow, but the resulting exception 
> isn't that much better :)
>
> {noformat}
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriter 
> -Dtests.method=testMassiveField -Dtests.seed=251DAB4FB5DAD334 
> -Dtests.locale=uk-UA -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true 
> -Dtests.file.encoding=UTF-8
>[junit4] ERROR   37.4s | TestIndexWriter.testMassiveField <<<
>[junit4]> Throwable #1: 
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>[junit4]>at 
> __randomizedtesting.SeedInfo.seed([251DAB4FB5DAD334:E1314D20F8A334C3]:0)
>[junit4]>at 
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:740)
>[junit4]>at 
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:754)
>[junit4]>at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1558)
>[junit4]>at 
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
>[junit4]>at 
> org.apache.lucene.index.TestIndexWriter.testMassiveField(TestIndexWriter.java:2791)
>[junit4]>at java.lang.Thread.run(Thread.java:745)
>[junit4]> Caused by: java.lang.ArithmeticException: integer overflow
>[junit4]>at java.lang.Math.multiplyExact(Math.java:867)
>[junit4]>at 
> org.apache.lucene.util.UnicodeUtil.maxUTF8Length(UnicodeUtil.java:618)
>[junit4]>at 
> org.apache.lucene.codecs.compressing.GrowableByteArrayDataOutput.writeString(GrowableByteArrayDataOutput.java:67)
>[junit4]>at 
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.writeField(CompressingStoredFieldsWriter.java:292)
>[junit4]>at 
> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsWriter.writeField(AssertingStoredFieldsFormat.java:143)
>[junit4]>at 
> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:434)
>[junit4]>at 
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373)
>[junit4]>at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231)
>[junit4]>at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
>[junit4]>at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562)
>[junit4]>at 
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
>[junit4]>at 
> org.apache.lucene.index.TestIndexWriter.testMassiveField(TestIndexWriter.java:2784)
>[junit4]>... 36 more
> {noformat}
>
> I'll think about how to catch it more cleanly up front w/o closing the 
> {{IndexWriter}}...
>
>> Uploading large text file to a field causes "this IndexWriter is closed" 
>> error
>> --
>>
>> Key: LUCENE-7538
>> URL: https://issues.apache.org/jira/browse/LUCENE-7538
>> Project: Lucene - Core
>>  Issue Type: Bug
>>Affects Versions: 5.5.1
>>Reporter: Steve Chen
>> Attachments: LUCENE-7538.patch
>>
>>
>> We have seen "this IndexWriter is closed" error after we tried to upload a 
>> large text file to a single Solr text field. The field definition in the 
>> schema.xml is:
>> {noformat}
>> > termVectors="true" termPositions="true" termOffsets="true"/>
>> {noformat}
>> After that, the IndexWriter remained closed and couldn't be recovered until 
>> we reloaded the Solr core.  The text file

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-04 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636551#comment-15636551
 ] 

Michael McCandless commented on LUCENE-7538:


Unfortunately, yes, {{IndexWriter}} will still close itself if you try to index 
a too-massive text field.

All my patch does is catch the int overflow, but the resulting exception isn't 
that much better :)

{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriter 
-Dtests.method=testMassiveField -Dtests.seed=251DAB4FB5DAD334 
-Dtests.locale=uk-UA -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   37.4s | TestIndexWriter.testMassiveField <<<
   [junit4]> Throwable #1: org.apache.lucene.store.AlreadyClosedException: 
this IndexWriter is closed
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([251DAB4FB5DAD334:E1314D20F8A334C3]:0)
   [junit4]>at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:740)
   [junit4]>at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:754)
   [junit4]>at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1558)
   [junit4]>at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]>at 
org.apache.lucene.index.TestIndexWriter.testMassiveField(TestIndexWriter.java:2791)
   [junit4]>at java.lang.Thread.run(Thread.java:745)
   [junit4]> Caused by: java.lang.ArithmeticException: integer overflow
   [junit4]>at java.lang.Math.multiplyExact(Math.java:867)
   [junit4]>at 
org.apache.lucene.util.UnicodeUtil.maxUTF8Length(UnicodeUtil.java:618)
   [junit4]>at 
org.apache.lucene.codecs.compressing.GrowableByteArrayDataOutput.writeString(GrowableByteArrayDataOutput.java:67)
   [junit4]>at 
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.writeField(CompressingStoredFieldsWriter.java:292)
   [junit4]>at 
org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsWriter.writeField(AssertingStoredFieldsFormat.java:143)
   [junit4]>at 
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:434)
   [junit4]>at 
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373)
   [junit4]>at 
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231)
   [junit4]>at 
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
   [junit4]>at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562)
   [junit4]>at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]>at 
org.apache.lucene.index.TestIndexWriter.testMassiveField(TestIndexWriter.java:2784)
   [junit4]>... 36 more
{noformat}

I'll think about how to catch it more cleanly up front w/o closing the 
{{IndexWriter}}...

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Attachments: LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
>

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-04 Thread Eric B (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636513#comment-15636513
 ] 

Eric B commented on LUCENE-7538:


[~mikemccand] thanks for the quick response on this.   

with this patch, will we continue to see the AlreadyClosedException after this 
issue is encountered?  Not sure if this is a separate issue or all part of this 
one, but is it expected that an attempt to index a too-large document is 
considered an "unrecoverable disaster"* such that subsequent requests to index 
different documents will be rejected with an {{AlreadyClosedException}}?  I'm 
pretty sure that's what we experience... that once we attempt an index a 
too-large document, subsequent requests to index different documents of 
"normal" size are rejected with the {{AlreadyClosedException}} error.  

At this point, we find the only way to recover is to reload the core.  For 
example..
{noformat}
curl 
http://localhost:19081/solr/admin/cores?action=UNLOAD=instance_7889_CONTENT
curl 
http://localhost:19081/solr/admin/cores?action=CREATE=instance_7889_CONTENT=instance_7889_CONTENT=instance_7889_CONTENT=shard1=contentConf=7889=true=false
{noformat}

\* I used the word "unrecoverable disaster" since that's how it's referred to 
in the code at {{org.apache.lucene.index.IndexWriter}}:
{code}
  // when unrecoverable disaster strikes, we populate this with the reason that 
we had to close IndexWriter
  volatile Throwable tragedy;
{code}

Thanks,

Eric



> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
> Attachments: LUCENE-7538.patch
>
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at

[jira] [Commented] (LUCENE-7538) Uploading large text file to a field causes "this IndexWriter is closed" error

2016-11-04 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635896#comment-15635896
 ] 

Michael McCandless commented on LUCENE-7538:


Thanks [~steve_chen_veeva]

I'll fix this logic to use {{Math.multiplyExact}} instead so we find out right 
away that the massive string is too large for us to convert to UTF8.

> Uploading large text file to a field causes "this IndexWriter is closed" error
> --
>
> Key: LUCENE-7538
> URL: https://issues.apache.org/jira/browse/LUCENE-7538
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.5.1
>Reporter: Steve Chen
>
> We have seen "this IndexWriter is closed" error after we tried to upload a 
> large text file to a single Solr text field. The field definition in the 
> schema.xml is:
> {noformat}
>  termVectors="true" termPositions="true" termOffsets="true"/>
> {noformat}
> After that, the IndexWriter remained closed and couldn't be recovered until 
> we reloaded the Solr core.  The text file had size of 800MB, containing only 
> numbers and English characters.
> Stack trace is shown below:
> {noformat}
> 2016-11-02 23:00:17,913 [http-nio-19082-exec-3] ERROR 
> org.apache.solr.handler.RequestHandlerBase - 
> org.apache.solr.common.SolrException: Exception writing document id 1487_0_1 
> to the index; possible analysis error.
> at 
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> at 
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> at 
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:651)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> at 
> veeva.ecm.common.interfaces.web.SolrDispatchOverride.doFilter(SolrDispatchOverride.java:43)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
>