This ticket shows how it is done in the solrconfig.xml: https://issues.apache.org/jira/browse/SOLR-9103
Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Mar 8, 2021 at 9:18 AM ufuk yılmaz <[email protected]> wrote: > How do you “register” something like a CloudSolrStream btw? Using Blob > Store API? > > Sent from Mail for Windows 10 > > From: Susmit > Sent: 06 March 2021 23:03 > To: [email protected] > Subject: Re: Idle timeout expired and Early Client Disconnect errors > > better to use solr 8.9 and configure http timeouts from solr.in.sh > workaround is bigger - need to extend cloudsolrstream , register it and > install custom solrclientcache with overridden setcontext method > > Sent from my iPhone > > > On Mar 6, 2021, at 9:25 AM, ufuk yılmaz <[email protected]> > wrote: > > > > How? O_O > > > > Sent from Mail for Windows 10 > > > > From: Susmit > > Sent: 06 March 2021 18:35 > > To: [email protected] > > Subject: Re: Idle timeout expired and Early Client Disconnect errors > > > > i have used a workaround to increase the default (hard coded) timeout of > 2 min in solrclientcache. > > i can run 9+ hour long streaming queries with no issues. > > > > Sent from my iPhone > > > >> On Mar 2, 2021, at 5:32 PM, ufuk yılmaz <[email protected]> > wrote: > >> > >> I divided the query to 1000 pieces and removed the parallel stream > clause, it seems to be working without timeout so far, if it does I just > can divide it to even smaller pieces I guess. > >> > >> I tried to send all 1000 pieces in a “list” expression to be executed > linearly, it didn’t work but I was just curious if it could handle such a > large query 😃 > >> > >> Now I’m just generating expression strings from java code and sending > them one by one. I tried to use SolrJ for this, but encountered a weird > problem where even the simplest expression (echo) stops working after a few > iterations in a loop. I’m guessing the underlying HttpClient is not closing > connections timely, hitting the OS per-host connection limit. I asked a > separate question about this. I was following the example on lucidworks: > https://lucidworks.com/post/streaming-expressions-in-solrj/ > >> > >> I just modified my code to use regular REST calls using okhttp3, it’s a > shame that I couldn’t use SolrJ since it truly streams every result 1 by 1 > continuously. REST just returns a single large response at the very end of > the stream. > >> > >> Thanks again for your help. > >> > >> Sent from Mail for Windows 10 > >> > >> From: Joel Bernstein > >> Sent: 02 March 2021 00:19 > >> To: [email protected] > >> Subject: Re: Idle timeout expired and Early Client Disconnect errors > >> > >> Also the parallel function builds hash partitioning filters that could > lead > >> to timeouts if they take too long to build. Try the query without the > >> parallel function if you're still getting timeouts when making the query > >> smaller. > >> > >> > >> > >> Joel Bernstein > >> http://joelsolr.blogspot.com/ > >> > >> > >>>> On Mon, Mar 1, 2021 at 4:03 PM Joel Bernstein <[email protected]> > wrote: > >>> > >>> The settings in your version are 30 seconds and 15 seconds for socket > and > >>> connection timeouts. > >>> > >>> Typically timeouts occur because one or more shards in the query are > idle > >>> beyond the timeout threshold. This happens because lot's of data is > being > >>> read from other shards. > >>> > >>> Breaking the query into small parts would be a good strategy. > >>> > >>> > >>> > >>> > >>> Joel Bernstein > >>> http://joelsolr.blogspot.com/ > >>> > >>> > >>> On Mon, Mar 1, 2021 at 3:30 PM ufuk yılmaz <[email protected] > > > >>> wrote: > >>> > >>>> Hello Mr. Bernstein, > >>>> > >>>> I’m using version 8.4. So, if I understand correctly, I can’t increase > >>>> timeouts and they are bound to happen in such a large stream. Should > I just > >>>> reduce the output of my search expressions? > >>>> > >>>> Maybe I can split my search results into ~100 parts and run the same > >>>> query 100 times in series. Each part would emit ~3M documents so they > >>>> should finish before timeout? > >>>> > >>>> Is this a reasonable solution? > >>>> > >>>> Btw how long is the default hard-coded timeout value? Because > yesterday I > >>>> ran another query which took more than 1 hour without any timeouts and > >>>> finished successfully. > >>>> > >>>> Sent from Mail for Windows 10 > >>>> > >>>> From: Joel Bernstein > >>>> Sent: 01 March 2021 23:03 > >>>> To: [email protected] > >>>> Subject: Re: Idle timeout expired and Early Client Disconnect errors > >>>> > >>>> Oh wait, I misread your email. The idle timeout issue is configurable > in: > >>>> > >>>> https://issues.apache.org/jira/browse/SOLR-14672 > >>>> > >>>> This unfortunately missed the 8.8 release and will be 8.9. > >>>> > >>>> > >>>> > >>>> This i > >>>> > >>>> > >>>> > >>>> Joel Bernstein > >>>> http://joelsolr.blogspot.com/ > >>>> > >>>> > >>>>> On Mon, Mar 1, 2021 at 2:56 PM Joel Bernstein <[email protected]> > wrote: > >>>> > >>>>> What version are you using? > >>>>> > >>>>> Solr 8.7 has changes that caused these errors to hit the logs. These > >>>> used > >>>>> to be suppressed. This has been fixed in Solr 9.0 but it has not been > >>>> back > >>>>> ported to Solr 8.x. > >>>>> > >>>>> The errors are actually normal operational occurrences when doing > joins > >>>> so > >>>>> should be suppressed in the logs and were before the specific > release. > >>>>> > >>>>> It might make sense to do a release that specifically suppresses > these > >>>>> errors without backporting the full Solr 9.0 changes which impact the > >>>>> memory footprint of export. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Joel Bernstein > >>>>> http://joelsolr.blogspot.com/ > >>>>> > >>>>> > >>>>> On Mon, Mar 1, 2021 at 10:29 AM ufuk yılmaz > <[email protected] > >>>>> > >>>>> wrote: > >>>>> > >>>>>> Hello all, > >>>>>> > >>>>>> I’m running a large streaming expression and feeding the result to > >>>> update > >>>>>> expression. > >>>>>> > >>>>>> update(targetCollection, ...long running stream here..., > >>>>>> > >>>>>> I tried sending the exact same query multiple times, it sometimes > works > >>>>>> and indexes some results, then gives exception, other times fails > with > >>>> an > >>>>>> exception after 2 minutes. > >>>>>> > >>>>>> Response is like: > >>>>>> "EXCEPTION":"java.util.concurrent.ExecutionException: > >>>>>> java.io.IOException: params distrib=false&numWorkers=4.... and my > long > >>>>>> stream expression > >>>>>> > >>>>>> Server log (short): > >>>>>> [c:DNM s:shard1 r:core_node2 x:DNM_shard1_replica_n1] > >>>>>> o.a.s.s.HttpSolrCall null:java.io.IOException: > >>>>>> java.util.concurrent.TimeoutException: Idle timeout expired: > >>>> 120000/120000 > >>>>>> ms > >>>>>> o.a.s.s.HttpSolrCall null:java.io.IOException: > >>>>>> java.util.concurrent.TimeoutException: Idle timeout expired: > >>>> 120000/120000 > >>>>>> ms > >>>>>> > >>>>>> I tried to increase the jetty idle timeout value on the node which > >>>> hosts > >>>>>> my target collection to something like an hour. It didn’t affect. > >>>>>> > >>>>>> > >>>>>> Server logs (long) > >>>>>> ERROR (qtp832292933-589) [c:DNM s:shard1 r:core_node2 > >>>>>> x:DNM_shard1_replica_n1] o.a.s.s.HttpSolrCall > null:java.io.IOException: > >>>>>> java.util.concurrent.TimeoutException: Idle timeout expired: 1 > >>>>>> 20000/120000 ms > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:235) > >>>>>> solr-01 | at > >>>>>> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:226) > >>>>>> solr-01 | at > >>>>>> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:524) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.apache.solr.servlet.ServletOutputStreamWrapper.write(ServletOutputStreamWrapper.java:134) > >>>>>> solr-01 | at > >>>>>> > java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > >>>>>> solr-01 | at > >>>>>> java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) > >>>>>> solr-01 | at > >>>>>> java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) > >>>>>> solr-01 | at > >>>>>> java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) > >>>>>> solr-01 | at java.base/java.io > >>>>>> .OutputStreamWriter.write(OutputStreamWriter.java:211) > >>>>>> solr-01 | at > >>>>>> org.apache.solr.common.util.FastWriter.flush(FastWriter.java:140) > >>>>>> solr-01 | at > >>>>>> org.apache.solr.common.util.FastWriter.write(FastWriter.java:54) > >>>>>> solr-01 | at > >>>>>> org.apache.solr.response.JSONWriter._writeChar(JSONWriter.java:173) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.apache.solr.common.util.JsonTextWriter.writeStr(JsonTextWriter.java:86) > >>>>>> solr-01 | at > >>>>>> org.apache.solr.common.util.TextWriter.writeVal(TextWriter.java:52) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:152) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.apache.solr.common.util.JsonTextWriter$2.put(JsonTextWriter.java:176) > >>>>>> solr-01 | at > >>>>>> org.apache.solr.common.MapWriter$EntryWriter.put(MapWriter.java:154) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.apache.solr.handler.export.StringFieldWriter.write(StringFieldWriter.java:77) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.apache.solr.handler.export.ExportWriter.writeDoc(ExportWriter.java:313) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.apache.solr.handler.export.ExportWriter.lambda$addDocsToItemWriter$4(ExportWriter.java:263) > >>>>>> -- > >>>>>> solr-01 | at org.eclipse.jetty.io > >>>>>> .FillInterest.fillable(FillInterest.java:103) > >>>>>> solr-01 | at org.eclipse.jetty.io > >>>>>> .ChannelEndPoint$2.run(ChannelEndPoint.java:117) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) > >>>>>> solr-01 | at > >>>>>> > >>>> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) > >>>>>> solr-01 | at java.base/java.lang.Thread.run(Thread.java:834) > >>>>>> solr-01 | Caused by: java.util.concurrent.TimeoutException: Idle > >>>>>> timeout expired: 120000/120000 ms > >>>>>> solr-01 | at org.eclipse.jetty.io > >>>>>> .IdleTimeout.checkIdleTimeout(IdleTimeout.java:171) > >>>>>> solr-01 | at org.eclipse.jetty.io > >>>>>> .IdleTimeout.idleCheck(IdleTimeout.java:113) > >>>>>> solr-01 | at > >>>>>> > >>>> > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > >>>>>> solr-01 | at > >>>>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > >>>>>> solr-01 | at > >>>>>> > >>>> > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > >>>>>> solr-01 | at > >>>>>> > >>>> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > >>>>>> solr-01 | at > >>>>>> > >>>> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > >>>>>> solr-01 | ... 1 more > >>>>>> > >>>>>> > >>>>>> My expression, in case it helps. To summarize, it finds the document > >>>> ids > >>>>>> which exists on sourceCollection but not on target collection (DNM). > >>>> Joins > >>>>>> on itself to duplicate some fields (I couldn’t find another way to > >>>>>> duplicate the value of field into 2 fields). Then sends the result > to > >>>>>> update. Source collection has about 300M documents, 24GB heap, 2 > >>>> shards, 2 > >>>>>> replicas of each shard. > >>>>>> > >>>>>> update( > >>>>>> DNM, > >>>>>> batchSize=1000, > >>>>>> parallel( > >>>>>> WorkerCollection, > >>>>>> leftOuterJoin( > >>>>>> fetch( > >>>>>> sourceCollection, > >>>>>> complement( > >>>>>> search( > >>>>>> sourceCollection, > >>>>>> q="*:*", > >>>>>> qt="/export", > >>>>>> fq="...some filters...", > >>>>>> sort="id_str asc", > >>>>>> fl="id_str", > >>>>>> partitionKeys="id_str" > >>>>>> ), > >>>>>> search( > >>>>>> DNM, > >>>>>> q="*:*", > >>>>>> qt="/export", > >>>>>> sort="id_str asc", > >>>>>> fl="id_str", > >>>>>> partitionKeys="id_str" > >>>>>> ), > >>>>>> on="id_str" > >>>>>> ), > >>>>>> fl="...my many fields...", > >>>>>> on="id_str", > >>>>>> batchSize="1000" > >>>>>> ), > >>>>>> select( > >>>>>> fetch( > >>>>>> sourceCollection, > >>>>>> complement( > >>>>>> search( > >>>>>> sourceCollection, > >>>>>> q="*:*", > >>>>>> qt="/export", > >>>>>> fq="...some other filters...", > >>>>>> sort="id_str asc", > >>>>>> fl="id_str", > >>>>>> partitionKeys="id_str" > >>>>>> ), > >>>>>> search( > >>>>>> DNM, > >>>>>> q="*:*", > >>>>>> qt="/export", > >>>>>> sort="id_str asc", > >>>>>> fl="id_str", > >>>>>> partitionKeys="id_str" > >>>>>> ), > >>>>>> on="id_str" > >>>>>> ), > >>>>>> fl="...some other fields...", > >>>>>> on="id_str", > >>>>>> batchSize="1000" > >>>>>> ), > >>>>>> id_str, ..some other fields as... > >>>>>> ), > >>>>>> on="id_str" > >>>>>> ), > >>>>>> workers="4", sort="id_str asc" > >>>>>> ) > >>>>>> ) > >>>>>> > >>>>>> Sent from Mail for Windows 10 > >>>>>> > >>>>>> > >>>> > >>>> > >> > > > >
