FYI, providing the logs attached, the code already sends docs in 1K batch. On Fri, Dec 15, 2023 at 7:11 PM Dmitri Maziuk <dmitri.maz...@gmail.com> wrote:
> On 12/15/23 05:41, Vince McMahon wrote: > > Ishan, you are right. Doing multithreaded Indexing is going much faster. > > I found out after the remote machine became unresponsive very quickly ; > it > > crashed. lol. > FWIW I got better results posting docs in batches from a single thread. > Work is in a "private org" on gitlab so I can't post the link to the > code, but the basic layout is a DB reader that yields rows and a writer > that does requests.post() of a list of JSON docs. With the DB row -> > JSON doc transformer in-between. > > I played with the size of the batch as well as async/await queue before > leaving it single-threaded w/ batch size of 5K docs: I had no speed > advantage with larger batches in our setup. And it doesn't DDoS the > index. ;) > > Dima > > -- Sincerely yours Mikhail Khludnev
