[
https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345951#comment-16345951
]
Rupa Shankar commented on SOLR-11277:
-------------------------------------
Thanks [~anshumg] for the feedback! Just uploaded a new patch incorporating
your suggestions.
As for the benchmarking, I wrote a quick script that has 5 threads concurrently
adding X documents to a collection. Here were the results:
*MASTER*
*5 threads, each adding 1000 docs*
Iteration #0: 5232 ms
Iteration #1: 4238 ms
Iteration #2: 3798 ms
Iteration #3: 3144 ms
Iteration #4: 2774 ms
Iteration #5: 2589 ms
Iteration #6: 2513 ms
Iteration #7: 2452 ms
Iteration #8: 2452 ms
Iteration #9: 2482 ms
AVG: 3167.4 ms
*5 threads, each adding 5000 docs*
Iteration #0: 12720 ms
Iteration #1: 12191 ms
Iteration #2: 12340 ms
Iteration #3: 12124 ms
Iteration #4: 12170 ms
Iteration #5: 12058 ms
Iteration #6: 11491 ms
Iteration #7: 12274 ms
Iteration #8: 12208 ms
Iteration #9: 12100 ms
AVG: 12167.6 ms
*HARD COMMIT FEATURE BRANCH*
*5 threads, each adding 1000 docs*
Iteration #0: 2758 ms
Iteration #1: 2576 ms
Iteration #2: 2440 ms
Iteration #3: 2416 ms
Iteration #4: 2443 ms
Iteration #5: 2406 ms
Iteration #6: 2419 ms
Iteration #7: 2430 ms
Iteration #8: 2421 ms
Iteration #9: 2448 ms
AVG: 2475.7 ms
*5 threads, each adding 5000 docs*
Iteration #0: 12683 ms
Iteration #1: 12520 ms
Iteration #2: 12264 ms
Iteration #3: 12223 ms
Iteration #4: 12148 ms
Iteration #5: 12161 ms
Iteration #6: 11810 ms
Iteration #7: 12398 ms
Iteration #8: 12168 ms
Iteration #9: 12522 ms
AVG: 12289.7 ms
Seems like the impact is minimal.
> Add auto hard commit setting based on tlog size
> -----------------------------------------------
>
> Key: SOLR-11277
> URL: https://issues.apache.org/jira/browse/SOLR-11277
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Rupa Shankar
> Assignee: Anshum Gupta
> Priority: Major
> Attachments: max_size_auto_commit.patch, max_size_auto_commit.patch
>
>
> When indexing documents of variable sizes and at variable schedules, it can
> be hard to estimate the optimal auto hard commit maxDocs or maxTime settings.
> We’ve had some occurrences of really huge tlogs, resulting in serious issues,
> so in an attempt to avoid this, it would be great to have a “maxSize” setting
> based on the tlog size on disk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
