nickva commented on issue #504: Opimize writing KV node append writes URL: https://github.com/apache/couchdb/pull/504#issuecomment-299491424 Another set of tests ran with this version of the script: https://gist.github.com/nickva/2304bc2f738769c15fac116b8453ec04 ``` ./scripts/viewsize.py -n 1000 -s 10 -s 10000 -q true -q false -x true -x false -m 'function(d){emit(1,1);}' -m 'function(d){emit("K"+d._id, d.v);}' -r "" -r "_count" PATCH MASTER --------------- --------------- file dtavg file dtavg 2367619 0.026 2424963 0.024 2.4% size=10,query=True,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun= 2560141 0.008 2392205 0.008 -7.0% size=10,query=True,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun=_count 2592899 0.027 2564227 0.021 -1.1% size=10,query=True,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 2601101 0.009 2564237 0.009 -1.4% size=10,query=True,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count 577667 0.012 598147 0.020 3.4% size=10,query=True,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun= 581773 0.007 598157 0.008 2.7% size=10,query=True,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun=_count 880771 0.012 917635 0.013 4.0% size=10,query=True,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 880781 0.007 921741 0.007 4.4% size=10,query=True,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count 180355 0.157 221315 0.168 18.5% size=10,query=False,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun= 229517 0.131 233613 0.139 1.8% size=10,query=False,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun=_count 299139 0.202 245891 0.182 -21.7% size=10,query=False,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 307341 0.154 307341 0.159 0.0% size=10,query=False,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count 16515 0.115 16515 0.133 0.0% size=10,query=False,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun= 16525 0.103 16525 0.108 0.0% size=10,query=False,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun=_count 32899 0.119 32899 0.115 0.0% size=10,query=False,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 32909 0.104 32909 0.105 0.0% size=10,query=False,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count 2371715 0.022 2400387 0.024 1.2% size=10000,query=True,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun= 2543757 0.009 2568333 0.009 1.0% size=10000,query=True,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun=_count 28131459 0.098 28356739 0.083 0.8% size=10000,query=True,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 28229773 0.013 28471437 0.014 0.8% size=10000,query=True,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count 589955 0.021 598147 0.019 1.4% size=10000,query=True,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun= 585869 0.008 602253 0.008 2.7% size=10000,query=True,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun=_count 12251267 0.109 27328643 0.078 55.2% size=10000,query=True,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 12218509 0.011 27299981 0.012 55.2% size=10000,query=True,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count 270467 0.649 270467 0.721 0.0% size=10000,query=False,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun= 266381 0.616 270477 0.612 1.5% size=10000,query=False,random_keys=True,map_fun=function(d){emit(1,1);},reduce_fun=_count 18886787 1.366 18673795 1.364 -1.1% size=10000,query=False,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 18854029 1.221 18800781 1.211 -0.3% size=10000,query=False,random_keys=True,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count 16515 0.587 16515 0.691 0.0% size=10000,query=False,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun= 16525 0.601 16525 0.579 0.0% size=10000,query=False,random_keys=False,map_fun=function(d){emit(1,1);},reduce_fun=_count 7590019 0.896 7757955 0.935 2.2% size=10000,query=False,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun= 7594125 0.795 7757965 0.774 2.1% size=10000,query=False,random_keys=False,map_fun=function(d){emit("K"+d._id, d.v);},reduce_fun=_count ``` The observation is the same as above. Massive improvement in the case of large emit sizes and sequential keys. Small improvements in other cases where keys are not random. Biggest regression was in the case of: ` -21.7% size=10,query=False,random_keys=True` That is small emit values, querying the view once at the end of all the updates and using random keys. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
With regards, Apache Git Services
