peter-toth commented on issue #23731: [SPARK-26572][SQL] fix aggregate codegen result evaluation URL: https://github.com/apache/spark/pull/23731#issuecomment-462347519 @mgaido91 @maropu @cloud-fan @viirya I've just collected the results of full `TPCDSQueryBenchmark` runs on master vs. this PR and overall it doesn't seem to have that big impact if we force the evaluation in aggregate. But I will try to change the PR and fix broadcast join to force evaluation of non-deterministic expressions out of the loop. Here are the benchmark results if you are interested: |test case (values in ms)|master|master 2nd run|this PR|this PR 2nd run| |-|-|-|-|-| |total|27023411|26650436|27076386|27024466| |q1|76017|75679|78988|81349| |q10|90319|88032|93112|97605| |q10a-v2.7|93878|93220|95113|94199| |q11|187918|187985|198156|201343| |q11-v2.7|186384|183901|191372|195608| |q12|35709|34768|37109|39020| |q12-v2.7|33617|34629|35666|37382| |q13|76088|75420|79224|79844| |q14-v2.7|647991|593543|611725|638555| |q14a|793687|779841|786366|780845| |q14a-v2.7|835538|849404|845488|824285| |q14b|632994|659007|647556|628163| |q15|50407|48712|53039|52104| |q16|173310|176263|181887|185434| |q17|358234|349108|384112|380049| |q18|67506|67611|70547|71098| |q18a-v2.7|260611|263082|268121|260403| |q19|45170|45566|48270|48136| |q2|62042|63219|64080|65965| |q20|37064|36726|38922|38857| |q20-v2.7|35919|35109|37882|37615| |q21|30900|30886|31394|32062| |q22|134810|140961|139015|146047| |q22-v2.7|881799|666436|736824|786129| |q22a-v2.7|77062|76621|76412|77984| |q23a|551085|562690|570236|554450| |q23b|509188|529924|514952|504751| |q24-v2.7|232464|223424|235800|242875| |q24a|235519|239793|240399|239449| |q24b|242000|235056|240476|237701| |q25|365661|364580|375756|388874| |q26|50035|49711|52832|53557| |q27|54088|53121|55344|57900| |q27a-v2.7|156000|153863|156385|149929| |q28|253946|253869|264420|266887| |q29|364358|368899|371732|377960| |q3|32569|32863|34460|35264| |q30|86921|87179|90322|93566| |q31|195233|195026|204116|208575| |q32|70098|70355|72000|73724| |q33|113700|114023|117777|121381| |q34|48608|48859|50382|51431| |q34-v2.7|47455|46366|48917|49559| |q35|96340|93386|97375|99735| |q35-v2.7|92379|93076|95759|97744| |q35a-v2.7|98323|97102|98990|95888| |q36|52161|52314|53873|55724| |q36a-v2.7|56251|55359|55688|55844| |q37|71609|72788|74007|75574| |q38|113581|111851|115416|117293| |q39a|67328|70285|69553|67597| |q39b|67457|69747|70059|67390| |q4|645699|632785|630790|662483| |q40|92189|90362|94774|97688| |q41|3931|3906|4157|4170| |q42|32727|33828|33368|35238| |q43|41522|40911|43279|44498| |q44|118338|118557|122399|126860| |q45|42166|42729|44086|43529| |q46|57798|57073|59813|60527| |q47|92922|92823|94551|99835| |q47-v2.7|90674|90166|91696|96707| |q48|64766|65324|66017|69578| |q49|290310|306006|294090|303586| |q49-v2.7|288623|320468|294292|290968| |q5|242664|244383|258006|263323| |q50|177935|187770|193884|188241| |q51|180839|180103|184773|189253| |q51a-v2.7|1261774|1166285|1173536|1156590| |q52|32827|32561|33622|33896| |q53|42069|42203|43387|44729| |q54|178694|180335|185816|193587| |q55|32863|32091|33049|33913| |q56|113489|110395|116137|116632| |q57|77085|78120|80313|80765| |q57-v2.7|75062|75184|78971|78424| |q58|102682|104439|106576|109776| |q59|58396|57408|59314|62043| |q5a-v2.7|264955|257989|264370|266317| |q6|109696|107457|107418|109251| |q6-v2.7|114195|109684|113283|114997| |q60|115152|111163|116920|118019| |q61|88653|87815|88905|92676| |q62|45958|45002|45842|49475| |q63|42271|41499|42553|44231| |q64|375608|366708|371139|389530| |q64-v2.7|398085|387919|378419|381406| |q65|120201|121013|121007|124101| |q66|107631|110160|110194|116045| |q67|491712|496057|502941|498816| |q67a-v2.7|582984|582585|588929|561230| |q68|57644|57396|58574|62110| |q69|92364|86473|87804|92744| |q7|53092|54107|56050|56440| |q70|76027|76236|78249|80208| |q70a-v2.7|81529|81288|80193|79803| |q71|111032|118959|114687|115729| |q72|1263412|1215258|1204783|1194142| |q72-v2.7|1274719|1297230|1224349|1188723| |q73|50254|47880|48539|50459| |q74|169414|163799|163079|169219| |q74-v2.7|162417|163068|161341|164335| |q75|365739|366476|372510|377480| |q75-v2.7|352554|372481|374001|359940| |q76|114328|110486|114099|114004| |q77|181185|174504|177397|179485| |q77a-v2.7|191487|188109|196948|188439| |q78|381549|366170|382482|377300| |q78-v2.7|373872|381872|400429|363220| |q79|55042|53635|55588|55914| |q8|40407|39912|41745|42687| |q80|515092|512636|542615|522297| |q80a-v2.7|527537|532262|546034|529508| |q81|85570|85808|88579|86749| |q82|99038|96385|99921|100342| |q83|106249|99379|104401|105534| |q84|39353|37683|40165|39717| |q85|167298|167352|165008|169000| |q86|37422|37246|38432|39093| |q86a-v2.7|40440|40165|41697|41213| |q87|120091|117061|125508|123227| |q88|283596|293759|296313|300676| |q89|44557|44882|46236|46012| |q9|499645|497991|514793|525809| |q90|70330|74905|72998|74123| |q91|45903|46732|48843|48705| |q92|66107|66381|68388|67379| |q93|281893|293320|292280|295871| |q94|117957|117824|126470|124528| |q95|632772|591228|607336|608948| |q96|37997|38276|39832|39946| |q97|105541|104452|113225|108664| |q98|38516|38436|41049|41108| |q98-v2.7|37586|36939|39786|39280| |q99|50378|51491|53608|52571|
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
