Ugh... ok. I'll take care of that. -- C
> On Jan 30, 2022, at 11:15 AM, Daniel Clark <[email protected]> wrote: > > Thanks for the quick turnaround Charles! I compiled the latest snapshot > build this morning. I didn't see the allowDiskUse parameter listed in the > mongo storage plugin configuration, so I added it manually under the > "batchSize" parameter as following: > > "batchSize": 100, > "allowDiskUse": true, > "enabled": true > > Updating the configuration is successful, but if I go back to update the > mongo configuration again, I don't see the parameter listed. > > On Sat, Jan 29, 2022 at 6:46 PM Charles Givre <[email protected]> wrote: > >> Hi Daniel, >> I just submitted DRILL-8118 (https://github.com/apache/drill/issues/2440 < >> https://github.com/apache/drill/issues/2440>). Please take a look at let >> us know if this solves your issue. >> Best, >> -- C >> >>> On Jan 28, 2022, at 12:35 PM, Daniel Clark <[email protected]> wrote: >>> >>> Sounds good. Thank you Charles. >>> >>> On Fri, Jan 28, 2022 at 12:30 PM Charles Givre <[email protected] >> <mailto:[email protected]>> wrote: >>> >>>> Hey Daniel, >>>> I looked at the mongo docs, and I'm going to add a config option to add >>>> the "spill to disk" for mongo. I'll do it over the weekend and submit >> a PR >>>> to Drill. This is a very minor fix. >>>> Best, >>>> -- C >>>> >>>> >>>> >>>>> On Jan 28, 2022, at 12:28 PM, Daniel Clark <[email protected]> wrote: >>>>> >>>>> I've attached the json version of the profile. Is that what you are >>>> requesting? The latest run ended in another error. >>>>> >>>>> On Fri, Jan 28, 2022 at 9:47 AM luoc <[email protected] <mailto: >> [email protected]> <mailto: >>>> [email protected] <mailto:[email protected]>>> wrote: >>>>> >>>>> Hi Daniel, >>>>> The 1.20 is actually included a feature to improve performance using >>>> the new push down framework. >>>>> For describe the specific things, could you please post the profile >>>> file of query? >>>>> >>>>>> On Jan 28, 2022, at 22:32, Daniel Clark <[email protected] <mailto: >> [email protected]> <mailto: >>>> [email protected]>> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> While evaluating 1.20.0-SNAPSHOT release performance, I ran a mongo >>>> query that runs in 15 minutes in the 1.19 release (below). >>>>>> >>>>>> SELECT `Elements_Efforts`.`EffortTypeName` AS `EffortTypeName`, >>>>>> `Elements`.`ElementSubTypeName` AS `ElementSubTypeName`, >>>>>> `Elements`.`ElementTypeName` AS `ElementTypeName`, >>>>>> `Elements`.`PlanID` AS `PlanID` >>>>>> FROM `mongo.grounds`.`Elements` `Elements` >>>>>> INNER JOIN `mongo.grounds`.`Elements_Efforts` `Elements_Efforts` ON >>>> (`Elements`.`_id` = `Elements_Efforts`.`_id`) >>>>>> WHERE (`Elements`.`PlanID` = '1623263140') >>>>>> GROUP BY `Elements_Efforts`.`EffortTypeName`, >>>>>> `Elements`.`ElementSubTypeName`, >>>>>> `Elements`.`ElementTypeName`, >>>>>> `Elements`.`PlanID` >>>>>> >>>>>> The query runs for 34 minutes before returning this error; "Sort >>>> exceeded memory limit of 104857600 bytes, but did not opt in to external >>>> sorting. Aborting operation. Pass allowDiskUse:true to opt in.' on >> server >>>> localhost:27017." Any ideas? I realize that it's a mongodb error, but >> the >>>> mongo database doesn't raise this error with the 1.19 release. I was >>>> expecting improved performance with the mongo storage plugin in the >>>> upcoming 1.20 release. Nothing in my environment has changed. I've >> attached >>>> the full stacktrace. >>>>>> >>>>>> <stacktrace.txt> >>>>> <profile.json> >> >>
