Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Ah, I think I'd missed this message.  Thank you.  In case you didn't see, we've logged DRILL-8119 - MongoDB query regression: mem leak and data type error On 2022/01/28 19:28, Daniel Clark wrote: I've attached the json version of the

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

James is right however. It may end up that the query returns faster with Drill doing the sorting rather than Mongo. -- C > On Jan 30, 2022, at 11:15 AM, Daniel Clark wrote: > > Thanks for the quick turnaround Charles! I compiled the latest snapshot > build this morning. I didn't see the

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Ugh... ok. I'll take care of that. -- C > On Jan 30, 2022, at 11:15 AM, Daniel Clark wrote: > > Thanks for the quick turnaround Charles! I compiled the latest snapshot > build this morning. I didn't see the allowDiskUse parameter listed in the > mongo storage plugin configuration, so I

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Thanks for the quick turnaround Charles! I compiled the latest snapshot build this morning. I didn't see the allowDiskUse parameter listed in the mongo storage plugin configuration, so I added it manually under the "batchSize" parameter as following: "batchSize": 100, "allowDiskUse": true,

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

I think that the reported issue, that Drill 1.20 takes longer than the 15 minutes 1.19 needs for this query, won't be solved.  Instead of failing after 34 minutes, Mongo will carry on processing for what will probably be quite a lot longer.  The new option in DRILL-8118 is obviously great to

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Hi Daniel, I just submitted DRILL-8118 (https://github.com/apache/drill/issues/2440 ). Please take a look at let us know if this solves your issue. Best, -- C > On Jan 28, 2022, at 12:35 PM, Daniel Clark wrote: > > Sounds good. Thank you

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Sounds good. Thank you Charles. On Fri, Jan 28, 2022 at 12:30 PM Charles Givre wrote: > Hey Daniel, > I looked at the mongo docs, and I'm going to add a config option to add > the "spill to disk" for mongo. I'll do it over the weekend and submit a PR > to Drill. This is a very minor fix. >

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Hey Daniel, I looked at the mongo docs, and I'm going to add a config option to add the "spill to disk" for mongo. I'll do it over the weekend and submit a PR to Drill. This is a very minor fix. Best, -- C > On Jan 28, 2022, at 12:28 PM, Daniel Clark wrote: > > I've attached the json

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

I've attached the json version of the profile. Is that what you are requesting? The latest run ended in another error. On Fri, Jan 28, 2022 at 9:47 AM luoc wrote: > > Hi Daniel, > The 1.20 is actually included a feature to improve performance using the > new push down framework. > For

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Good question. I don't know enough about Mongo config to answer that, but let me look into that. Best, -- C > On Jan 28, 2022, at 10:20 AM, Daniel Clark wrote: > > Hi Charles, > > I was under the impression that the allowDiskUse parameter is passed by the > client making the call to the

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Hi Charles, I was under the impression that the allowDiskUse parameter is passed by the client making the call to the mongodb server. Is it possible to add this parameter to the mongo storage plugin, similar to how you added the "batchSize" parameter for the 1.20 release? On Fri, Jan 28, 2022 at

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

So it's faster _not_ to push the sorting work to mongo here, hehe. On 2022/01/28 16:47, luoc wrote:  Hi Daniel, The 1.20 is actually included a feature to improve performance using the new push down framework. For describe the specific things, could you please post the profile file of

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Daniel, Thanks for flagging this. One thing I noticed in your logs is this: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in. What's happening here is that in the newer version of Drill, Drill is

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

 Hi Daniel, The 1.20 is actually included a feature to improve performance using the new push down framework. For describe the specific things, could you please post the profile file of query? > On Jan 28, 2022, at 22:32, Daniel Clark wrote: >  > Hello, > > While evaluating

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Hi Charles, Yes "supportsSortPushdown" is set to true. I left it at the default. I'll try setting it to false, and try again. Thanks for the feedback. On Fri, Jan 28, 2022 at 9:38 AM Charles Givre wrote: > Hey Daniel, > Did you have the sort pushdown enabled? This is one change that we added

Re: 1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Hey Daniel, Did you have the sort pushdown enabled? This is one change that we added to the mongo pushdown since 1.19 and might be affecting your query. Best, -- C > On Jan 28, 2022, at 9:32 AM, Daniel Clark wrote: > > Hello, > > While evaluating 1.20.0-SNAPSHOT release performance, I

1.20.0-SNAPSHOT: Sort exceeded memory limit of 104857600 bytes

Hello, While evaluating 1.20.0-SNAPSHOT release performance, I ran a mongo query that runs in 15 minutes in the 1.19 release (below). SELECT `Elements_Efforts`.`EffortTypeName` AS `EffortTypeName`, `Elements`.`ElementSubTypeName` AS `ElementSubTypeName`, `Elements`.`ElementTypeName` AS