Rich-T-kid commented on PR #23187: URL: https://github.com/apache/datafusion/pull/23187#issuecomment-4929038692
> Your optimization relies on the pointer being the same, generally, what are the odds of that? Im not sure of the exact number, in #21765 the e2e benchmarks showed a a large re-use in the values ptr because slicing a dictionary clones the value ptr. > I suggested earlier to count how many times .unwrap_or(null_sentinel_index) is used and if theres no null keys, skip the concat. Is that more or less likely? One way to find out is to run the benches and print out how many times each of those things happen, assuming there's enough interesting cases in the benches. in the benchmarks the null rate is set to 0% and 10%. in the case where [there are no nulls in the current batch](https://github.com/apache/datafusion/pull/23187/changes#diff-3a7e151acf3b161c9ffe712d8b834a397429508efd5710d0a2e6a1363397eabcR111) there is no concat operation. the alternative to using concat is to use a non-vectorized approach. Example ```rust // Null keys but values has no null entry: scalar fallback per row. for &row_index in rows { match dict.key(row_index) { None => self.inner.append_val(&self.null_array, 0)?, Some(key) => self.inner.append_val(dict.values(), key)?, } } ``` > Pointer checking is a bit of a smell. We used to do it in dynamic filtering to check if two filters are the same but everyone disliked that. We have since moved away from pointer equality checking. make sense, Ive removed the pointer check, as well as the concat call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
