I'm working on my next PR which addresses the "scalar output modality" and removes usages of ValueDescr and output shapes in general from the kernel execution machinery. The ability to provide all scalar input and scalar output of course is preserved — it's just that the "boxing" and "unboxing" of the inputs and outputs is happening at a higher level (i.e. there is only one place that boxes scalars into arrays and unboxes the output into a scalar) rather than being implemented on a piecemeal basis in the kernel implementations.
Please keep an eye out for any non-trivial PRs that affect arrow/compute/kernels since it could cause some rebasing headache. I am working as fast as I can to complete this next PR. After this, the next thing is porting the aggregate kernels to use ExecSpan. Then I think the mass refactoring will cool off and we can address follow-on improvements like rewriting expression evaluation to utilize the span data structures to yield performance gains. On Mon, Jun 13, 2022 at 12:37 PM Wes McKinney <wesmck...@gmail.com> wrote: > > I merged the PR a little while ago — thanks for David, Sasha for > helping review. If you have more comments or things you would like to > fix from that PR, please comment there and I can help address them in > follow up PRs. > > I'm going to work next on migrating the rest of the kernels to use > ExecSpan (and thus ExecBatchIterator can be removed in favor of > ExecSpanIterator — I'll add benchmarks so we can see the performance > difference), and then my next priority will be removing the ValueDescr > concept and the ValueDescr::SCALAR output path for the > scalar/elementwise kernels which will help us delete a lot of code > > I'll attach related Jiras to this umbrella issue: > > https://issues.apache.org/jira/browse/ARROW-16755 > > On Fri, Jun 10, 2022 at 12:56 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > PR is up: https://github.com/apache/arrow/pull/13364 > > > > Look forward to getting this in since there's a bunch of follow on > > work that I'd like to get started on ASAP! > > > > On Thu, Jun 9, 2022 at 7:34 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > I'm making good progress getting my branch PR-ready -- working through > > > the compute-scalar-test suite and fixing the little things I broke. I > > > hope I'll have it done by the end of the week. > > > > > > On Mon, Jun 6, 2022 at 3:21 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > > > I created https://issues.apache.org/jira/browse/ARROW-16755 as an > > > > umbrella issue to track improvements to reduce overhead in the > > > > expression and kernel execution machinery. Please help by attaching > > > > related issues and creating new issues for specific individual efforts > > > > here. I'll work as quickly as I can to have my initial patch > > > > ARROW-16756 ready which will unblock the next few projects here > > > > > > > > On Mon, Jun 6, 2022 at 10:35 AM Wes McKinney <wesmck...@gmail.com> > > > > wrote: > > > > > > > > > > This is definitely only the first stage of cleanup and streamlining — > > > > > I anticipate multiple rounds of refactoring (maybe not as invasive and > > > > > painful as this one), and this patch I'm not sure will do a lot to > > > > > alleviate bottom line expression evaluation overhead but it creates > > > > > the environment (i.e. where a whole chain of scalar functions that all > > > > > write into preallocated memory can execute without having to touch > > > > > shared_ptrs or deal with other objects with excess microperformance > > > > > overhead) where such optimization can happen more easily. > > > > > > > > > > > > > > > On Mon, Jun 6, 2022 at 4:08 AM Antoine Pitrou <anto...@python.org> > > > > > wrote: > > > > > > > > > > > > > > > > > > Le 06/06/2022 à 09:34, Sasha Krassovsky a écrit : > > > > > > > Wow that's a lot of progress! > > > > > > > Definitely agree on the scalar outputs point. > > > > > > > > > > > > > > One point about the ArraySpan - why does it need to know its data > > > > > > > type? > > > > > > > Once a kernel has been resolved by the registry, the kernel will > > > > > > > only know > > > > > > > how to execute on the specific type it was resolved for, right? > > > > > > > > > > > > Because of parametric types for example (e.g. timestamps with a > > > > > > unit and > > > > > > timezone, or decimals with a precision and scale). > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine.
