[jira] [Commented] (ARROW-4333) [C++] Sketch out design for kernels and "query" execution in compute layer
[ https://issues.apache.org/jira/browse/ARROW-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123933#comment-17123933 ] Wes McKinney commented on ARROW-4333: - I'd suggest we close this and pursue answers to some of the unanswered questions as more specific followups. In particular, I plan to be building multi-kernel expression evaluation in the near future so some of the pipelining/memory reuse questions must be addressed as a part of this > [C++] Sketch out design for kernels and "query" execution in compute layer > -- > > Key: ARROW-4333 > URL: https://issues.apache.org/jira/browse/ARROW-4333 > Project: Apache Arrow > Issue Type: New Feature > Components: C++ >Reporter: Micah Kornfield >Priority: Major > Labels: analytics > > It would be good to formalize the design of kernels and the controlling query > execution layer (e.g. volcano batch model?) to understand the following: > Contracts for kernels: > * Thread safety of kernels? > * When Kernels should allocate memory vs expect preallocated memory? How to > communicate requirements for a kernels memory allocaiton? > * How to communicate the whether a kernels execution is parallelizable > across a ChunkedArray? How to determine if the order to execution across a > ChunkedArray is important? > * How to communicate when it is safe to re-use the same buffers and input > and output to the same kernel? > What does the threading model look like for the higher level of control? > Where should synchronization happen? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-4333) [C++] Sketch out design for kernels and "query" execution in compute layer
[ https://issues.apache.org/jira/browse/ARROW-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116096#comment-17116096 ] Wes McKinney commented on ARROW-4333: - I partially addressed some of these questions in ARROW-8792, but there are other questions viz-a-viz memory reuse and dealing with ChunkedArrays. Perhaps it would be useful to go through these questions and discuss them in the context of the new generic kernel execution framework in arrow/compute > [C++] Sketch out design for kernels and "query" execution in compute layer > -- > > Key: ARROW-4333 > URL: https://issues.apache.org/jira/browse/ARROW-4333 > Project: Apache Arrow > Issue Type: New Feature > Components: C++ >Reporter: Micah Kornfield >Priority: Major > Labels: analytics > > It would be good to formalize the design of kernels and the controlling query > execution layer (e.g. volcano batch model?) to understand the following: > Contracts for kernels: > * Thread safety of kernels? > * When Kernels should allocate memory vs expect preallocated memory? How to > communicate requirements for a kernels memory allocaiton? > * How to communicate the whether a kernels execution is parallelizable > across a ChunkedArray? How to determine if the order to execution across a > ChunkedArray is important? > * How to communicate when it is safe to re-use the same buffers and input > and output to the same kernel? > What does the threading model look like for the higher level of control? > Where should synchronization happen? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-4333) [C++] Sketch out design for kernels and "query" execution in compute layer
[ https://issues.apache.org/jira/browse/ARROW-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767264#comment-16767264 ] Francois Saint-Jacques commented on ARROW-4333: --- References I suggest you to check. - [The Design and Implementation of Modern Column-Oriented Database Systems, 2012|http://db.csail.mit.edu/pubs/abadi-column-stores.pdf]: This is a long read, but worth the investment. Will give you a broad overview of what are columnar databases and what makes them fast. - [MonetDB/X100: Hyper-Pipelining Query Execution, 2005|https://pdfs.semanticscholar.org/2e84/4872e32a4a4e94e229a9a9e70ac47d710252.pdf] foundation paper on how to implement fast query engine for analytics on modern hardware. - [Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask, 2018|https://pdfs.semanticscholar.org/2e84/4872e32a4a4e94e229a9a9e70ac47d710252.pdf]: This is an update to the paper you linked, which studies Compilation vs Vectorization. - [Vectorization vs. Compilation in Query Execution, 2011|https://15721.courses.cs.cmu.edu/spring2019/papers/21-vectorization2/p5-sompolski.pdf] - [Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last, 2017|https://pdfs.semanticscholar.org/1a21/509b67d3ed06cd7062f2f9b7e5b0b32a32e6.pdf] - [Make the Most out of Your SIMD Investments: Counter Control Flow Divergence in Compiled Query Pipelines, 2018|http://db.in.tum.de/~lang/papers/simd_divergence.pdf], talks about using AVX-512 masked instructions. Anything by: [Daniel J. Abadi|https://www.semanticscholar.org/author/Daniel-J.-Abadi/2254232] part of the team that wrote CStore which became Vertica. Write less about columnar execution in the last years. [Peter Boncz|https://www.semanticscholar.org/author/Peter-A.-Boncz/1687211] behind MonetDB/Vectorwize. [Thomas Neumann|https://www.semanticscholar.org/author/Thomas-Neumann/1706846] behind [Hyper|https://hyper-db.de/] bought by Tableau. [Andrew Pavlo|https://www.semanticscholar.org/author/Andrew-Pavlo/1774210] teaches database course at CMU Amazing video lectures of courses at CMU, you can ignore most of the storage layer, concurrency, transaction context. We're interested in execution engine, vectorization, and compilation. * [https://www.youtube.com/playlist?list=PLSE8ODhjZXjYplQRUlrgQKwIAV3es0U6t] * [https://www.youtube.com/playlist?list=PLSE8ODhjZXjbjOyrcqgE6_lCV6xvzffSN] * [https://www.youtube.com/playlist?list=PLSE8ODhjZXjY2xvwxuKjZT5qFH0sQga8_] * [https://www.youtube.com/playlist?list=PLSE8ODhjZXjY0GMWN4X8FIkYNfiu8_Wl9] The [CMU 15-721 Advanced Database Systems schedule|https://15721.courses.cs.cmu.edu/spring2019/schedule.html] is usually a good source of papers. Relevant code base: - [Impala|https://github.com/apache/impala/tree/master/be/src] - [ClickHouse|https://github.com/yandex/ClickHouse/tree/master/dbms/src] - [MapD|https://github.com/omnisci/mapd-core/tree/master] - [Supersonic|https://github.com/google/supersonic] (googlers can probably see the more recent version behind closed wall) > [C++] Sketch out design for kernels and "query" execution in compute layer > -- > > Key: ARROW-4333 > URL: https://issues.apache.org/jira/browse/ARROW-4333 > Project: Apache Arrow > Issue Type: New Feature > Components: C++ >Reporter: Micah Kornfield >Priority: Major > Labels: analytics > Fix For: 0.13.0 > > > It would be good to formalize the design of kernels and the controlling query > execution layer (e.g. volcano batch model?) to understand the following: > Contracts for kernels: > * Thread safety of kernels? > * When Kernels should allocate memory vs expect preallocated memory? How to > communicate requirements for a kernels memory allocaiton? > * How to communicate the whether a kernels execution is parallelizable > across a ChunkedArray? How to determine if the order to execution across a > ChunkedArray is important? > * How to communicate when it is safe to re-use the same buffers and input > and output to the same kernel? > What does the threading model look like for the higher level of control? > Where should synchronization happen? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4333) [C++] Sketch out design for kernels and "query" execution in compute layer
[ https://issues.apache.org/jira/browse/ARROW-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766766#comment-16766766 ] Micah Kornfield commented on ARROW-4333: [http://www.vldb.org/pvldb/vol4/p539-neumann.pdf] might be of interest. It seems like this reverses the model to some extent (and then goes ahead and disposes of it entirely to compile code). Caveat (I'm still ramping up on DB systems in general). > [C++] Sketch out design for kernels and "query" execution in compute layer > -- > > Key: ARROW-4333 > URL: https://issues.apache.org/jira/browse/ARROW-4333 > Project: Apache Arrow > Issue Type: New Feature > Components: C++ >Reporter: Micah Kornfield >Priority: Major > Labels: analytics > Fix For: 0.13.0 > > > It would be good to formalize the design of kernels and the controlling query > execution layer (e.g. volcano batch model?) to understand the following: > Contracts for kernels: > * Thread safety of kernels? > * When Kernels should allocate memory vs expect preallocated memory? How to > communicate requirements for a kernels memory allocaiton? > * How to communicate the whether a kernels execution is parallelizable > across a ChunkedArray? How to determine if the order to execution across a > ChunkedArray is important? > * How to communicate when it is safe to re-use the same buffers and input > and output to the same kernel? > What does the threading model look like for the higher level of control? > Where should synchronization happen? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4333) [C++] Sketch out design for kernels and "query" execution in compute layer
[ https://issues.apache.org/jira/browse/ARROW-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763321#comment-16763321 ] Wes McKinney commented on ARROW-4333: - Yes, I agree with coming up with a plan for all these concerns. I think the volcano batch model is the way to go. This is what Impala and many other systems use quite successfully > [C++] Sketch out design for kernels and "query" execution in compute layer > -- > > Key: ARROW-4333 > URL: https://issues.apache.org/jira/browse/ARROW-4333 > Project: Apache Arrow > Issue Type: New Feature > Components: C++ >Reporter: Micah Kornfield >Priority: Major > Labels: analytics > Fix For: 0.13.0 > > > It would be good to formalize the design of kernels and the controlling query > execution layer (e.g. volcano batch model?) to understand the following: > Contracts for kernels: > * Thread safety of kernels? > * When Kernels should allocate memory vs expect preallocated memory? How to > communicate requirements for a kernels memory allocaiton? > * How to communicate the whether a kernels execution is parallelizable > across a ChunkedArray? How to determine if the order to execution across a > ChunkedArray is important? > * How to communicate when it is safe to re-use the same buffers and input > and output to the same kernel? > What does the threading model look like for the higher level of control? > Where should synchronization happen? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
