Hey Andrew,
Sorry for the late reply. For some reason, your email never came through to
me and I didn't know about it until someone just referenced it within the
apache mail archive. I'll respond to your questions inline. In general,
we're experimenting with the GitHub discussions functionality as opposed to
using a mailing list [1] so would be great if you also continued discussion
there.
> This seems like something that would be beneficial if you can get other
> projects to buy into it. [xkcd]
Completely agree. This project is designed specifically to help other
projects work together and will only be impactful if it can be leveraged by
several projects. I think we've made good progress thus far with
contributions from several key communities (Arrow C++, Datafusion, Iceberg,
Presto/Velox, Singlestore and others). The xkcd reference is rough. I agree
with the general sentiment but I'm not aware of any initiative that is
actually trying to do this cross-project (there are initiatives within a
single project that take a very "local" perspective"). To me, the
opportunity is about solving for extensibility and a library of common
conversions between varying standards. This is much like the LLVM project.
There were many people building languages and compilers before LLVM. By
building a standardized intermediate representation, LLVM enabled frontends
to build stuff independent of the backend targets of those creations.
Hopefully the same thing will be true for processing systems with Substrait.
> How did you agree to the four indicator implementations?
They were a proposed sketch. We've had further discussion about that topic
in this discussion topic [2]. Keep in mind that those projects are not
fundamental to Substrait but were just used to help guide an
initial attempt at coming up with a system-agnostic set of common modern
data types.
Are those projects committed to making breaking changes to move closer
> together?
No and Substrait doesn't strive to achieve that. The goal of Substrait is
to come up with a framework, a set of common primitives, and a strong
extensibility system so systems that have specialized needs can still work
with the Substrait format and specification and cross-operate with other
systems.
We are working on something similar in Apache Beam. Our goal is a
> unified model with high level APIs that allow users to switch out
> different engines ("runners"), our implementation could be simplified
> by a standard like yours. We take plans as SQL from Apache Calcite and
> Google ZetaSQL, via a programmatic API, and hopefully other sources in
> the future (Dataframes). We execute them on Beam runners (currently as
> a Java implementation, possibly Arrow in the future). Eventually we
> want these plans to run natively where supported (Flink, Spark or
> Google Dataflow). Previously we've focused on the top end (turning
> plans into Beam Java pipelines) but we are working to push the
> relational metadata down the stack now. We need something that is a
> superset of existing implementations,
This is exactly why I think there is a need for Substrait. I've talked to
people at nearly a dozen different companies in the last few months and
they are almost all struggling with these issues in one form or another.
People are starting to think about frontend data processing plan producers
and backend data processing engines independently in a more mature
manner. Without a standardized way to connect the two, we get a lot of
independent engines that can't work together (and a lot of dialect
variations that can only target a single engine).
> you appear more focused on a subset.
This is a misinterpretation of the goals of Substrait. While it is true
that Substrait will only have a subset of all possible data types, function
signatures and relational operations in the project proper, the goals of
the extension system are that any project can use additional git-uri based
extensions to describe the additional functionality.
It could still work but Google's funding to build Relational
> Beam is dependent on us providing support for internal use cases,
> which means native execution on Dataflow with ZetaSQL. Apache Beam
> isn't tied to ZetaSQL but we won't be able to adopt a standard that
> prevents us from passing the ZetaSQL tests.
The functions are significantly more difficult than the types. Inside
> Google, ZetaSQL is the standard and that team has defined a huge set
> of test cases[4] for their functions. It took them several years to
> get a catalog built and for everyone to standardize on it. I'm not
> sure the same thing is possible outside a company with a top down
> mandate to unify. My understanding is that they based their function
> catalog on the existing implementations that were being unified and
> used external references (SQL standard, Postgres) to resolve
> conflicts. They have a reference implementation that they consider the
> source of truth. This specification is
