I think the performance performance depends on where the flattenoperator is resolved. If the flattenoperator remains as an actual execution step, then I agree that repeatedly introducing it into large relational workloads could add overhead, especially for complex queries with many joins. But from my experience while working with generic JDBC join support the SQL engine already produces a flat relational result. In that case I would expect a JDBC implementation of flattenoperator to be effectively a no-op and therefore contribute little or no runtime overhead. The generated SQL query itself would remain essentially the same. And for this reason my main concern with proposal 2 is less about runtime performance and more about planning complexity.The flattening step would need to be inserted and handled consistently and ensuring this works correctly across platforms may be more challenging than the runtime cost itself. And this is why I am aligning towards proposal 1 as it seems to align more easily with the relational model and avoids introducing additional planning logic around flattening.
On Thu, Jul 2, 2026 at 2:17 PM Zoi Kaoudi <[email protected]> wrote: > Hello again, > > Thank you for sharing your thoughts. > > What do you think about performance? We also need to consider which > approach would be more performant. > > For instance, adding a hidden flattenoperator could also increase the > runtime of a job running on relational data. Maybe slightly, but if the job > is very large it could be considerable (think of the Join Order Benchmark > (JOB) queries with 20+ joins). Don't you think so? > > Best > -- > Zoi > > On 2026/06/29 14:45:42 Mohit Bisht wrote: > > Hi all, > > In addition to implementing the Generic JDBC Join support, I also had to > > deal with some of the problems described by Zoi. > > There are several things which make me favor Proposal 1, first, the SQL > > generation layer inherently deals with flat relational rows. The SQL JOIN > > operation in JDBC produces a flat output, whereas the Tuple2<Record, > > Record> is a logical/operator concept. Regarding implementation, I > believe > > adding a join operator specifically for relations is easier than adding a > > flattening operation which would be consistently added and dealt with > > during planning, type checking, optimization and platform > implementations. > > It is not the flatten operation itself that I am worried about in > Proposal > > 2, but its consistent introduction during planning. > > On the other hand, I definitely agree that introducing more operators > > results in higher maintenance cost, so there is a choice of operator > > complexity versus planner and type-system complexity. I would be > interested > > to hear how the dataframe API will behave in each case. > > > > Best, > > Mohit > > > > On Wed, Jun 24, 2026 at 5:42 PM Kaustubh Beedkar <[email protected]> > wrote: > > > > > Agree with Jun > > > > > > On Wed, Jun 24, 2026 at 2:57 PM Jun Wang via dev < > [email protected]> > > > wrote: > > > > > > > I currently lean towards Proposal 1. > > > > > > > > A FlattenOutPut option could work if it is resolved before type > checking > > > > and optimization. However, I would be cautious about allowing > platform > > > > implementations to override it, since the logical output type should > > > remain > > > > platform-independent. > > > > > > > > Trino does support the SQL ROW type, so the composite-type approach > is > > > > interesting. Still, relying on it would require checking connector, > JDBC, > > > > and cross-platform support, and probably extending Wayang’s type > system. > > > > For the current scope, a dedicated relational join seems to me the > > > clearest > > > > and most practical solution. > > > > > > > > Best, > > > > > > > > Jun > > > > > > > > On Wed, Jun 24, 2026 at 3:38 PM Juri Petersen <[email protected]> wrote: > > > > > > > > > Hi, > > > > > I also think building a solution into our type system is the > cleanest > > > > way. > > > > > A composite type that would allow a view as a Tuple2<?, ?> or a > flat > > > type > > > > > could solve this already, if I am not missing something. > > > > > Having multiple instances of the same logical operator just for the > > > > > purpose of flattening seems counterintuitive. > > > > > > > > > > Best, > > > > > Juri > > > > > ________________________________ > > > > > From: Yasser Idris <[email protected]> > > > > > Sent: 24 June 2026 04:02 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] output type of Join operator issue > > > > > > > > > > +1 for What Gabor said, I don't think we need a new join operator > for > > > > > relational dbs. I think this breaks the abstraction of the logical > > > > > operator. I'm more inclined towards solution 2 but with a twist. > The > > > > > logical(wayang) join operator should have 2 modes > > > > > (FlattenOutput=True/False), and it defaults to False. Every > platform > > > > > implementation of the join operator can override the default, but > also > > > > > every plan (ie. this can be exposed to the user). This is not > specific > > > to > > > > > relational database. If this boolean is True the output is chained > into > > > > our > > > > > brand new FlattenOperator which as Zoi said, has platform specific > > > > > implementations. The Tuple 2 problem can be addressed with > > > abstraction, i > > > > > don't see it as an issue and actually this insertion of the > > > > Flattenoperator > > > > > happens before optimization since it's part of the Join definition. > > > > > > > > > > > > > > > > > > > > On Tue, Jun 23, 2026 at 12:37 PM Zoi Kaoudi <[email protected]> > > > wrote: > > > > > > > > > > > Thank you Gabor. I was not aware about this composite type for > > > > relational > > > > > > databases. However, I doubt that newest engines such as Trino or > > > Presto > > > > > > support this but we could check that. > > > > > > > > > > > > I still also prefer to add a new operator to have things cleaner. > > > Then > > > > a > > > > > > dataframe api can be built on top of that in an easier manner I > > > think. > > > > > > > > > > > > Best > > > > > > -- > > > > > > Zoi > > > > > > > > > > > > > > > > > > On 2026/06/23 08:55:38 Gábor Gévay wrote: > > > > > > > Btw. many relational engines also have Record or Row or some > kind > > > of > > > > > > > composite type, in which case it's actually possible to > represent > > > > > > > Tuple2<Record, Record> directly. So, a 3rd solution could be to > > > rely > > > > > > > on this. But the caveat is that I'm not sure how well this is > > > > > > > supported across systems. Even though the SQL:1999 standard > has it > > > > > > > (calls it `ROW`), but unfortunately not every "standard" thing > is > > > > > > > implemented even by the mainstream systems. > > > > > > > > > > > > > > For example, in Postgres: > > > > > > > ``` > > > > > > > CREATE TYPE lhs AS (a int, b int); > > > > > > > CREATE TYPE rhs AS (x int, y int); > > > > > > > > > > > > > > CREATE TABLE joined ( > > > > > > > l lhs, > > > > > > > r rhs > > > > > > > ); > > > > > > > > > > > > > > INSERT INTO joined VALUES (ROW(1,2), ROW(3,4)); > > > > > > > INSERT INTO joined VALUES (ROW(5,6), ROW(7,8)); > > > > > > > > > > > > > > SELECT * FROM joined; > > > > > > > > > > > > > > SELECT (l).*, (r).* FROM joined; > > > > > > > ``` > > > > > > > gives you: > > > > > > > ``` > > > > > > > l | r > > > > > > > -------+------- > > > > > > > (1,2) | (3,4) > > > > > > > (5,6) | (7,8) > > > > > > > (2 rows) > > > > > > > > > > > > > > a | b | x | y > > > > > > > ---+---+---+--- > > > > > > > 1 | 2 | 3 | 4 > > > > > > > 5 | 6 | 7 | 8 > > > > > > > (2 rows) > > > > > > > ``` > > > > > > > (The second SELECT shows how to flatten it.) > > > > > > > > > > > > > > Best, > > > > > > > Gábor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexander Alten <[email protected]> ezt írta (időpont: 2026. > jún. > > > > 23., > > > > > > K, 10:13): > > > > > > > > > > > > > > > > Hi community, > > > > > > > > > > > > > > > > Good discussion - I’d support Kaustubh. It is an additional > > > > operator > > > > > > for a different case, and should be also defined as one. > > > > > > > > > > > > > > > > Best, > > > > > > > > —Alex > > > > > > > > > > > > > > > > > On Jun 23, 2026, at 06:35, Kaustubh Beedkar < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi Zoi, > > > > > > > > > > > > > > > > > > This is exactly the issue I had to deal with when > implementing > > > > the > > > > > > initial > > > > > > > > > join operator in SQL api. > > > > > > > > > > > > > > > > > > The Tuple2<..> is a must for the dataflow case as it > "encodes" > > > > the > > > > > > pairing > > > > > > > > > of two inputs while preserving the sides. But of course > this is > > > > the > > > > > > wrong > > > > > > > > > algebra for the relational case. > > > > > > > > > > > > > > > > > > Prposal 1 seems like a better option IMO. > > > > > > > > > > > > > > > > > > Best > > > > > > > > > Kaustubh > > > > > > > > > > > > > > > > > > On Mon, Jun 22, 2026 at 7:45 PM Zoi Kaoudi via dev < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Dear all, > > > > > > > > >> > > > > > > > > >> this is a long email, please bare with me. > > > > > > > > >> > > > > > > > > >> We have a fundamental and recurring issue with the output > type > > > > of > > > > > > the > > > > > > > > >> current Join operator. The Join operator as is now takes > as > > > > input > > > > > > two > > > > > > > > >> generic datatypes (potentially different) and outputs a > Tuple2 > > > > of > > > > > > these > > > > > > > > >> input datatypes. See here: > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_url-3Fq-3Dhttps-3A__github.com_apache_wayang_blob_cb4bb0fd0245240124174b562e4ad1abbc9cd0c0_wayang-2Dcommons_wayang-2Dbasic_src_main_java_org_apache_wayang_basic_operators_JoinOperator.java-2523L37-26source-3Dgmail-2Dimap-26ust-3D1782794153000000-26usg-3DAOvVaw3d4SXsHbG8-2DgRlC0sCSPNI&d=DwIGaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=K-pGUp478bHBxJaHksONsg&m=qJpq2b8-Boxfz0iGjP52w8YwQ6PomnnhJepFaWpeogFjf9fQlhHGRSVNCv_iVJ19&s=cj42uI7ljZf7BV9opO9YjAqil63_oZ3cCHcZB-Ad4cw&e= > > > > > > > > >> > > > > > > > > >> This is fine for a general dataflow type of engine. It > also > > > > > enables > > > > > > > > >> different types of joins, like joining specific java > object > > > > > > (Employee with > > > > > > > > >> Employee) with a custom join key. However, when we are > > > handling > > > > > > relational > > > > > > > > >> data and have JDBC platforms for execution this creates > > > > problems. > > > > > > > > >> > > > > > > > > >> *Problem 1* > > > > > > > > >> When the inputs are Records, the join output is a > > > Tuple2<Record, > > > > > > Record> > > > > > > > > >> rather than a flat Record. Any relational operator we > chain > > > > after > > > > > > the join > > > > > > > > >> therefore cannot consume the output directly; the user > has to > > > > > > insert a > > > > > > > > >> flatten step (a MapOperator) to turn the Tuple2 into a > single > > > > > > Record. This > > > > > > > > >> is very cumbersome of course. > > > > > > > > >> > > > > > > > > >> *Problem 2* > > > > > > > > >> Even if we "hide" a flatten map operator step somehow and > we > > > > > insert > > > > > > it > > > > > > > > >> automatically, this step cannot run inside a database. So > even > > > > > when > > > > > > the > > > > > > > > >> whole pipeline could execute on a JDBC platform, the > flatten > > > > > > MapOperator > > > > > > > > >> forces the optimizer to move data out of the database > into the > > > > JVM > > > > > > just to > > > > > > > > >> reshape it. This is ok as long as every in-database > pipeline > > > > ends > > > > > by > > > > > > > > >> streaming its results back to Java anyway. However, if we > want > > > > to > > > > > > have > > > > > > > > >> fully in-database pipelines, this becomes problematic: For > > > > > example, > > > > > > with an > > > > > > > > >> in-database sink that writes the join result back into a > table > > > > > > (CREATE > > > > > > > > >> TABLE AS SELECT) and expects a Record as input, > connecting the > > > > > join > > > > > > to the > > > > > > > > >> sink fails at plan construction because Tuple2 != Record, > even > > > > > > though the > > > > > > > > >> SQL engine would naturally produce a flat row. > > > > > > > > >> > > > > > > > > >> *Proposed solution 1* > > > > > > > > >> We introduce a new join operator for relational data that > > > takes > > > > > two > > > > > > > > >> Records and outputs a single, flat Record. On JDBC this > maps > > > > > > directly to a > > > > > > > > >> SQL join statement which already yields flat rows; on > > > Java/Spark > > > > > it > > > > > > > > >> concatenates the two Records instead of wrapping them in a > > > > Tuple2. > > > > > > The > > > > > > > > >> current Tuple2-based Join should stay for the general > dataflow > > > > > > case, and > > > > > > > > >> the Record-based one is used for the relational case. The > same > > > > > idea > > > > > > should > > > > > > > > >> be extended to the other operators that share this > wrapping > > > > > > behaviour (e.g. > > > > > > > > >> the cartesian product). > > > > > > > > >> Pros: flat types end to end and in-database pipelines can > stay > > > > > > fully in > > > > > > > > >> the database. > > > > > > > > >> Cons: a second join operator to maintain and the > > > relational/SQL > > > > > > path has > > > > > > > > >> to target it. > > > > > > > > >> > > > > > > > > >> *Proposed solution 2* > > > > > > > > >> We keep the current Join and add a new dedicated Wayang > > > flatten > > > > > > operator > > > > > > > > >> (eg FlattenOperator) that converts Tuple2<Record, Record> > > > into a > > > > > > flat > > > > > > > > >> Record, with a platform-specific implementations: on JDBC > it > > > > > > essentially > > > > > > > > >> adds an empty string (the SQL already produces flat rows, > so > > > it > > > > > > contributes > > > > > > > > >> nothing to the query), while on Java/Spark it performs the > > > > actual > > > > > > > > >> concatenation. Ideally the optimizer should insert it > > > > > automatically > > > > > > > > >> wherever the join will be executed. > > > > > > > > >> Pros: reuses the existing, fully supported operators, > with a > > > > > single > > > > > > > > >> mechanism. > > > > > > > > >> Cons: we keep the Tuple2 in the logical plan and since the > > > type > > > > > > check > > > > > > > > >> happens before optimization, the automatic insertion may > need > > > to > > > > > > happen > > > > > > > > >> somewhere else. > > > > > > > > >> > > > > > > > > >> I'd appreciate any thoughts on this and also any > experiences > > > you > > > > > > had and > > > > > > > > >> how you solved them or how you would like to have them > solved. > > > > > > > > >> Also feel free to propose some other solution. > > > > > > > > >> > > > > > > > > >> Best > > > > > > > > >> -- > > > > > > > > >> Zoi > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *Scalytics* > > > > > > > > The foundation for secure, scalable, and transparent AI. > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scalytics.io&d=DwIGaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=K-pGUp478bHBxJaHksONsg&m=qJpq2b8-Boxfz0iGjP52w8YwQ6PomnnhJepFaWpeogFjf9fQlhHGRSVNCv_iVJ19&s=Rd9RXVrx-cbXbHHUTUTmgfLBerwSxY5LSD6amtKGjvo&e= > > > > > < > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scalytics.io&d=DwIGaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=K-pGUp478bHBxJaHksONsg&m=qJpq2b8-Boxfz0iGjP52w8YwQ6PomnnhJepFaWpeogFjf9fQlhHGRSVNCv_iVJ19&s=Rd9RXVrx-cbXbHHUTUTmgfLBerwSxY5LSD6amtKGjvo&e= > > > > > > < > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scalytics.io&d=DwIGaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=K-pGUp478bHBxJaHksONsg&m=qJpq2b8-Boxfz0iGjP52w8YwQ6PomnnhJepFaWpeogFjf9fQlhHGRSVNCv_iVJ19&s=Rd9RXVrx-cbXbHHUTUTmgfLBerwSxY5LSD6amtKGjvo&e= > > > > > > > > > > > > > > > > > > > > > > -- Please consider the > > > > > > > > environment before printing this email -- > > > > > > > > > > > > > > > > Disclaimer: > > > > > > > > The content of this > > > > > > > > message is confidential. If you have received it by mistake, > > > please > > > > > > inform > > > > > > > > us by an email reply and then delete the message. It is > forbidden > > > > to > > > > > > copy, > > > > > > > > forward, or in any way reveal the contents of this message to > > > > anyone. > > > > > > The > > > > > > > > integrity and security of this email cannot be guaranteed > over > > > the > > > > > > > > Internet. Therefore, the sender will not be held liable for > any > > > > > damage > > > > > > > > caused by the message. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
