Re: Towards a spec for robust streaming SQL, Part 1

There's something to be said about having different triggering depending on which side of a join data comes from, perhaps? (delightful doc, as usual) Kenn On Fri, Apr 21, 2017 at 1:33 PM, Tyler Akidau wrote: > Thanks for reading, Luke. The simple answer is that

Re: Best practice for exhaustive planning

Bumping this, as it ended up in spam for some people. Kenn On Tue, May 15, 2018 at 9:26 AM Kenneth Knowles <k...@google.com> wrote: > Hi all, > > Beam SQL uses Calcite for parsing and (naive) planning. Currently it is > pretty easy to write a SQL query that parses and causes

Best practice for exhaustive planning

Hi all, Beam SQL uses Calcite for parsing and (naive) planning. Currently it is pretty easy to write a SQL query that parses and causes a "could not plan" dump when we ask the planner to convert things to the Beam calling convention. One current example is using UNNEST on a column to yield a

Assertions failing in VolcanoPlanner from logical optimization / CalcRemoveRule

Hi all, I have been working on converting Beam SQL from have its own Project/Filter implementations to just using Calc. (PR at https://github.com/apache/beam/pull/5544) Basically, this diff to the rule set: + // Rules so we only have to implement Calc +

Re: Best practice for exhaustive planning

t; > Le mar. 22 mai 2018 à 19:01, Kenneth Knowles a > écrit : > > > Bumping this, as it ended up in spam for some people. > > > > Kenn > > > > On Tue, May 15, 2018 at 9:26 AM Kenneth Knowles wrote: > > > > > Hi all, > > > > > &g

Re: Best practice for exhaustive planning

d, May 30, 2018 at 6:10 AM Michael Mior wrote: > Unfortunately, I'm not sure of the best way how to proceed from here, but > it seems like you're making progress :) > -- > Michael Mior > mm...@apache.org > > > > Le mar. 29 mai 2018 à 18:29, Kenneth Knowles a > é

Re: Avatica struct support

Yes, thanks! I've got it. What can I do to help with the review process? Kenn On Wed, Jun 27, 2018 at 9:11 PM Francis Chuang wrote: > Hey Kenn, > > I've assigned the Contributor role to your account. Can you try and see > if self-assigning works? > > Francis > > On 28/

Avatica struct support

Hi all, I'm new to the project(s) so I wanted to share something I want to contribute to: maturing the ability to use sqlline + Avatica to process complex types (rows, arrays, maps - I don't care about more esoteric stuff just yet). It seems like the functionality is mostly there but maybe not

Re: [DISCUSS] reasonable duration for tests in the Calcite codebase

FWIW Beam has "pre-commit" and "post-commit" suites [1], with the latter being quite a lot longer and including lots of integration tests with e.g. cloud data processing engines and storage systems, so at least the tests have a home. You get "quick" feedback on pull requests (let's ignore how slow

Re: Sessionizing raw events / does Calcite support ARRAY_AGG?

istent with other aggregate functions, and > would allow us to supply ARRAY_AGG without extra parser work. > > Julian > > > On Fri, Dec 14, 2018 at 5:58 PM Kenneth Knowles wrote: > > > > Hello! > > > > My use case is sessionizing raw events without an aggrega

Sessionizing raw events / does Calcite support ARRAY_AGG?

Hello! My use case is sessionizing raw events without an aggregation function. Approximate code that I tried out: SELECT ARRAY_AGG(ROW(...)) FROM ... GROUP BY SESSION(...) (followed by UNNEST to get the raw events, tagged with session info, back out into a stream) I get a parser error on the

Nanosecond precision timestamps?

Hello! Beam SQL uses Calcite's codegen for Calc relations. Ideally, we would support nanosecond precision like java.sql.Timestamp and the new Java 8 date/time libraries. it looks like there is an assumption of millisecond precision. Is this nanosecond support possible? I have not grokked the

Re: CALCITE-2905: Maven -> Gradle: any thoughts

I don't modify Calcite often enough to deserve a full vote, but I am +0.5 as an occasional contributor. Beam transitioned from Maven to Gradle. We saw clean build times speed up by 5x or so. And of course clean builds are irrelevant for correct build systems, as their whole purpose is to build

Re: Planner state vs CannotPlanException vs humans

On a previous thread, I asked about this, as it is not just for debugging but user-facing. It seems there is no reliable way to build a tool on top of Calcite that gives high assurance of avoiding CannotPlanException reaching a user. Are other planners better for this? Beam SQL would probably

Re: CALCITE-2905: Maven -> Gradle: any thoughts

FWIW Beam also has gpg signing which is a one liner [1] and generates a pom [2] where the latter is very manual unfortunately. We've used all this for the last 5 releases. One negative is that default of very high parallelism, while good for end-to-end time, can cause OOMs and require JVM heap

Re: Streams Query Optimization Using Rate and Window Size

Following this discussion, I have a question which I think is on topic. Seems like there's two places that from my brief reading are not quite extensible enough. 1. RelNode.computeSelfCost returns RelOptCost has particular measures built in. Would Alireza's proposal require extensibility here to

Re: Streams Query Optimization Using Rate and Window Size

gt;> assumed any value that we are getting as a cost is going to be function > of > >> (row_count, CPU, I/O). Note that in the streaming model there is no need > >> for window to be in the cost (the cost does not depend on it), I am > >> including it in the cost

Re: Streaming extensions to the SQL Standard, call for participation

Thanks from me as well. I too would be interested in participating. Kenn On Tue, Nov 26, 2019 at 3:29 PM Tyler Akidau wrote: > Thanks Julian. Nice to meet you Rick. I'm interested as well, although I'll > have some difficult-to-move conflicts for some of those dates. I'll sign up > and do what

Re: [ DISCUSS] The window table functions join syntax

It happens in "raw Beam" somewhat often that there is one windowed aggregation followed by another. It is pretty unheard of to "assign" windows twice before an aggregation, because Beam cannot do anything useful with this: Beam does not have the ability to have two different fields/columns that

Re: [DISCUSS] Fixing things that aren't broken

>From the peanut gallery... I have been spending my "free" hours adding checkerframework nullness typing to Beam and reliably finding bugs. It is super valuable. It is not at all surprising that Vladimir has uncovered bugs. The likelihood of a Calcite-sized project successfully defending against

[jira] [Created] (CALCITE-2386) Support columns containing struct / nested rows

Kenneth Knowles created CALCITE-2386: Summary: Support columns containing struct / nested rows Key: CALCITE-2386 URL: https://issues.apache.org/jira/browse/CALCITE-2386 Project: Calcite

[jira] [Created] (CALCITE-2394) Avatica applies calendar offset to timestamps when they should remain unchanged

Kenneth Knowles created CALCITE-2394: Summary: Avatica applies calendar offset to timestamps when they should remain unchanged Key: CALCITE-2394 URL: https://issues.apache.org/jira/browse/CALCITE-2394

[jira] [Created] (CALCITE-2634) Query parses but fails to convert to rel

Kenneth Knowles created CALCITE-2634: Summary: Query parses but fails to convert to rel Key: CALCITE-2634 URL: https://issues.apache.org/jira/browse/CALCITE-2634 Project: Calcite Issue