Hi, Jan: Thanks for the update. Left some comments and let's continue the discussion in doc.
On Tue, Nov 7, 2023 at 4:26 PM Jan Kaul <jank...@mailbox.org.invalid> wrote: > Hi all, > > thanks again to those who left some comments on the Iceberg Materialized > View spec. The discussions showed that there are still some open questions > that need to be answered. I added a section to the end of the document to > highlight the decisions that need to be made. Please have another look, I > would really appreciate your input. > > Best wiches, > > Jan > On 27.10.23 11:48, Jan Kaul wrote: > > Thank you Dan and the others for your helpful comments. I've added some > sections to address the points that you mentioned. I'm not really sure what > you mean by fail after grace period. > > I've found a design document for the trino materialized views and tried to > incorporate some of the points. I'm not really sure where to find the rest > of the discussion. If you can think of important points please make a > comment in the google doc so that I can add them to the document. > > Best wishes > > Jan > On 26.10.23 20:57, Daniel Weeks wrote: > > I added a few comments to the doc, but I think there a few other things > that probably need to be considered: > > Do we define behaviors around freshness (fail after grace period or fall > back to view definition), what is the expectation for refreshes > (manual/just-in-time/lazy), etc. > > I think there was a fair amount of discussion around this when Trino > implemented materialized views, so it may be good to revisit those > discussions and make sure we're covering the major areas. > > -Dan > > On Thu, Oct 26, 2023 at 11:50 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Daniel is right, we deviated :) >> >> OK Brian, let's do that. >> >> Apologies. >> >> Regards >> JB >> >> On Thu, Oct 26, 2023 at 8:40 PM Brian Olsen <bitsondata...@gmail.com> >> wrote: >> > >> > Agreed, apologies to Jan :). JB, let's discuss this at the sync this >> Wed, and after that we can create a new thread if needed. >> > >> > On Thu, Oct 26, 2023 at 1:38 PM Daniel Weeks <dwe...@apache.org> wrote: >> >> >> >> JB and Brian, >> >> >> >> I think we should probably move this discussion to a discuss thread >> specifically for the topics you want to address. >> >> >> >> We've had a few instances now where the original intent of the thread >> is redirected to talk about other subjects. I don't feel this is a good >> approach because, while it is on the apache mailing list, the topic of the >> thread doesn't reflect the content, so you don't get the right >> audience/level of engagement or buy-in. >> >> >> >> I'm not disagreeing with trying to improve how we communicate and >> track improvements/proposals/etc, but I think we should try to keep the >> thread on topic. >> >> >> >> Thanks, >> >> -Dan >> >> >> >> On Thu, Oct 26, 2023 at 9:26 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> >> >>> Oh, I don't say we have to provide a user mailing list. Personally, I >> >>> like mailing list mainly because we have https://lists.apache.org/ >> >>> where we can browse and search on the mailing lists. >> >>> A lot of Apache projects are using Slack or Zulip, but in parallel of >> >>> mailing lists. As we say at Apache: "if it doesn't happen on the >> >>> mailing list, it never happens". >> >>> That said I would distinguish: >> >>> - for dev, obviously we can use Slack for discussion, community >> >>> meetings, etc, but we have to send main topics/discussions on the dev >> >>> mailing list. >> >>> - for user, I think Slack is good, but I like the user mailing list, >> >>> to track/search/async communication as well. >> >>> >> >>> That's another discussion anyway, let's focus on the design proposals >> >>> space: my understanding is that we want to have a space listing all >> >>> proposals, for review, tagged as "done" or "in progress". Right ? >> >>> I don't think a forum/stack overflow like would help here (it helps >> >>> for users, not for dev/technical/design proposals). >> >>> >> >>> At Apache Beam, we have a similar page as at Iceberg: >> >>> https://beam.apache.org/roadmap/ where you can click on roadmap items >> >>> for details (https://beam.apache.org/roadmap/portability/). >> >>> So, initially, I proposed to update >> >>> https://iceberg.apache.org/roadmap/ with proposals (status >> >>> "discussion"). As most of the proposals (all ?) come as Google Link, >> >>> we can change a bit the look'n feel of this page including the list of >> >>> proposals. >> >>> >> >>> That could be a first move, we can update later. >> >>> >> >>> Regards >> >>> JB >> >>> >> >>> On Thu, Oct 26, 2023 at 5:54 PM Brian Olsen <bitsondata...@gmail.com> >> wrote: >> >>> > >> >>> > Yeah, unfortunately there's no way to limit the functionality to >> only facilitate this. In fact, the product that gets closest to it is >> GitHub Issues. >> >>> > >> >>> > I believe putting the onus on developers deeply involved in the >> project makes sense. Expecting users, especially newer users of a newer >> generation will use an email list is unlikely, especially if they're in a >> discovery mode and figuring out how to solve an issue. A lot of garnering >> adoption from users is lowering every barrier to entry as well as lowering >> time to that first hello world dopamine hit. >> >>> > >> >>> > I'm middle millennial and even I find using email for discussion >> outside of my mental model/preference but I also see the benefits. >> >>> > >> >>> > On Thu, Oct 26, 2023 at 10:45 AM Jean-Baptiste Onofré < >> j...@nanthrax.net> wrote: >> >>> >> >> >>> >> The idea is really to "square" GH Discussion only to >> roadmap/design proposals. >> >>> >> >> >>> >> For "user support", more than Slack, I would love to see >> >>> >> u...@iceberg.apache.org. >> >>> >> >> >>> >> So I would distinguish: >> >>> >> - the design/spec proposals where we could use GH Discussions. If >> >>> >> people use GH Discussion for support questions, then we can move >> to GH >> >>> >> Issue or direct to the mailing list/slack. >> >>> >> - the user "support" should be on user mailing list and/or Slack >> >>> >> >> >>> >> You have a valid point: GH Discussions could be hard to manage >> because >> >>> >> most users will use it as a "support forum". >> >>> >> >> >>> >> My point is really: >> >>> >> - we need a central space for design/spec proposals >> >>> >> - it has to be on Iceberg community and visible for all >> >>> >> >> >>> >> Regards >> >>> >> JB >> >>> >> >> >>> >> On Thu, Oct 26, 2023 at 5:30 PM Brian Olsen < >> bitsondata...@gmail.com> wrote: >> >>> >> > >> >>> >> > GitHub Discussions could be a solution that we should consider. >> We used it on the Trino side but still have mixed results with it. On one >> hand, there's a lot of overlap between creating Issues and Discussions. In >> fact, GitHub allows you to migrate Issues that only involve discussing a >> topic, or something that can't immediately be tied to any upcoming work to >> be a discussion. This keeps the Issue backlog focused on actionable >> requests. >> >>> >> > >> >>> >> > That said, Discussions can become difficult to maintain if no >> person or body of people drives it. Of course, the community will drive it >> to some degree, especially when it's new and shiny, but GitHub Discussions, >> much like Slack, becomes a support channel that encourages the messy human >> interactions that help us arrive at a solution. So the question is do we >> want to open Discussions knowing that it may become a second support >> channel compared to Slack? Would we want to use Discussions in place of >> Slack so that there's still a single triage channel? >> >>> >> > >> >>> >> > I personally lean towards keeping a single real-time >> "support-like" channel in the community, otherwise, you will fragment the >> attention of the community. Most of what we would need to support the >> centralization of proposals can be accomplished with Issues. Slack still >> seems to be the dominant interactive system of choice and where we are now >> so I wouldn't suggest moving that. I do think this is worth a discussion at >> the next sync so I'll add it. >> >>> >> > >> >>> >> > In full transparency, Tabular is building an Iceberg-focused >> Discourse forum (not to be confused with Discord) instance to solve the >> problem of centralizing discussions in the community to wiki-style answers >> we can link to and having dedicated content curators to those solutions. >> Think of it as an Iceberg-specific Stack Overflow with lightened rules to >> allow more open discussion. Adding GitHub discussions wouldn't collide with >> our goals as it would become another signal that we could use to inform the >> answers on our forum. It still comes back to the value given the cost for >> the community to manage it. >> >>> >> > >> >>> >> > I know I have a lot of thoughts around this and its because I've >> been down this road before, but perhaps there's a nuance I'm not seeing yet. >> >>> >> > >> >>> >> > On Thu, Oct 26, 2023 at 7:15 AM Jean-Baptiste Onofré < >> j...@nanthrax.net> wrote: >> >>> >> >> >> >>> >> >> Just to be clear: we can GH Discussions subjects template via >> >>> >> >> .asf.yaml but we have to open a ticket to INFRA to enable it. >> >>> >> >> >> >>> >> >> Regards >> >>> >> >> JB >> >>> >> >> >> >>> >> >> On Thu, Oct 26, 2023 at 1:56 PM Jean-Baptiste Onofré < >> j...@nanthrax.net> wrote: >> >>> >> >> > >> >>> >> >> > Hi Brian >> >>> >> >> > >> >>> >> >> > I like the idea of GitHub. Why not enabling (in .asf.yml) >> GitHub >> >>> >> >> > discussions ? A GitHub Discussion could be a good place to >> share the >> >>> >> >> > doc and exchange both in the doc and in the discussion >> comments. >> >>> >> >> > >> >>> >> >> > Regards >> >>> >> >> > JB >> >>> >> >> > >> >>> >> >> > On Thu, Oct 26, 2023 at 1:13 PM Brian Olsen < >> bitsondata...@gmail.com> wrote: >> >>> >> >> > > >> >>> >> >> > > Hey JB, >> >>> >> >> > > >> >>> >> >> > > I totally agree we need a place to centralize this but I'm >> nit a huge fan of all the lists we currently have going on the site. SSGs >> are just not an accessible method of storing lists. ( roadmap, blogs, >> videos, etc..). >> >>> >> >> > > >> >>> >> >> > > The roadmap is barely touched for this reason. I want to >> propose we move roadmap to GitHub projects. >> >>> >> >> > > >> >>> >> >> > > Likewise, I feel like somewhere on GitHub might be a better >> location for this type of thing. >> >>> >> >> > > >> >>> >> >> > > Maybe posting these in GitHub issues and adding a proposal >> label? >> >>> >> >> > > >> >>> >> >> > > On Tue, Oct 24, 2023 at 9:28 AM Jean-Baptiste Onofré < >> j...@nanthrax.net> wrote: >> >>> >> >> > >> >> >>> >> >> > >> Hi Jan >> >>> >> >> > >> >> >>> >> >> > >> Thanks for the reminder. I will take a look. >> >>> >> >> > >> >> >>> >> >> > >> As proposed by Renjie a few days ago, it would be great to >> >>> >> >> > >> gather/store all document proposals in a central place. >> >>> >> >> > >> >> >>> >> >> > >> If there are no objections, I will prepare a PR for the >> website about >> >>> >> >> > >> that (with a space listing/linking all proposals). >> >>> >> >> > >> >> >>> >> >> > >> Regards >> >>> >> >> > >> JB >> >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> > >> On Tue, Oct 24, 2023 at 9:22 AM Jan Kaul >> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> wrote: >> >>> >> >> > >> > >> >>> >> >> > >> > Hi all, >> >>> >> >> > >> > >> >>> >> >> > >> > I've created an issue to propose a design for a >> Materialized View Spec a while ago. After further discussion we reached a >> first draft for the spec. It would be great if you could have another look >> at the design and share your feedback. >> >>> >> >> > >> > >> >>> >> >> > >> > Here is the google doc: >> https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?usp=sharing >> >>> >> >> > >> > >> >>> >> >> > >> > Thanks in advance, >> >>> >> >> > >> > >> >>> >> >> > >> > Jan >> >
