Nice, just saw that. We are adding the definitions as a part of ttps:// github.com/apache/iceberg/pull/9695, we can help review the PRs listed here and then update the OpenAPI spec accordingly.
-Jack On Wed, Feb 14, 2024 at 4:12 PM Steven Wu <stevenz...@gmail.com> wrote: > @Ryan, the JSON serialization is also used by Flink for checkpoint state. > so it is not purely a REST API thing. > > @Jack, Ryan also had the same suggestion in the PR comment. I have updated > the naming > > On Wed, Feb 14, 2024 at 4:08 PM Jack Ye <yezhao...@gmail.com> wrote: > >> > It would fail if the FileScanTask is some other implementation (like >> StaticDataTask). >> Actually we faced exactly the same issue, and we have an internal patch >> to fix the parser for that. +1 for the proposal. >> >> For the type names, can we come up with a different name from " >> base-file-task"? "base" is very Java abstract class specific. In fact, >> the StaticDataTask is not really scanning a file anyway, maybe we should >> just call these like file-scan-task, data-task, etc.? >> >> Best, >> Jack Ye >> >> >> On Wed, Feb 14, 2024 at 4:01 PM Ryan Blue <b...@tabular.io> wrote: >> >>> Thanks, Steven! Looks like the right direction to add other task types >>> with their own serialization. >>> >>> I hadn't realized that these were in the table spec and not just the >>> REST spec. What do you think about keeping JSON serialization that isn't >>> part of table metadata in the REST spec? I'm actually pretty happy with >>> OpenAPI for defining our JSON structures, so I think this would be easier >>> in the REST spec. I would also consider an OpenAPI extension to the table >>> spec for JSON objects since it is pretty easy to work with and does a good >>> job defining the metadata. >>> >>> Ryan >>> >>> On Wed, Feb 14, 2024 at 3:48 PM Steven Wu <stevenz...@gmail.com> wrote: >>> >>>> The first linked reference is the PR for spec update. >>>> >>>> [3] https://github.com/apache/iceberg/pull/9728 >>>> >>>> On Wed, Feb 14, 2024 at 3:36 PM Steven Wu <stevenz...@gmail.com> wrote: >>>> >>>>> We just ran out of time and didn't get a chance to discuss this in the >>>>> community sync meeting today. Hence, I am raising the discussion here. >>>>> >>>>> We added JSON parsers for content file and file scan task a year ago >>>>> [1]. Recently, I just realized the implementation only handles >>>>> BaseFileScanTask. It would fail if the FileScanTask is some other >>>>> implementation (like StaticDataTask). >>>>> >>>>> Eduard, Anton, and I have been discussing a solution in issue-9597 >>>>> [2]. We reached a consensus that we need to define a new `task-type` enum >>>>> field to indicate the implementation class/type [3]. For backward >>>>> compatibility, the lack of this new `task-type` field should be >>>>> interpreted as `base-file-task`. >>>>> >>>>> Since this is a spec change, Anton suggested more visibility. Hence I >>>>> am starting this discussion thread. >>>>> >>>>> [1] https://github.com/apache/iceberg/pull/6934 >>>>> [2] https://github.com/apache/iceberg/issues/9597 >>>>> [3] https://github.com/apache/iceberg/pull/9728 >>>>> >>>> >>> >>> -- >>> Ryan Blue >>> Tabular >>> >>
