Hi Gabor Thanks for your comments. The intention is to move the HadoopCatalog in test space, not to remove it. If there are contributions to bring on production grade, we can always move again :) We can add an option to HadoopCatalog to check if the backend storage is HDFS, but we have to keep the local fs option for test purposes.
About REST, it's a discussion we had. Iceberg is spec+table format, not the "core" purpose to have REST runtime/server because it means a lot in terms of features (observability, storage, auth, etc). There are several good implementations available (gravitino, nessie, ...). The proposal I made during Catalog meeting is: - Iceberg should provide a TCK (I started to sketch something about that using CatalogHandler) for both server and client - The implementations will be able to test their compatibility based on the TCK and we can have a list of "TCK verified" implementations Thoughts ? Regards JB On Tue, Jul 30, 2024 at 9:43 AM Gabor Kaszab <gaborkas...@apache.org> wrote: > Hey Iceberg Community, > > Sorry, for being late to this conversation. I just wanted to share that > I'm against deprecating HadoopCatalog or moving it to tests. Currently > Impala relies heavily on HadoopCatalog for it's own tests and I personally > find HadoopCatalog pretty handy when I just want to do some cross-engine > experiments where my data is already on HDFS and I just write a table with > engineA and see if engineB can read it and I don't want to bother with > setting up any services to serve as an Iceberg catalog (HMS for instance). > > I believe that even though we don't consider HadoopCatalog a production > grade solution as it is now, it has its benefits for lightweight > experimentation. > > - I'm +1 for keeping HadoopCatalog > - We should emphasize that HDFS is the desired storage for > HadoopCatalog (can we force this in the code?) > - Apparently, there is a part of this community that is open to add > enhancements to HadoopCatalog to bring it closer to production gradeness > (lisoda). I don't think we shouldn't block these contributions. > - If we say that REST Catalog is preferred over HadoopCatalog I think > the Iceberg project should offer its own open-source solution available for > everyone. > > Regards, > Gabor > > On Thu, Jul 25, 2024 at 9:04 PM Ryan Blue <b...@databricks.com.invalid> > wrote: > >> There are ways to use object store or file system features to do this, >> but there are a lot of variations. Building implementations and trying to >> standardize each one is a lot of work. And then you still get a catalog >> that doesn't support important features. >> >> I don't think that this is a good direction to build for the Iceberg >> project. But I also have no objection to someone doing it in a different >> project that uses the Iceberg metadata format. >> >> On Tue, Jul 23, 2024 at 5:57 PM lisoda <lis...@yeah.net> wrote: >> >>> >>> Sir, regarding this point, we have some experience. In my view, as long >>> as the file system supports atomic single-file writing, where the file >>> becomes immediately visible upon the client's successful write operation, >>> that is sufficient. We can do without the rename operation as long as the >>> file system guarantees this feature. Of course, if the object storage >>> system supports mutex operations, we can also uniformly use the rename >>> operation for committing. We can theoretically avoid the situation of >>> providing a large number of commit strategies for different file systems. >>> ---- Replied Message ---- >>> From Jack Ye<yezhao...@gmail.com> <yezhao...@gmail.com> >>> Date 07/24/2024 02:52 >>> To dev@iceberg.apache.org >>> Cc >>> Subject Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to >>> tests in 2.0 >>> If we come up with a new storage-only catalog implementation that could >>> solve those limitations and also leverage the new features being developed >>> in object storage, would that be a potential alternative strategy? so >>> HadoopCatalog users has a way to move forward with still a storage-only >>> catalog that can run on HDFS, and we can fully deprecate HadoopCatalog. >>> >>> -Jack >>> >>> On Tue, Jul 23, 2024 at 10:00 AM Ryan Blue <b...@databricks.com.invalid> >>> wrote: >>> >>>> I don't think we would want to put this in a module with other catalog >>>> implementations. It has serious limitations and is actively discouraged, >>>> while the other catalog implementations still have value as either REST >>>> back-end catalogs or as regular catalogs for many users. >>>> >>>> On Tue, Jul 23, 2024 at 9:11 AM Jack Ye <yezhao...@gmail.com> wrote: >>>> >>>>> For some additional information, we also have some Iceberg HDFS users >>>>> on EMR. Those are mainly users that have long-running Hadoop and HBase >>>>> installations. They typically refresh their installation every 1-2 years. >>>>> From my understanding, they use S3 for data storage, but metadata is kept >>>>> in the local HDFS cluster, thus HadoopCatalog works well for them. >>>>> >>>>> I remember we discussed moving all catalog implementations in the main >>>>> repo right now to a separated iceberg-catalogs repo. Could we do this move >>>>> as a part of that effort? >>>>> >>>>> -Jack >>>>> >>>>> On Tue, Jul 23, 2024 at 8:46 AM Ryan Blue <b...@databricks.com.invalid> >>>>> wrote: >>>>> >>>>>> Thanks for the context, lisoda. I agree that it's good to understand >>>>>> the issues you're facing with the HadoopCatalog. One follow up question >>>>>> that I have is what the underlying storage is. Are you using HDFS for >>>>>> those >>>>>> 30,000 customers? >>>>>> >>>>>> I think you're right that there is a challenge to migrating. Because >>>>>> there is no catalog requirement, it's hard to make sure you have all of >>>>>> the >>>>>> writers migrated. I think that means we do need to have a plan or >>>>>> recommendation for people currently using this catalog in production, but >>>>>> it also puts more pressure on us to deprecate this catalog and avoid more >>>>>> people having this problem. >>>>>> >>>>>> I think it's a good idea to make the spec change, which we have >>>>>> agreement for and to ensure that the FS catalog and table operations are >>>>>> properly deprecated to show that they should not be used. I'm not sure >>>>>> whether there is support in the community for moving the implementation >>>>>> into a new iceberg-hadoop module, but at a minimum we can't just remove >>>>>> this right away. I think that a separate iceberg-hadoop module would make >>>>>> the most sense. >>>>>> >>>>>> On Thu, Jul 18, 2024 at 11:09 PM lisoda <lis...@yeah.net> wrote: >>>>>> >>>>>>> Hi team. >>>>>>> I am not a pmc member, just a regular user. Instead of >>>>>>> discussing whether hadoopcatalog needs to continue to exist, I'd like to >>>>>>> share a more practical issue. >>>>>>> >>>>>>> We currently serve over 30,000 customers, all of whom use >>>>>>> Iceberg to store their foundational data, and all business analyses are >>>>>>> conducted based on Iceberg. However, all the Iceberg tables are >>>>>>> hadoop_catalog. At least, this has been the case since I started working >>>>>>> with our production environment system. >>>>>>> >>>>>>> In recent days, I've attempted to migrate hadoop_catalog to >>>>>>> jdbc-catalog, but I failed. We store 2PB of data, and replacing the >>>>>>> current >>>>>>> catalogues has become an almost impossible task. Users not only create >>>>>>> hadoop_catalog tables through Spark, they also continuously use >>>>>>> third-party >>>>>>> OLAP systems/FLINK and other means to write data into Iceberg in the >>>>>>> form >>>>>>> of hadoop_catalog. Given this situation, we can only continue to fix >>>>>>> hadoop_catalog and provide services to customers. >>>>>>> >>>>>>> I understand that the community wants to make a big push into >>>>>>> rest-catalog, and I agree with the direction the community is going.But >>>>>>> considering >>>>>>> that there might be a significant number of users facing similar issues, >>>>>>> can we at least retain a module similar to iceberg-hadoop to extend >>>>>>> hadoop_catalog? If it is removed, we won't be able to continue providing >>>>>>> services to customers. So, if possible, please consider this option. >>>>>>> >>>>>>> Thank you all. >>>>>>> >>>>>>> Kind regards, >>>>>>> lisoda >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> At 2024-07-19 01:28:18, "Jack Ye" <yezhao...@gmail.com> wrote: >>>>>>> >>>>>>> Thank you for bringing this up Ryan. I have been also in the camp of >>>>>>> saying HadoopCatalog is not recommended, but after thinking about this >>>>>>> more >>>>>>> deeply last night, I now have mixed feelings about this topic. Just to >>>>>>> comment on the reasons you listed first: >>>>>>> >>>>>>> * For reason 1 & 2, it looks like the root cause is that people try >>>>>>> to use HadoopCatalog outside native HDFS because there are HDFS >>>>>>> connectors >>>>>>> to other storages like S3AFileSystem. However, the norm for such usage >>>>>>> has >>>>>>> been that those connectors do not strictly follow HDFS semantics, and >>>>>>> it is >>>>>>> assumed that people acknowledge the implication of such usage and accept >>>>>>> the risk. For example, S3AFileSystem was there even before S3 was >>>>>>> strongly >>>>>>> consistent, but people have been using that to write files. >>>>>>> >>>>>>> * For reason 3, there are multiple catalogs that do not support all >>>>>>> operations (e.g. Glue for atomic table rename) and people still widely >>>>>>> use >>>>>>> it. >>>>>>> >>>>>>> * For reason 4, I see that more as a missing feature. More features >>>>>>> could definitely be developed in that catalog implementation. >>>>>>> >>>>>>> So the key question to me is, how can we prevent people from using >>>>>>> HadoopCatalog outside native HDFS. We know HadoopCatalog is popular >>>>>>> because >>>>>>> it is a storage only solution. For object storages specifically, >>>>>>> HadoopCatalog is not suitable for 2 reasons: >>>>>>> >>>>>>> (1) file write does not enforce mutual exclusion, thus cannot >>>>>>> enforce Iceberg optimistic concurrency requirement (a.k.a. cannot do >>>>>>> atomic >>>>>>> and swap) >>>>>>> >>>>>>> (2) directory-based design is not preferred in object storage and >>>>>>> will result in bad performance. >>>>>>> >>>>>>> However, now I look at these 2 issues, they are getting outdated. >>>>>>> >>>>>>> (1) object storage is starting to enforce file mutual exclusion. GCS >>>>>>> supports file generation number [1] that increments monotonically, and >>>>>>> can >>>>>>> use x-goog-if-generation-match [2] to perform atomic swap. Similar >>>>>>> feature >>>>>>> [3] exists in Azure Blob Storage. I cannot speak for the S3 team >>>>>>> roadmap. >>>>>>> But Amazon S3 is clearly falling behind in this domain, and with market >>>>>>> competition, it is very clear that similar features will come in >>>>>>> reasonably >>>>>>> near future. >>>>>>> >>>>>>> (2) directory bucket is becoming the norm. Amazon S3 announced >>>>>>> directory bucket in 2023 re:invent [4], which does not have the same >>>>>>> performance limitation even if you have very nested folders and many >>>>>>> objects in a folder. GCS also has a similar feature launched in preview >>>>>>> [5] >>>>>>> right now. Azure also already has this feature since 2021 [6]. >>>>>>> >>>>>>> With these new developments in the industry, a storage-only Iceberg >>>>>>> catalog becomes very attractive. It is simple with only one service >>>>>>> dependency. It can safely perform atomic compare-and-swap. It is >>>>>>> performant >>>>>>> without the need to worry about folder and file organization. If you >>>>>>> want >>>>>>> to add additional features for things like access control, there are >>>>>>> also >>>>>>> integrations like access grant [7] that can be integrated to do it in a >>>>>>> very scalable way. >>>>>>> >>>>>>> I know the direction in the community so far is to go with the REST >>>>>>> catalog, and I am personally a big advocate for that. However, that >>>>>>> requires either building a full REST catalog, or choosing a catalog >>>>>>> vendor >>>>>>> that supports REST. There are many capabilities that REST would unlock, >>>>>>> but >>>>>>> those are visions which I expect will take many years down the road for >>>>>>> the >>>>>>> community to continue to drive consensus and build those features. If I >>>>>>> am >>>>>>> the CTO of a small company and I just want an Iceberg data lake(house) >>>>>>> right now, do I choose REST, or do I choose (or even just build) a >>>>>>> storage-only Iceberg catalog? I feel I would actually choose the later. >>>>>>> >>>>>>> Going back to the discussion points, my current take of this topic >>>>>>> is that: >>>>>>> >>>>>>> (1) +1 for clarifying that HadoopCatalog should only work with HDFS >>>>>>> in the spec. >>>>>>> >>>>>>> (2) +1 if we want to block non-HDFS use cases in HadoopCatalog by >>>>>>> default (e.g. fail if using S3A), but we should allow a feature flag to >>>>>>> unblock the usage so that people can use it after understanding the >>>>>>> implications and risks, just like how people use S3A today. >>>>>>> >>>>>>> (3) +0 for removing HadoopCatalog from the core library. It could be >>>>>>> in a different module like iceberg-hdfs if that is more suitable. >>>>>>> >>>>>>> (4) -1 for moving HadoopCatalog to tests, because HDFS is still a >>>>>>> valid use case for Iceberg. After the measures 1-3 above, people >>>>>>> actually >>>>>>> having a HDFS use case should be able to continue to innovate and >>>>>>> optimize >>>>>>> the HadoopCatalog implementation. Although "HDFS is becoming much less >>>>>>> common", looking at GitHub issues and discussion forums, it still has a >>>>>>> pretty big user base. >>>>>>> >>>>>>> (5) In general, I propose we separate the discussion of >>>>>>> HadoopCatalog from a "storage only catalog" that also deals with other >>>>>>> object stages when evaluating it. With these latest industry >>>>>>> developments, >>>>>>> we should evaluate the direction for building a storage only Iceberg >>>>>>> catalog and see if the community has an interest in that. I could help >>>>>>> raise a thread about it after this discussion is closed. >>>>>>> >>>>>>> Best, >>>>>>> Jack Ye >>>>>>> >>>>>>> [1] >>>>>>> https://cloud.google.com/storage/docs/object-versioning#file_restoration_behavior >>>>>>> [2] >>>>>>> https://cloud.google.com/storage/docs/xml-api/reference-headers#xgoogifgenerationmatch >>>>>>> [3] >>>>>>> https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-conditional-headers-for-blob-service-operations >>>>>>> [4] >>>>>>> https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html >>>>>>> [5] https://cloud.google.com/storage/docs/buckets#enable-hns >>>>>>> [6] >>>>>>> https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace >>>>>>> [7] >>>>>>> https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-grants.html >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 18, 2024 at 7:16 AM Eduard Tudenhöfner < >>>>>>> etudenhoef...@apache.org> wrote: >>>>>>> >>>>>>>> +1 on deprecating now and removing them from the codebase with >>>>>>>> Iceberg 2.0 >>>>>>>> >>>>>>>> On Thu, Jul 18, 2024 at 10:40 AM Ajantha Bhat < >>>>>>>> ajanthab...@gmail.com> wrote: >>>>>>>> >>>>>>>>> +1 on deprecating the `File System Tables` from spec and >>>>>>>>> `HadoopCatalog`, `HadoopTableOperations` in code for now >>>>>>>>> and removing them permanently during 2.0 release. >>>>>>>>> >>>>>>>>> For testing we can use `InMemoryCatalog` as others mentioned. >>>>>>>>> >>>>>>>>> I am not sure about moving to test or keeping them only for HDFS. >>>>>>>>> Because, it leads to confusion to existing users of Hadoop catalog. >>>>>>>>> >>>>>>>>> I wanted to have it deprecated 2 years ago >>>>>>>>> <https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1647950504955309> >>>>>>>>> and I remember that we discussed it in sync that time and left it as >>>>>>>>> it is. >>>>>>>>> Also, when the user brought this up in slack >>>>>>>>> <https://apache-iceberg.slack.com/archives/C03LG1D563F/p1720075009593789?thread_ts=1719993403.208859&cid=C03LG1D563F> >>>>>>>>> recently about lockmanager and refactoring the HadoopTableOperations, >>>>>>>>> I have asked to open this discussion on the mailing list. So, that >>>>>>>>> we can conclude it once and for all. >>>>>>>>> >>>>>>>>> - Ajantha >>>>>>>>> >>>>>>>>> On Thu, Jul 18, 2024 at 12:49 PM Fokko Driesprong < >>>>>>>>> fo...@apache.org> wrote: >>>>>>>>> >>>>>>>>>> Hey Ryan and others, >>>>>>>>>> >>>>>>>>>> Thanks for bringing this up. I would be in favor of removing the >>>>>>>>>> HadoopTableOperations, mostly because of the reasons that you already >>>>>>>>>> mentioned, but also about the fact that it is not fully in line with >>>>>>>>>> the >>>>>>>>>> first principles of Iceberg (being object store native) as it uses >>>>>>>>>> file-listing. >>>>>>>>>> >>>>>>>>>> I think we should deprecate the HadoopTables to raise the >>>>>>>>>> attention of their users. I would be reluctant to move it to test to >>>>>>>>>> just >>>>>>>>>> use it for testing purposes, I'd rather remove it and replace its >>>>>>>>>> use in >>>>>>>>>> tests with the InMemoryCatalog. >>>>>>>>>> >>>>>>>>>> Regarding the StaticTable, this is an easy way to have a >>>>>>>>>> read-only table by directly pointing to the metadata. This also >>>>>>>>>> lives in >>>>>>>>>> Java under StaticTableOperations >>>>>>>>>> <https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/StaticTableOperations.java>. >>>>>>>>>> It isn't a full-blown catalog where you can list {tables,schemas}, >>>>>>>>>> update tables, etc. As ZENOTME pointed out already, it is all up to >>>>>>>>>> the >>>>>>>>>> user, for example, there is no listing of directories to determine >>>>>>>>>> which >>>>>>>>>> tables are in the catalog. >>>>>>>>>> >>>>>>>>>> is there a probability that the strategy used by HadoopCatalog is >>>>>>>>>>> not compatible with the table managed by other catalogs? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Yes, so they are different, you can see in the spec the section >>>>>>>>>> on File System tables >>>>>>>>>> <https://github.com/apache/iceberg/blob/main/format/spec.md#file-system-tables>, >>>>>>>>>> is used by the HadoopTable implementation. Whereas the other catalogs >>>>>>>>>> follow the Metastore Tables >>>>>>>>>> <https://github.com/apache/iceberg/blob/main/format/spec.md#metastore-tables> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> Fokko >>>>>>>>>> >>>>>>>>>> Op do 18 jul 2024 om 07:19 schreef NOTME ZE < >>>>>>>>>> st810918...@gmail.com>: >>>>>>>>>> >>>>>>>>>>> According to our requirements, this function is for some users >>>>>>>>>>> who want to read iceberg tables without relying on any catalogs, I >>>>>>>>>>> think >>>>>>>>>>> the StaticTable may be more flexible and clear in semantics. For >>>>>>>>>>> StaticTable, it's the user's responsibility to decide which >>>>>>>>>>> metadata of the >>>>>>>>>>> table to read. But for read-only HadoopCatalog, the metadata may be >>>>>>>>>>> decided by Catalog, is there a probability that the strategy used by >>>>>>>>>>> HadoopCatalog is not compatible with the table managed by other >>>>>>>>>>> catalogs? >>>>>>>>>>> >>>>>>>>>>> Renjie Liu <liurenjie2...@gmail.com> 于2024年7月18日周四 11:39写道： >>>>>>>>>>> >>>>>>>>>>>> I think there are two ways to do this: >>>>>>>>>>>> 1. As Xuanwo said, we refactor HadoopCatalog to be read only, >>>>>>>>>>>> and throw unsupported operation exception for other operations that >>>>>>>>>>>> manipulate tables. >>>>>>>>>>>> 2. Totally deprecate HadoopCatalog, and add StaticTable as we >>>>>>>>>>>> did in pyiceberg or iceberg-rust. >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 18, 2024 at 11:26 AM Xuanwo <xua...@apache.org> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, Renjie >>>>>>>>>>>>> >>>>>>>>>>>>> Are you suggesting that we refactor HadoopCatalog as a >>>>>>>>>>>>> FileSystemCatalog to enable direct reading from file systems like >>>>>>>>>>>>> HDFS, S3, >>>>>>>>>>>>> and Azure Blob Storage? This catalog will be read-only that don't >>>>>>>>>>>>> support >>>>>>>>>>>>> write operations. >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 18, 2024, at 10:23, Renjie Liu wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, Ryan: >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for raising this. I agree that HadoopCatalog is >>>>>>>>>>>>> dangerous in manipulating tables/catalogs given limitations of >>>>>>>>>>>>> different >>>>>>>>>>>>> file systems. But I see that there are some users who want to >>>>>>>>>>>>> read iceberg >>>>>>>>>>>>> tables without relying on any catalogs, this is also the >>>>>>>>>>>>> motivational use >>>>>>>>>>>>> case of StaticTable in pyiceberg and iceberg-rust, is there >>>>>>>>>>>>> similar things >>>>>>>>>>>>> in java implementation? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 18, 2024 at 7:01 AM Ryan Blue <b...@apache.org> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hey everyone, >>>>>>>>>>>>> >>>>>>>>>>>>> There has been some recent discussion about improving >>>>>>>>>>>>> HadoopTableOperations and the catalog based on those tables, but >>>>>>>>>>>>> we've >>>>>>>>>>>>> discouraged using file system only table (or "hadoop" tables) for >>>>>>>>>>>>> years now >>>>>>>>>>>>> because of major problems: >>>>>>>>>>>>> * It is only safe to use hadoop tables with HDFS; most local >>>>>>>>>>>>> file systems, S3, and other common object stores are unsafe >>>>>>>>>>>>> * Despite not providing atomicity guarantees outside of HDFS, >>>>>>>>>>>>> people use the tables in unsafe situations >>>>>>>>>>>>> * HadoopCatalog cannot implement atomic operations for rename >>>>>>>>>>>>> and drop table, which are commonly used in data engineering >>>>>>>>>>>>> * Alternative file names (for instance when using metadata >>>>>>>>>>>>> file compression) also break guarantees >>>>>>>>>>>>> >>>>>>>>>>>>> While these tables are useful for testing in non-production >>>>>>>>>>>>> scenarios, I think it's misleading to have them in the core >>>>>>>>>>>>> module because >>>>>>>>>>>>> there's an appearance that they are a reasonable choice. I >>>>>>>>>>>>> propose we >>>>>>>>>>>>> deprecate the HadoopTableOperations and HadoopCatalog >>>>>>>>>>>>> implementations and >>>>>>>>>>>>> move them to tests the next time we can make breaking API changes >>>>>>>>>>>>> (2.0). >>>>>>>>>>>>> >>>>>>>>>>>>> I think we should also consider similar fixes to the table >>>>>>>>>>>>> spec. It currently describes how HadoopTableOperations works, >>>>>>>>>>>>> which does >>>>>>>>>>>>> not work in object stores or local file systems. HDFS is becoming >>>>>>>>>>>>> much less >>>>>>>>>>>>> common and I propose that we note that the strategy in the spec >>>>>>>>>>>>> should ONLY >>>>>>>>>>>>> be used with HDFS. >>>>>>>>>>>>> >>>>>>>>>>>>> What do other people think? >>>>>>>>>>>>> >>>>>>>>>>>>> Ryan >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Ryan Blue >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Xuanwo >>>>>>>>>>>>> >>>>>>>>>>>>> https://xuanwo.io/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Databricks >>>>>> >>>>> >>>> >>>> -- >>>> Ryan Blue >>>> Databricks >>>> >>> >> >> -- >> Ryan Blue >> Databricks >> >
