Is it possible to move HadoopCatalog to a separate repo and let it evolve on its own?
On Thu, Jul 18, 2024 at 10:31 AM Renjie Liu <liurenjie2...@gmail.com> wrote: > Hi, Ryan: > > Thanks for raising this. I agree that HadoopCatalog is dangerous in > manipulating tables/catalogs given limitations of different file systems. > But I see that there are some users who want to read iceberg tables without > relying on any catalogs, this is also the motivational use case of > StaticTable in pyiceberg and iceberg-rust, is there similar things in java > implementation? > > On Thu, Jul 18, 2024 at 7:01 AM Ryan Blue <b...@apache.org> wrote: > >> Hey everyone, >> >> There has been some recent discussion about improving >> HadoopTableOperations and the catalog based on those tables, but we've >> discouraged using file system only table (or "hadoop" tables) for years now >> because of major problems: >> * It is only safe to use hadoop tables with HDFS; most local file >> systems, S3, and other common object stores are unsafe >> * Despite not providing atomicity guarantees outside of HDFS, people use >> the tables in unsafe situations >> * HadoopCatalog cannot implement atomic operations for rename and drop >> table, which are commonly used in data engineering >> * Alternative file names (for instance when using metadata file >> compression) also break guarantees >> >> While these tables are useful for testing in non-production scenarios, I >> think it's misleading to have them in the core module because there's an >> appearance that they are a reasonable choice. I propose we deprecate the >> HadoopTableOperations and HadoopCatalog implementations and move them to >> tests the next time we can make breaking API changes (2.0). >> >> I think we should also consider similar fixes to the table spec. It >> currently describes how HadoopTableOperations works, which does not work in >> object stores or local file systems. HDFS is becoming much less common and >> I propose that we note that the strategy in the spec should ONLY be used >> with HDFS. >> >> What do other people think? >> >> Ryan >> >> -- >> Ryan Blue >> >
