vinooganesh commented on issue #23: DataFile External Identifier Field URL: https://github.com/apache/incubator-iceberg/issues/23#issuecomment-447057968 So I see 2 uses for this: (1) Identifier shared across files - let's say that I have a bunch of files that make up a RDD that come from different systems (for example, let's say we're a bank and we have a bunch of customers from M&A, wealth management, etc..) and they each give us a list of their customers that we union together to make up the RDD. Let's say one of them is corrupt / doesn't work, and thus our RDD is in a bad state. Having this identifier would allow us to link the file to the source system the the file came from and allow us to talk to the data owners to remedy the issue. (2) Unique identifier on a per file basis - In this situation, we simply want a way to retrieve some static information on a per file basis outside of the path itself. For example, I think of this as something like the `Descriptor` in the SSTable object in Cassandra (https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L62). The object does include the directory (https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L56), but also includes something like the FormatType (https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L62 - an enum for different SSTable formats). Taking a step back, it does sound a bit like metadata, but I do think having this type of information somewhere is important. Cassandra kind of hacks around it by encoding things like the SSTable version in the name itself (the path contains it), but we don't really have a similar way to retrieve this type of information without this identifier.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
