Hi Yufei, Interesting proposal. I commented in the doc.
WDYT about using Iceberg metadata to list the stage files (in manifests)? Thanks, Dmitri. On Thu, Dec 5, 2024 at 6:21 PM Yufei Gu <flyrain...@gmail.com> wrote: > Hi Folks, > > Polaris has become a cornerstone for managing structured data across > diverse processing engines, ensuring high performance and reliability. To > further enhance its capabilities, we propose extending Polaris to support > unstructured data. This will enable it to handle a broader range of data > types efficiently, meeting the growing demands of AI/ML and other > data-intensive applications. > > You can find the proposal here: Proposal: Unstructured Data Support in > Polaris > < > https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?usp=sharing > > > > We welcome your feedback and insights. Please take a look and share your > thoughts. At the same time, we will make a POC pretty soon to test the idea > and gather more feedback. > > > Yufei >
