JigaoLuo commented on PR #624:
URL: https://github.com/apache/parquet-format/pull/624#issuecomment-6562447325

   Thanks for driving this!
   
   I’ve also been following the Iceberg vector proposal, and I’d like to raise 
two points about vector's metadata for discussion.
   
   I agree that min/max statistics don’t make sense for vectors. However, there 
is other metadata that would make vector data more self-describing. It could 
live in Parquet or in Iceberg:
   1. Distance function. See the reasoning from Harsha and the related 
discussion on the Iceberg side: 
https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?disco=AAACDqu3oWI&pli=1
   2. Lineage: which embedding model or API produced the vectors.
   
   Just raising for discussion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to