rok commented on PR #624:
URL: https://github.com/apache/parquet-format/pull/624#issuecomment-6564308362

   Hey @JigaoLuo thanks for reviewing! Discussion on points you raise would 
better fit the discusson on the ML 
([1](https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt), 
[2](https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3)). I'll 
still try to reply here, but if you want to push this further the dev@parquet 
ML is the platform you should use.
   
   > there is other metadata that would make vector data more self-describing. 
It could live in Parquet or in Iceberg:
   > 
   > 1. Distance function. See the reasoning from Harsha and the related 
discussion on the Iceberg side: 
https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?disco=AAACDqu3oWI&pli=1
   
   I am not an embeddings or vector DB expert, but wouldn't distance metric 
always an operation acting on pairs of vectors? It feels more like an 
application level concern and parquet already has a file-level metadata 
key-value metadata where such information could be stored. Or am I missing 
something here that could not be captured there?
   
   > 2. Lineage: which embedding model or API produced the vectors.
   
   Agreed that this is important information, but I'm not sure it's something 
that should live in the type. It would make parquet more self-contained, but it 
would require agreement on model identity between embedding producers which is 
out of reach for parquet. If you feel strongly about this, please sketch out 
how you imagine this would look like.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to