> On 10 Jan 2023, at 19:16, Dain Sundstrom <d...@iq80.com> wrote: > > 50,000 * 4 bytes ~= 200 kB, so this shouldn’t be a problem. Generally, > large values can be a problem for some compute engines, but 200 kB isn’t that > large.
Thanks for the input! What counts as ”large” in your opinion? I plan on using Spark + python on the read / process side. //hinko > >> On Jan 10, 2023, at 8:22 AM, Hinko Kocevar <hinko.koce...@ess.eu> wrote: >> >> I would like to use ORC file to hold several columns of data. One of the >> columns will be a list (array) of floats that could span 10 000 - 50 000 >> elements is length. Other columns will not be lists, but of different data >> types. >> >> Is having such long lists in any way an issue in terms of performance or >> otherwise for the ORC file? >> >> Thank you in advance! >> >> //Hinko >
