Hi
I am writing a huge (~100GB) Parquet file and running into the following
exception. After reading through some forums, I have changed the BlockSize
and Page Size respectively and set them to 128MB and 1 / 2 MB and I am
still running into the same issue.
I would to understand if the Block Size and Page Size could be impacting
the writing of the large file and any possible reasons for the exception
seen below ?
An exception occurred: java.lang.RuntimeException java.io.IOException: The
file being written is in an invalid state. Probably caused by an error
thrown previously. Current state: COLUMN
--
Srinivas
(*-*)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
You have to grow from the inside out. None can teach you, none can make you
spiritual.
-Narendra Nath Dutta(Swamy Vivekananda)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
