fanyue-xia commented on PR #45971:

   > Thanks for the effort! This really requires some deep understanding of 
spark internals...
   > There is still one important concern, that the golden file size is too 
big. I looked a bit, it seems that the largest golden file is ~7MB. We should 
find a way to limit the file size to < 10MB.
   > One improvement I can see is that, here you are storing both the rows and 
partition ids, but we don't need to store rows.
   > Instead, we store the random seed, and regenerate the random rows in the 
check. By doing this we only need to store the seed, the schemas, and for each 
   > 1. partition ids, and
   > 2. numRows
   > Now golden file size should be much smaller.
   > This means that we trust `RandomDataGenerator` to generate the same row 
for the same seed everytime. The code hasn't been touched like 10 years so I 
think this should be safe. Although unlikely, when people really need to touch 
that later, they'll notice this test failure.
   I have concerned that the seed might behave differently across runs/on 
different machines. Talked to @HeartSaVioR  about it, he mentioned that he 
isn't sure whether it is the case; it’s safer to just store the generated 
inputs; if there is any difference in any random generation, it’s going to be 
uneasy to find.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail:

For queries about this service, please contact Infrastructure at:

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to