sasmith-14 commented on issue #884: URL: https://github.com/apache/wayang/issues/884#issuecomment-6882781093
Hi @zkaoudi, Thanks for taking a look! I think you're right that the line is correct when the dataset size is known. The case I was worried about is when it isn't: `datasetSize` defaults to `UNKNOWN_DATASET_SIZE` (0), and `DataQuanta.sample(...)` leaves it at 0 unless the caller passes a size. In that case `sampleSize >= 0` is always true, so if I'm reading it right the operator returns the whole input even for a small sample. As a small example, a reservoir sample of 5 over 10 elements with no dataset size set returns all 10 instead of 5, which is what the test I added checks. I noticed `JavaRandomSampleOperator` already guards this with `isDataSetSizeKnown()` and falls back to the input size when it's unknown, so I tried to match that in the reservoir operator. I may well be missing something about how datasetSize gets set in practice though, so please let me know if you'd prefer a different approach or if this isn't actually a problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
