HyukjinKwon commented on PR #58597: URL: https://github.com/apache/spark/pull/58597#issuecomment-5582102497
Closing after review. The data whose declared feature count this bounds is read by the submitted application itself, and a single global cap on the read-back path also rejects Spark's own intermediate results (e.g. HashingTF's default of 262144 written into column metadata), so a configurable global cap is not the right control here. The one site with a genuinely unbounded allocation (AttributeGroup) can be revisited on its own with an always-on sanity check if needed. Not proceeding with this approach; closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
