PetarVasiljevic-DB commented on PR #48625: URL: https://github.com/apache/spark/pull/48625#issuecomment-2457281580
Maybe we're not on the same page here but this is what currently happens in Spark: - If array column on Postgres has such values that dimension of these values across all rows is same then job will be stable. - If array column on Postgres has such values that dimension of these values across all rows are not same, the job won't be stable. With the proposed fix, behaviour stays the same: - If array_ndims is same for all rows, job will be stable - If array_ndims returns multiple values across all rows, job won't be stable. Currently, jobs that are reading from CTAS tables will always fail. Have I misunderstood the term "stability of job" here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
