HeartSaVioR commented on PR #54299: URL: https://github.com/apache/spark/pull/54299#issuecomment-3895855137
Just to leave a history about closing the PR (to avoid confusion/conflict). We found the PR submitted prior than this - https://github.com/apache/spark/pull/54254. The PR was missing when @viirya worked on this, since it didn't follow the procedure of adding JIRA title into PR title. Also the intention of the JIRA ticket is to disallow assigning nested column to be an event time column, because the caller side finding event time column from schema only performs scanning against top-level columns. This PR avoids the query to fail but also event time column isn't recognizable, which could be a bigger issue. The other PR handles it properly. It might wonder why we don't allow nested column to be event time column - I'm not sure whether this is technically feasible or not, but at least we will need to expand the scan from top-level columns to all nested columns, which I see some impacts from such a change. Maybe something we need some evidence to believe that the impact won't be outstanding, before making change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
