aaa3824956 opened a new issue, #355: URL: https://github.com/apache/doris-spark-connector/issues/355
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version Spark: 3.5.6 Old Connector: spark-doris-connector-2_1.3.2 (V1 API) New Connector: spark-doris-connector-spark-3.5_25.2.0 (V2 API) Doris: 2.1.x ### What's Wrong? When using the new connector, the same SQL that worked with the old connector now fails with the following error: text Error in query: LEGACY store assignment policy is disallowed in Spark data source V2. Please set the configuration spark.sql.storeAssignmentPolicy to other Values. Our SQL relies on implicit type conversions (e.g., writing False into a numeric column, or a string literal into a date column). In the old connector (V1), Spark allowed the LEGACY store assignment policy to handle these conversions automatically. After upgrading to the V2-based connector, this is no longer possible, and setting spark.sql.storeAssignmentPolicy to ANSI or STRICT results in type mismatch errors. ### What You Expected? We would like to know: 1.Is there any connector-specific configuration that can enable the same level of implicit type conversion as the old V1 connector? 2.If not, does the community plan to add such a compatibility mode to help users transition? 3.If this is an intentional limitation, please clarify it in the documentation so users are aware before upgrading. ### How to Reproduce? 1.Create a temporary view using the new connector. 2.Execute an INSERT statement with values that do not strictly match the target column types (e.g., boolean to numeric, string to date). 3.Observe the error. ### Anything Else? What We Have Tried？ Setting spark.sql.storeAssignmentPolicy=ANSI or STRICT → fails due to type mismatch. Reverting to the old connector → works as expected. Modifying SQL to add explicit casts → works, but requires changes to many existing ETL jobs. Why We Consider This a Problem？ Behavioral regression: The same application code that worked with the old connector fails after upgrading, with no obvious migration path. Lack of compatibility option: Even though Data Source V2 restricts the use of LEGACY policy, the connector could potentially provide its own type coercion or a configuration flag to mimic the old behavior, ensuring a smoother upgrade for users. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
