[GitHub] spark issue #9973: [SPARK-11989][SQL] Only use commit in JDBC data source if...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/9973 Checked it with @huaxingao who worked for JDBC driver team before. Yeah, we are unable to do it using JDBC. In my previous team, we did it using the native connection methods instead of JDBC. It s

[GitHub] spark issue #9973: [SPARK-11989][SQL] Only use commit in JDBC data source if...

2016-10-13 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/9973 Oh, can you really have a transaction across connections somehow? I didn't think that was possible in general. I agree that this is really the ideal behavior but don't know how to implement it in JDBC

[GitHub] spark issue #9973: [SPARK-11989][SQL] Only use commit in JDBC data source if...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/9973 The basic problem is multiple connections work on the same transaction. It is doable but might not be applicable as a general JDBC data source connector. Let us keep it as an open problem. If nece

[GitHub] spark issue #9973: [SPARK-11989][SQL] Only use commit in JDBC data source if...

2016-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/9973 You're right but lots of output semantics are per partition. I don't think we can do the update in one transaction no matter what. This improves the behavior in many cases so is worthwhile behavior bu