[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Then, @rxin and @gatorsmile . I'll make a PR to merge `maxConnection` into `numPartitions`. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 My fault. In my initial thoughts, `numPartitions` does not indicate the max allowed number of JDBC connections, and the write path does not always create the exact number of partitions. We only

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 The rational is to give a name which has a clear meaning. Also, I agreed with the following advice. > numPartitions might be not a good name for this purpose. How about

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15868 Basically right now if I were to define a table, I'd need to define both the read and the write side, using numPartitions and maxPartitions. Even if we want to have two separate configs for read vs

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 The code seems to be gone during resolving conflicts. This is the my comment at the first review. > Thank you for review, @srowen . > > First, the property numPartitions

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15868 Got it - so this is to enforce the write side: in the future please make it clear this is for the write side. Coming back to the discussion: shouldn't we just use numPartitions, given it is

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Let me find the correct code. (It's weird I attached a wrong code example.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15868 Yes - otherwise it can get pretty confusing. I'd argue the user has already explicitly defined the degree of parallelism in the JDBC source using other options. Having this maxConnections

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Hi, @rxin . Do you mean changing SQL syntax like `BROADCAST HINT` PR we tried before? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 It's read-only option. At the first commit, I reuse that option for writing. But, as you see, it changed according the review advices. --- If your project is set up for it, you can reply to

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15868 Don't they already specify "numPartitions" ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Could you give the example of existing method you mention? > the user has already explicitly defined the degree of parallelism in the JDBC source using other options. --- If your project

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15868 Hm - wouldn't a more general solution be to introduce a parallelism hint or command (basically the coalesce equivalent) in SQL? --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Thank you for review and merging, @srowen , @gatorsmile , @cloud-fan , @lichenglin ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15868 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15868 LGTM, waiting @srowen for final sign off --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Hi, @cloud-fan . If there is something to do more, please let me know again. Thank you so much. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68929/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68929/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68929/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 It seems not a test failure. ``` Traceback (most recent call last): File "./dev/run-tests-jenkins.py", line 232, in main() File "./dev/run-tests-jenkins.py", line

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68922/ Test FAILed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68917/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68917/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68922 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68922/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68917/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68905/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68905/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68905/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Thank you, @srowen ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Hi, @srowen . Could you review this PR again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/6/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #6 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/6/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68886/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68886 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68886/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #6 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/6/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68886/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68871/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68871/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Documentation and PR descriptions are updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68871/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 Yeah, please improve the document comments and PR descriptions. It sounds like this is almost ready to merge after these changes. --- If your project is set up for it, you can reply to

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Ah, right. Now always! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 > This option applies only to writing with additional coalesce operation. It is not clear. We do not always call `coalesce`. : ) --- If your project is set up for it, you can reply to

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 For document, may I add more description about the behavior to help the user understanding? **BEFORE** > The number of JDBC connections, which specifies the maximum number of

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Thank you for review again, @cloud-fan , @liancheng , and @srowen . I had the same opinion with @srowen . Also, let's think about the background of this issue. Currently, in

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15868 Conceptually, it does seem nicer to resolve this with a connection pool. There is however some complexity in implementing our own connection pool. I think you've pointed out a key issue:

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread lichenglin
Github user lichenglin commented on the issue: https://github.com/apache/spark/pull/15868 In fact,I have idea . We can change the param to "connections pool size on each executor",every partition task on the same executor use the same connection pool to save data. we can

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-18 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15868 I see, because a user can specify this option through SQL only? Ok, then that logic makes sense to me. It's still kinda a coarse way to achieve a limit on concurrent connections. Some

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15868 > We cannot call repartition in SQL language environment This is a good point, we need to provide a workaround for sql users, and this PR did. --- If your project is set up for it, you

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Sorry for late response, @gatorsmile and @srowen . --- > However, when users use the DataFrameWriter, this is not available. Maybe, in the future, we can add another option

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 FYI, SQOOP2 also [adds a feature](http://blog.cloudera.com/blog/2012/01/apache-sqoop-highlights-of-sqoop-2/) for limiting the max number of connections: > By limiting the total number

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 Yeah, although we change the operator from `repartition` to `coalesce`, but yeah still costly. When users use JDBC source to connect RDBMS, the max number of concurrent connections is

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15868 OK I have an overall question about this approach. The point is to limit concurrent JDBC connections? this does that, but, requires repartitioning data to accomplish that, which is expensive. One

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68788/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 LGTM. Maybe @srowen and @cloud-fan can double check the code. BTW, in the code path of regular SQL, we are able to explain the INSERT/CTAS. However, when users use the DataFrameWriter,

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Thank you, @gatorsmile . I added the negative test case. But, for the positive test case, `DataFrameWriter` keeps it `private` and `save` function returns `Unit`. ```scala private

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68788/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 Yeah, add one negative test case and one positive test case for the newly added parm. For example, below is a negative test case. ``` val df =

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Could you review this PR again, @gatorsmile? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68708/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68708/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 I rebased and squashed to resolve conflicts. The PR is updated the followings. - Revert `DataSource.scala` change. - Rename `JDBC_MAX_CONNECTION` to `JDBC_MAX_CONNECTIONS`. - Add

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68708/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Hi, @gatorsmile . Indeed, I really want to add a test case for this. Could you give me some advice how to test this kind of features? --- If your project is set up for it, you can

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Just FYI, the only failure is irrelevant. ``` [info] StateStoreSuite: [info] - maintenance *** FAILED *** (10 seconds, 120 milliseconds) [info] The code passed to eventually

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68696/ Test FAILed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68696/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 Could we have a test case for this new parameter? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 LGTM except one minor comment. Also CC @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68696/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68672/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68672/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68672/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-15 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Hi, @srowen and @gatorsmile . How do you think about the updated PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68647/ Test PASSed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68647 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68647/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 Thank you for retriggering, @gatorsmile . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68647/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15868 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68641/ Test FAILed. ---

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15868 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68641/consoleFull)** for PR 15868 at commit

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15868 @gatorsmile , I addressed all comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15868: [SPARK-18413][SQL] Add `maxConnections` JDBCOption

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15868 **[Test build #68641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68641/consoleFull)** for PR 15868 at commit

  1   2   >