[
https://issues.apache.org/jira/browse/DRILL-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432339#comment-17432339
]
ASF GitHub Bot commented on DRILL-8016:
---------------------------------------
dzamo commented on pull request #2342:
URL: https://github.com/apache/drill/pull/2342#issuecomment-948448967
@cgivre, thank you. I've updated the docs, removed the dead code and
uncommented the line which depends on JDBC writer. That last step means I
can't push something that builds until we've merged JDBC writer and I've
rebased, so I'll keep this PR on hold until then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Default to lazy outbound connections for storage-jdbc and storage-splunk
> ------------------------------------------------------------------------
>
> Key: DRILL-8016
> URL: https://issues.apache.org/jira/browse/DRILL-8016
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - JDBC
> Affects Versions: 1.19.0
> Reporter: James Turton
> Assignee: James Turton
> Priority: Major
> Fix For: 1.20.0
>
>
> Currently we are eager about initiating outbound JDBC
> connections, bringing up 10 per storage config per drillbit. For
> example, if a user creates 3 storage configs pointing to a single DBMS
> (the configs differing in their DB path and credentials, say) on a
> cluster of 5 drillbits then we'll bring up 10x3x5 = 150 connections as
> soon as we can and try to keep them up permanently. The fixed pool size
> of 10 is a default we picked up from HikariCP which surely set it with
> application servers in mind.
> We've had a report from the field of a MySQL server declining to provide
> said 150 connections, leaving the Drill user unable to proceed.
> Additionally, as you can imagine, almost all 150 connections will be
> idle most of the time for typical Drill cluster workloads. Furthermore,
> while connections pools are ubiquitous in the OLTP world they are rare
> in the OLAP world where the cost of creating and destroying them is
> negligible compared to the cost of a single user query, while the
> benefits of per-user access control, resource management and session
> management which they bring over shared pools are valuable. Bringing
> these latter benefits to Drill's outbound JDBC connections is not in the
> scope of this email, the point made is in only "traditionally, OLAP
> environments have avoided connection pools because the losses far
> outweigh the gains".
> In light of the above I suggest that we transition from eager to lazy
> outbound JDBC connections, more like Apache Spark (I'm told). I propose
> initially that we only change our *default* HikariCP configuration to
> maintain small, finitely scalable pools (e.g. baseline 1, up to 10)
> instead of fixed pools. The HikariCP configuration is already
> overridable today for users that prefer the current eager connection
> behaviour.
>
> http://mail-archives.apache.org/mod_mbox/drill-dev/202110.mbox/browser
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
