[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user Shawon91sust commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-135968555 How to work the patch ? I am newbie in spark and facing the same problem. Can anyone tell me how to work this through. I mean how to resolve the suitable JDBC driver not found problem ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user wangxj1127 commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-111075271 Later on I found by adding spark.executor.extraClassPath property with value pointing to the jar works. But as Spark document below, the best way is to edit the compute-classpath.sh. Don't know why it doesn't work for me. https://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other-databases --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user wangxj1127 commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-111014423 Hi, What is the workaround for this issue if I am using Spark 1.3.1? I have modified the compute-classpath.sh for appeding the mysql-connector jar, but still not working in yarn-cluster or yan-client mode. Only local mode works for me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-98236755 @marmbrus @SlavikBaranov @rxin this break the build with jdk6 [error] /home/tgraves/tgravescs_spark/sql/core/src/main/scala/org/apache/spark/sql/jdbc/jdbc.scala:198: value getParentLogger is not a member of java.sql.Driver [error] override def getParentLogger: java.util.logging.Logger = wrapped.getParentLogger [error] ^ java.sql.Driver.getParentLogger doesn't exist in jdk6, only jdk7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-98237772 https://issues.apache.org/jira/browse/SPARK-7312 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-98240182 @tgravescs, thanks for the heads up. We are taking a look and will revert the patch if there isn't an easy fix. /cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-98251395 #5847 tries to fix it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user SlavikBaranov commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97683350 Thanks for comments, fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97715156 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97684830 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97684816 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97685088 [Test build #31394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31394/consoleFull) for PR 5782 at commit [`510c43f`](https://github.com/apache/spark/commit/510c43f691f4b3901ca877585e25b28121a0cf36). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97684199 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97715129 [Test build #31394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31394/consoleFull) for PR 5782 at commit [`510c43f`](https://github.com/apache/spark/commit/510c43f691f4b3901ca877585e25b28121a0cf36). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch **adds the following new dependencies:** * `jaxb-api-2.2.7.jar` * `jaxb-core-2.2.7.jar` * `jaxb-impl-2.2.7.jar` * `pmml-agent-1.1.15.jar` * `pmml-model-1.1.15.jar` * `pmml-schema-1.1.15.jar` * This patch **removes the following dependencies:** * `activation-1.1.jar` * `jaxb-api-2.2.2.jar` * `jaxb-impl-2.2.3-1.jar` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97715160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31394/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5782 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-98020645 This is very clever. Thanks! I ran the docker based mysql and postgres tests manually. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97635277 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97635289 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97635442 [Test build #31370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31370/consoleFull) for PR 5782 at commit [`b2a727c`](https://github.com/apache/spark/commit/b2a727c7ed4412c9b9d42b75c5a52369bd9ac43e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5782#discussion_r29399631
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/jdbc.scala ---
@@ -179,4 +184,63 @@ package object jdbc {
}
}
+
+ private [sql] case class DriverWrapper(wrapped: Driver) extends Driver {
+override def acceptsURL(url: String): Boolean = wrapped.acceptsURL(url)
+
+override def jdbcCompliant(): Boolean = wrapped.jdbcCompliant()
+
+override def getPropertyInfo(url: String, info: Properties):
Array[DriverPropertyInfo] = {
+ wrapped.getPropertyInfo(url, info)
+}
+
+override def getMinorVersion: Int = wrapped.getMinorVersion
+
+override def getParentLogger: java.util.logging.Logger =
wrapped.getParentLogger
+
+override def connect(url: String, info: Properties): Connection =
wrapped.connect(url, info)
+
+override def getMajorVersion: Int = wrapped.getMajorVersion
+ }
+
+ /**
+ * java.sql.DriverManager is always loaded by bootstrap classloader,
+ * so it can't load JDBC drivers accessible by Spark ClassLoader.
+ *
+ * To solve the problem, drivers from user-supplied jars are wrapped
+ * into thin wrapper.
+ */
+ private [sql] object DriverRegistry extends Logging {
+
+val wrapperMap: mutable.Map[String, DriverWrapper] = mutable.Map.empty
--- End diff --
private
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97635113 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5782#discussion_r29399579
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/jdbc.scala ---
@@ -179,4 +184,63 @@ package object jdbc {
}
}
+
+ private [sql] case class DriverWrapper(wrapped: Driver) extends Driver {
--- End diff --
this doesn't need to be a case class, does it? Looks like you can just
slightly change the pattern matching down below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5782#discussion_r29399610
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/jdbc.scala ---
@@ -179,4 +184,63 @@ package object jdbc {
}
}
+
+ private [sql] case class DriverWrapper(wrapped: Driver) extends Driver {
+override def acceptsURL(url: String): Boolean = wrapped.acceptsURL(url)
+
+override def jdbcCompliant(): Boolean = wrapped.jdbcCompliant()
+
+override def getPropertyInfo(url: String, info: Properties):
Array[DriverPropertyInfo] = {
+ wrapped.getPropertyInfo(url, info)
+}
+
+override def getMinorVersion: Int = wrapped.getMinorVersion
+
+override def getParentLogger: java.util.logging.Logger =
wrapped.getParentLogger
+
+override def connect(url: String, info: Properties): Connection =
wrapped.connect(url, info)
+
+override def getMajorVersion: Int = wrapped.getMajorVersion
+ }
+
+ /**
+ * java.sql.DriverManager is always loaded by bootstrap classloader,
+ * so it can't load JDBC drivers accessible by Spark ClassLoader.
+ *
+ * To solve the problem, drivers from user-supplied jars are wrapped
+ * into thin wrapper.
+ */
+ private [sql] object DriverRegistry extends Logging {
+
+val wrapperMap: mutable.Map[String, DriverWrapper] = mutable.Map.empty
+
+val lock = new ReentrantLock()
+
+def register(className: String): Unit = {
+ val cls = Utils.getContextOrSparkClassLoader.loadClass(className)
+ if (cls.getClassLoader == null) {
+logTrace(s$className has been loaded with bootstrap ClassLoader,
wrapper is not required)
+ } else if (wrapperMap.get(className).isDefined) {
+logTrace(sWrapper for $className already exists)
+ } else {
+lock.lock()
--- End diff --
given this is unlikely to be a performance bottleneck, why not just use a
synchronized block?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97654329 [Test build #31370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31370/consoleFull) for PR 5782 at commit [`b2a727c`](https://github.com/apache/spark/commit/b2a727c7ed4412c9b9d42b75c5a52369bd9ac43e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97654336 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97654337 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31370/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed java.sql.SQLException...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5782#issuecomment-97457751 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
