Hi,
You've got two spark sessions up and running (and given Spark SQL uses
Derby-managed Hive MetaStock hence the issue)
Please don't start spark-submit from inside bin. Rather bin/spark-submit...
Jacek
On 26 Apr 2017 1:57 a.m., "Afshin, Bardia" <bardia.afs...@capitalone.com>
wrote:
I’m having issues when I fire up pyspark on a fresh install.
When I submit the same process via *spark-submit* it works.
Here’s a dump of the trace:
at org.apache.spark.sql.SparkSession.sessionState(
SparkSession.scala:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(
ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.
java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Failed to create database 'metastore_db',
see the next exception for details.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown
Source)
at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown
Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown
Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(
BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
... 92 more
Caused by: ERROR XJ041: Failed to create database 'metastore_db', see the
next exception for details.
at
org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.
wrapArgsForTransportAcrossDRDA(Unknown Source)
... 108 more
Caused by: ERROR XBM0H: Directory
/home/ubuntu/spark-2.1.0-bin-hadoop2.7/bin/metastore_db
cannot be created.
at
org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at
org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.impl.services.monitor.
StorageFactoryService$10.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown
Source)
at
org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.
createPersistentService(Unknown Source)
at org.apache.derby.impl.services.monitor.FileMonitor.
createPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.
createPersistentService(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection$5.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.jdbc.EmbedConnection.
createPersistentService(Unknown Source)
... 105 more
Traceback (most recent call last):
File "/home/ubuntu/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py",
line 43, in <module>
spark = SparkSession.builder\
File "/home/ubuntu/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/session.py",
line 179, in getOrCreate
session._jsparkSession.sessionState().conf().setConfString(key, value)
File "/home/ubuntu/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.
10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/home/ubuntu/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/utils.py",
line 79, in deco
raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating
'org.apache.spark.sql.hive.HiveSessionState':"
------------------------------
The information contained in this e-mail is confidential and/or proprietary
to Capital One and/or its affiliates and may only be used solely in
performance of work or services for Capital One. The information
transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended
recipient, you are hereby notified that any review, retransmission,
dissemination, distribution, copying or other use of, or taking of any
action in reliance upon this information is strictly prohibited. If you
have received this communication in error, please contact the sender and
delete the material from your computer.
