Re: creating database issue

[bitfox]

And, I can't start spark-sql shell, the error as below.
Does this mean I need to install Hive on local machine?

Caused by: java.sql.SQLException: Failed to start database 
'metastore_db' with class loader 
jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27, see the next 
exception for details.
	at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
	at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)

at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown 
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.(Unknown Source)
at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown 
Source)

at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.(BoneCP.java:416)
... 89 more
Caused by: ERROR XJ040: Failed to start database 'metastore_db' with 
class loader jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27, 
see the next exception for details.
	at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
	at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown 
Source)

... 105 more


Thanks.

On 2021/12/8 9:28, bitfox wrote:

Hello

This is just a standalone deployment for testing purpose.
The version:
Spark 3.2.0 (git revision 5d45a415f3) built for Hadoop 3.3.1
Build flags: -B -Pmesos -Pyarn -Pkubernetes -Psparkr -Pscala-2.12 
-Phadoop-3.2 -Phive -Phive-thriftserver


I just started one master and one worker for the test.

Thanks


On 2021/12/8 9:15, Qian Sun wrote:
   It seems to be a hms question. Would u like to provide the 
information about spark version, hive version and spark application 
configuration?


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org



-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: creating database issue

[bitfox]

Hello

This is just a standalone deployment for testing purpose.
The version:
Spark 3.2.0 (git revision 5d45a415f3) built for Hadoop 3.3.1
Build flags: -B -Pmesos -Pyarn -Pkubernetes -Psparkr -Pscala-2.12 
-Phadoop-3.2 -Phive -Phive-thriftserver


I just started one master and one worker for the test.

Thanks


On 2021/12/8 9:15, Qian Sun wrote:

   It seems to be a hms question. Would u like to provide the information about 
spark version, hive version and spark application configuration?


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: creating database issue

[Qian Sun]

Hi,
 
  It seems to be a hms question. Would u like to provide the information about 
spark version, hive version and spark application configuration?

Best

> 2021年12月8日 上午9:04，bitfox  写道：
> 
> sorry I am newbie to spark.
> 
> When I created a database in pyspark shell following the book content of 
> learning spark 2.0, it gets:
> 
> >>> spark.sql("CREATE DATABASE learn_spark_db")
> 21/12/08 09:01:34 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout 
> does not exist
> 21/12/08 09:01:34 WARN HiveConf: HiveConf of name hive.stats.retries.wait 
> does not exist
> 21/12/08 09:01:39 WARN ObjectStore: Version information not found in 
> metastore. hive.metastore.schema.verification is not enabled so recording the 
> schema version 2.3.0
> 21/12/08 09:01:39 WARN ObjectStore: setMetaStoreSchemaVersion called but 
> recording version is disabled: version = 2.3.0, comment = Set by MetaStore 
> pyh@185.213.174.249
> 21/12/08 09:01:40 WARN ObjectStore: Failed to get database default, returning 
> NoSuchObjectException
> 21/12/08 09:01:40 WARN ObjectStore: Failed to get database global_temp, 
> returning NoSuchObjectException
> 21/12/08 09:01:40 WARN ObjectStore: Failed to get database learn_spark_db, 
> returning NoSuchObjectException
> 
> Can you point to me where is wrong?
> 
> Thanks.
> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> 


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

creating database issue

[bitfox]

sorry I am newbie to spark.

When I created a database in pyspark shell following the book content of 
learning spark 2.0, it gets:


>>> spark.sql("CREATE DATABASE learn_spark_db")
21/12/08 09:01:34 WARN HiveConf: HiveConf of name 
hive.stats.jdbc.timeout does not exist
21/12/08 09:01:34 WARN HiveConf: HiveConf of name 
hive.stats.retries.wait does not exist
21/12/08 09:01:39 WARN ObjectStore: Version information not found in 
metastore. hive.metastore.schema.verification is not enabled so 
recording the schema version 2.3.0
21/12/08 09:01:39 WARN ObjectStore: setMetaStoreSchemaVersion called but 
recording version is disabled: version = 2.3.0, comment = Set by 
MetaStore pyh@185.213.174.249
21/12/08 09:01:40 WARN ObjectStore: Failed to get database default, 
returning NoSuchObjectException
21/12/08 09:01:40 WARN ObjectStore: Failed to get database global_temp, 
returning NoSuchObjectException
21/12/08 09:01:40 WARN ObjectStore: Failed to get database 
learn_spark_db, returning NoSuchObjectException


Can you point to me where is wrong?

Thanks.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: start-history-server.sh doesn't survive system reboot. Recommendation?

[Mich Talebzadeh]

One way would be to cron the startup script


   1. enable cron on the host and Unix login that runs
$SPARK_HOME/sbin/start-history-server.sh
   where $SPARK_HOME is set in below cron to absolute value say /opt/spark.
   You can run it as non root, putting it in the crontab of the
   corresponding user
   2. crontab -e
   3. Add the following to the cronfile
   4. @reboot (/opt/spark/sbin/start-history-server.sh >
   /var/tmp/start-history-server_"`date +%Y%m%d_%H%M%S`.log" 2>&1)
   5. crontab -l to verify that it is added OK to the cronfile.

The other way would be to add it to startup in /etc/init.d and create
symlink to /etc/rc.d. Do a Google search for it


HTH


   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 7 Dec 2021 at 21:23, James Yu  wrote:

> Hi Users,
>
> We found that the history server launched by using the
> "start-history-server.sh" command does not survive system reboot.  Any
> recommendation of making it always up even after reboot?
>
> Thanks,
>
> James
>

Re: start-history-server.sh doesn't survive system reboot. Recommendation?

[Sean Owen]

The scripts just launch the processes. To make any process restart on
system restart, you would need to set it up as a system service (i.e.
controlled by 'service'). That's easy enough but out of scope for Spark
itself.

On Tue, Dec 7, 2021 at 3:23 PM James Yu  wrote:

> Hi Users,
>
> We found that the history server launched by using the
> "start-history-server.sh" command does not survive system reboot.  Any
> recommendation of making it always up even after reboot?
>
> Thanks,
>
> James
>

start-history-server.sh doesn't survive system reboot. Recommendation?

[James Yu]

Hi Users,

We found that the history server launched by using the 
"start-history-server.sh" command does not survive system reboot.  Any 
recommendation of making it always up even after reboot?

Thanks,

James