I am now seeing this error.
15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang <zzh...@hortonworks.com> wrote: > You can do it in $SPARK_HOME/conf/spark-defaults.con > > spark.driver.extraJavaOptions -XX:MaxPermSize=512m > > Thanks. > > Zhan Zhang > > > On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: > > Where and how do i pass this or other JVM argument ? > -XX:MaxPermSize=512m > > On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang <zzh...@hortonworks.com> > wrote: > >> I solve this by increase the PermGen memory size in driver. >> >> -XX:MaxPermSize=512m >> >> Thanks. >> >> Zhan Zhang >> >> On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: >> >> I am facing same issue, posted a new thread. Please respond. >> >> On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang <zzh...@hortonworks.com> >> wrote: >> >>> Hi Folks, >>> >>> I am trying to run hive context in yarn-cluster mode, but met some >>> error. Does anybody know what cause the issue. >>> >>> I use following cmd to build the distribution: >>> >>> ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 >>> >>> 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: >>> YarnClusterScheduler.postStartHook done >>> 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering >>> block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, >>> BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) >>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE >>> IF NOT EXISTS src (key INT, value STRING) >>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed >>> 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store >>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >>> 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize >>> called >>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>> datanucleus.cache.level2 unknown - will be ignored >>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not >>> present in CLASSPATH (or one of dependencies) >>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not >>> present in CLASSPATH (or one of dependencies) >>> 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object >>> pin classes with >>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>> 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, >>> assuming we are not on mysql: Lexical error at line 1, column 5. >>> Encountered: "@" (64), after : "". >>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>> "embedded-only" so does not have its own datastore table. >>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>> "embedded-only" so does not have its own datastore table. >>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>> "embedded-only" so does not have its own datastore table. >>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>> "embedded-only" so does not have its own datastore table. >>> 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore >>> 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not >>> found in metastore. hive.metastore.schema.verification is not enabled so >>> recording the schema version 0.13.1aa >>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in >>> metastore >>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in >>> metastore >>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in >>> admin role, since config is empty >>> 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at >>> this point. hive.execution.engine=mr. >>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not >>> creating a lock manager >>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE >>> IF NOT EXISTS src (key INT, value STRING) >>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed >>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse >>> start=1421190003030 end=1421190003031 duration=1 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis >>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src >>> position=27 >>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : >>> db=default tbl=src >>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src >>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default >>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>> ip=unknown-ip-addr cmd=get_database: default >>> 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed >>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=semanticAnalyze >>> start=1421190003031 end=1421190003406 duration=375 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: >>> Schema(fieldSchemas:null, properties:null) >>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile >>> start=1421190002998 end=1421190003416 duration=418 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=Driver.execute >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE IF NOT >>> EXISTS src (key INT, value STRING) >>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit >>> start=1421190002995 end=1421190003421 duration=426 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=task.DDL.Stage-0 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe for >>> table src >>> 15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG method=Driver.execute >>> start=1421190003416 end=1421190005498 duration=2082 >>> from=org.apache.hadoop.hive.ql.Driver> >>> Exception in thread "Driver" >>> Exception: java.lang.OutOfMemoryError thrown from the >>> UncaughtExceptionHandler in thread "Driver" >>> -- >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to >>> which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified >>> that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender >>> immediately >>> and delete it from your system. Thank You. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> >> -- >> Deepak >> >> >> > > > -- > Deepak > > > -- Deepak
