Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422)
Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > Try to give the complete path to the file kv1.txt. > On 26 Mar 2015 11:48, "ÐΞ€ρ@Ҝ (๏̯͡๏)" <deepuj...@gmail.com> wrote: > >> I am now seeing this error. >> >> >> >> >> >> 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw >> exception: FAILED: SemanticException Line 1:23 Invalid path >> ''examples/src/main/resources/kv1.txt'': No files matching path >> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >> >> org.apache.spark.sql.execution.QueryExecutionException: FAILED: >> SemanticException Line 1:23 Invalid path >> ''examples/src/main/resources/kv1.txt'': No files matching path >> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >> >> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) >> >> at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) >> >> >> >> >> -sh-4.1$ pwd >> >> /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 >> >> -sh-4.1$ ls examples/src/main/resources/kv1.txt >> >> examples/src/main/resources/kv1.txt >> >> -sh-4.1$ >> >> >> >> On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang <zzh...@hortonworks.com> >> wrote: >> >>> You can do it in $SPARK_HOME/conf/spark-defaults.con >>> >>> spark.driver.extraJavaOptions -XX:MaxPermSize=512m >>> >>> Thanks. >>> >>> Zhan Zhang >>> >>> >>> On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: >>> >>> Where and how do i pass this or other JVM argument ? >>> -XX:MaxPermSize=512m >>> >>> On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang <zzh...@hortonworks.com> >>> wrote: >>> >>>> I solve this by increase the PermGen memory size in driver. >>>> >>>> -XX:MaxPermSize=512m >>>> >>>> Thanks. >>>> >>>> Zhan Zhang >>>> >>>> On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>> wrote: >>>> >>>> I am facing same issue, posted a new thread. Please respond. >>>> >>>> On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang <zzh...@hortonworks.com> >>>> wrote: >>>> >>>>> Hi Folks, >>>>> >>>>> I am trying to run hive context in yarn-cluster mode, but met some >>>>> error. Does anybody know what cause the issue. >>>>> >>>>> I use following cmd to build the distribution: >>>>> >>>>> ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn >>>>> -Phadoop-2.4 >>>>> >>>>> 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: >>>>> YarnClusterScheduler.postStartHook done >>>>> 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering >>>>> block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB >>>>> RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) >>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE >>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed >>>>> 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store >>>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >>>>> 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize >>>>> called >>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>> datanucleus.cache.level2 unknown - will be ignored >>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but >>>>> not present in CLASSPATH (or one of dependencies) >>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but >>>>> not present in CLASSPATH (or one of dependencies) >>>>> 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object >>>>> pin classes with >>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>>>> 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check >>>>> failed, assuming we are not on mysql: Lexical error at line 1, column 5. >>>>> Encountered: "@" (64), after : "". >>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore >>>>> 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not >>>>> found in metastore. hive.metastore.schema.verification is not enabled so >>>>> recording the schema version 0.13.1aa >>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in >>>>> metastore >>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in >>>>> metastore >>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in >>>>> admin role, since config is empty >>>>> 15/01/13 18:00:01 INFO session.SessionState: No Tez session required >>>>> at this point. hive.execution.engine=mr. >>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not >>>>> creating a lock manager >>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE >>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse >>>>> start=1421190003030 end=1421190003031 duration=1 >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic >>>>> Analysis >>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src >>>>> position=27 >>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : >>>>> db=default tbl=src >>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src >>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: >>>>> default >>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>> ip=unknown-ip-addr cmd=get_database: default >>>>> 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG >>>>> method=semanticAnalyze start=1421190003031 end=1421190003406 duration=375 >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: >>>>> Schema(fieldSchemas:null, properties:null) >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile >>>>> start=1421190002998 end=1421190003416 duration=418 >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=Driver.execute >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE IF >>>>> NOT EXISTS src (key INT, value STRING) >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit >>>>> start=1421190002995 end=1421190003421 duration=426 >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>> method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver> >>>>> 15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe for >>>>> table src >>>>> 15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG method=Driver.execute >>>>> start=1421190003416 end=1421190005498 duration=2082 >>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>> Exception in thread "Driver" >>>>> Exception: java.lang.OutOfMemoryError thrown from the >>>>> UncaughtExceptionHandler in thread "Driver" >>>>> -- >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to >>>>> which it is addressed and may contain information that is confidential, >>>>> privileged and exempt from disclosure under applicable law. If the >>>>> reader >>>>> of this message is not the intended recipient, you are hereby notified >>>>> that >>>>> any printing, copying, dissemination, distribution, disclosure or >>>>> forwarding of this communication is strictly prohibited. If you have >>>>> received this communication in error, please contact the sender >>>>> immediately >>>>> and delete it from your system. Thank You. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>> >>>> >>>> -- >>>> Deepak >>>> >>>> >>>> >>> >>> >>> -- >>> Deepak >>> >>> >>> >> >> >> -- >> Deepak >> >> -- Deepak
