[jira] [Created] (SPARK-4247) thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logda
qiaohaijun created SPARK-4247: - Summary: thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; then get error Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ultraman, access=WRITE, inode=/user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9:hive:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176) at
[jira] [Updated] (SPARK-4247) thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logda
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Summary: thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; then get errorr(was: thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; then get error ) thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; then get errorr --- Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at
[jira] [Commented] (SPARK-4247) thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and log
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198012#comment-14198012 ] qiaohaijun commented on SPARK-4247: --- Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/tmp/datadir-temp/hive_2014-11-05_16-57-49_712_8638564656249258514-1/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/12/_SUCCESS (state=,code=0) the thriftserver user is ultraman, we hope that the owner of dir /user/hive/warehouse/ultraman_tmp.db/12 is ultraman, then in fact the new dir's ower is hive thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; then get errorr --- Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at
[jira] [Updated] (SPARK-4247) beeline execute create table as
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Summary: beeline execute create table as (was: thriftserver user is ultraman when execute create table ultraman_tmp.12 as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; then get errorr ) beeline execute create table as Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Affects Versions: 1.1.1 Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ultraman, access=WRITE, inode=/user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9:hive:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at
[jira] [Updated] (SPARK-4247) beeline execute create table as
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Affects Version/s: 1.1.1 beeline execute create table as Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Affects Versions: 1.1.1 Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ultraman, access=WRITE, inode=/user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9:hive:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) at
[jira] [Updated] (SPARK-4247) beeline execute create table as
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Component/s: SQL beeline execute create table as Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.1.1 Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ultraman, access=WRITE, inode=/user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9:hive:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) at
[jira] [Updated] (SPARK-4247) use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Summary: use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive(was: beeline execute create table as ) use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive - Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.1.1 Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ultraman, access=WRITE, inode=/user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9:hive:hdfs:drwxr-xr-x at
[jira] [Updated] (SPARK-4247) use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest thriftserver with hive 0.12 hive :0.13.1 compile cmd: sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive was: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest hive server :0.13.1 compile env sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive - Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.1.1 Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest thriftserver with hive 0.12 hive :0.13.1 compile cmd: sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at
[jira] [Updated] (SPARK-4247) use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Description: 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ultraman, access=WRITE, inode=/user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9:hive:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5461) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInternal(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInt(FSNamesystem.java:3138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3106) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:680) at
[jira] [Updated] (SPARK-4247) use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Description: thriftserver start cmd: sudo -u ultraman sh start-thriftserver.sh --- beeline start cmd: sh beeline -u jdbc:hive2://x.x.x.x:1 -n ultraman -p ** --- sql: create table ultraman_tmp.test as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; the hdfs dir is follow: drwxr-xr-x - hive hdfs 0 2014-11-03 18:02 /user/hive/warehouse/ultraman_tmp.db/test 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ultraman, access=WRITE, inode=/user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9:hive:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5461) at
[jira] [Created] (SPARK-4248) not support add jar
qiaohaijun created SPARK-4248: - Summary: not support add jar Key: SPARK-4248 URL: https://issues.apache.org/jira/browse/SPARK-4248 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.1.1 Environment: java:1.7 hadoop:2.3.0-cdh5.0.0 spark:1.1.1 thriftserver-with-hive:0.12 hive metaserver:0.13.1 Reporter: qiaohaijun add jar not support the udf jar need use --jars upload -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-4248) [SQL] spark sql not support add jar
[ https://issues.apache.org/jira/browse/SPARK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4248: -- Summary: [SQL] spark sql not support add jar (was: not support add jar ) [SQL] spark sql not support add jar Key: SPARK-4248 URL: https://issues.apache.org/jira/browse/SPARK-4248 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.1.1 Environment: java:1.7 hadoop:2.3.0-cdh5.0.0 spark:1.1.1 thriftserver-with-hive:0.12 hive metaserver:0.13.1 Reporter: qiaohaijun add jar not support the udf jar need use --jars upload -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-4247) [SQL] use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive
[ https://issues.apache.org/jira/browse/SPARK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiaohaijun updated SPARK-4247: -- Summary: [SQL] use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive(was: use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive ) [SQL] use beeline execute create table as thriftserver is not use hive user ,but the new hdfs dir's owner is hive --- Key: SPARK-4247 URL: https://issues.apache.org/jira/browse/SPARK-4247 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.1.1 Environment: java: 1.7 hadoop: 2.3.0-cdh5.0.0 spark: branch-1.1 lastest thriftserver with hive 0.12 hive :0.13.1 compile cmd: sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive Reporter: qiaohaijun thriftserver start cmd: sudo -u ultraman sh start-thriftserver.sh --- beeline start cmd: sh beeline -u jdbc:hive2://x.x.x.x:1 -n ultraman -p ** --- sql: create table ultraman_tmp.test as select channel, subchannel from custom.common_pc_pv where logdate = '2014110210' and logdate = '2014110210' limit 10; the hdfs dir is follow: drwxr-xr-x - hive hdfs 0 2014-11-03 18:02 /user/hive/warehouse/ultraman_tmp.db/test 2014-11-03 20:12:52,498 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI http://10.141.77.221:9083 2014-11-03 20:12:52,509 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt. 2014-11-03 20:12:53,510 INFO [pool-10-thread-3] hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore. 2014-11-03 20:12:53,899 ERROR [pool-10-thread-3] server.SparkSQLOperationManager (Logging.scala:logError(96)) - Error executing query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourceviewfs://nsX/user/hive/datadir-tmp/hive_2014-11-03_20-12-43_561_4822588651544736505-2/-ext-1/_SUCCESS to destination /user/hive/warehouse/ultraman_tmp.db/litao_sparksql_test_9/_SUCCESS at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2173) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2227) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:652) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1443) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result$lzycompute(InsertIntoHiveTable.scala:243) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.result(InsertIntoHiveTable.scala:171) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:162) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.init(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at org.apache.spark.sql.hive.thriftserver.server.SparkSQLOperationManager$$anon$1.run(SparkSQLOperationManager.scala:172) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:193) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:175) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:150) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:207) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:58) at org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:55) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:526) at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:55) at
[jira] [Commented] (SPARK-3708) Backticks aren't handled correctly is aliases
[
https://issues.apache.org/jira/browse/SPARK-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193051#comment-14193051
]
qiaohaijun commented on SPARK-3708:
---
+1
Backticks aren't handled correctly is aliases
-
Key: SPARK-3708
URL: https://issues.apache.org/jira/browse/SPARK-3708
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.1.0
Reporter: Michael Armbrust
Assignee: Ravindra Pesala
Fix For: 1.2.0
Here's a failing test case:
{code}
sql(SELECT k FROM (SELECT `key` AS `k` FROM src) a)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2420) Dependency changes for compatibility with Hive
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193165#comment-14193165 ] qiaohaijun commented on SPARK-2420: --- +1 Dependency changes for compatibility with Hive -- Key: SPARK-2420 URL: https://issues.apache.org/jira/browse/SPARK-2420 Project: Spark Issue Type: Wish Components: Build Affects Versions: 1.0.0 Reporter: Xuefu Zhang Labels: Hive Fix For: 1.2.0 Attachments: spark_1.0.0.patch During the prototyping of HIVE-7292, many library conflicts showed up because Spark build contains versions of libraries that's vastly different from current major Hadoop version. It would be nice if we can choose versions that's in line with Hadoop or shading them in the assembly. Here are the wish list: 1. Upgrade protobuf version to 2.5.0 from current 2.4.1 2. Shading Spark's jetty and servlet dependency in the assembly. 3. guava version difference. Spark is using a higher version. I'm not sure what's the best solution for this. The list may grow as HIVE-7292 proceeds. For information only, the attached is a patch that we applied on Spark in order to make Spark work with Hive. It gives an idea of the scope of changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2848) Shade Guava in Spark deliverables
[ https://issues.apache.org/jira/browse/SPARK-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193172#comment-14193172 ] qiaohaijun commented on SPARK-2848: --- +1 Shade Guava in Spark deliverables - Key: SPARK-2848 URL: https://issues.apache.org/jira/browse/SPARK-2848 Project: Spark Issue Type: Sub-task Components: Spark Core Reporter: Marcelo Vanzin Assignee: Marcelo Vanzin Fix For: 1.2.0 As discussed in SPARK-2420, this task covers the work of shading Guava in Spark deliverables so that they don't conflict with the Hadoop classpath (nor user's classpath). Since one Guava class is exposed through Spark's API, that class will be forked from 14.0.1 (current version used by Spark) and excluded from any shading. The end result is that Spark's Guava won't be exposed to users anymore. This has the side-effect of effectively downgrading to version 11 (the one used by Hadoop) for those that do not explicitly depend on / package Guava with their apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3217) Shaded Guava jar doesn't play well with Maven build when SPARK_PREPEND_CLASSES is set
[
https://issues.apache.org/jira/browse/SPARK-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193177#comment-14193177
]
qiaohaijun commented on SPARK-3217:
---
+1
Shaded Guava jar doesn't play well with Maven build when
SPARK_PREPEND_CLASSES is set
-
Key: SPARK-3217
URL: https://issues.apache.org/jira/browse/SPARK-3217
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 1.2.0
Reporter: Cheng Lian
Assignee: Marcelo Vanzin
Fix For: 1.2.0
PR [#1813|https://github.com/apache/spark/pull/1813] shaded Guava jar file
and moved Guava classes to package {{org.spark-project.guava}} when Spark is
built by Maven. But if developers set the environment variable
{{SPARK_PREPEND_CLASSES}} to {{true}}, commands like {{bin/spark-shell}}
throws {{ClassNotFoundException}}:
{code}
# Set the env var
$ export SPARK_PREPEND_CLASSES=true
# Build Spark with Maven
$ mvn clean package -Phive,hadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests
...
# Then spark-shell complains
$ ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
Exception in thread main java.lang.NoClassDefFoundError:
com/google/common/util/concurrent/ThreadFactoryBuilder
at org.apache.spark.util.Utils$.init(Utils.scala:636)
at org.apache.spark.util.Utils$.clinit(Utils.scala)
at org.apache.spark.repl.SparkILoop.init(SparkILoop.scala:134)
at org.apache.spark.repl.SparkILoop.init(SparkILoop.scala:65)
at org.apache.spark.repl.Main$.main(Main.scala:30)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:317)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
com.google.common.util.concurrent.ThreadFactoryBuilder
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more
# Check the assembly jar file
$ jar tf
assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar |
grep -i ThreadFactoryBuilder
org/spark-project/guava/common/util/concurrent/ThreadFactoryBuilder$1.class
org/spark-project/guava/common/util/concurrent/ThreadFactoryBuilder.class
{code}
SBT build is fine since we don't shade Guava with SBT right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2420) Dependency changes for compatibility with Hive
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14189830#comment-14189830 ] qiaohaijun commented on SPARK-2420: --- compatiblity confuse me Dependency changes for compatibility with Hive -- Key: SPARK-2420 URL: https://issues.apache.org/jira/browse/SPARK-2420 Project: Spark Issue Type: Wish Components: Build Affects Versions: 1.0.0 Reporter: Xuefu Zhang Labels: Hive Fix For: 1.2.0 Attachments: spark_1.0.0.patch During the prototyping of HIVE-7292, many library conflicts showed up because Spark build contains versions of libraries that's vastly different from current major Hadoop version. It would be nice if we can choose versions that's in line with Hadoop or shading them in the assembly. Here are the wish list: 1. Upgrade protobuf version to 2.5.0 from current 2.4.1 2. Shading Spark's jetty and servlet dependency in the assembly. 3. guava version difference. Spark is using a higher version. I'm not sure what's the best solution for this. The list may grow as HIVE-7292 proceeds. For information only, the attached is a patch that we applied on Spark in order to make Spark work with Hive. It gives an idea of the scope of changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2706) Enable Spark to support Hive 0.13
[
https://issues.apache.org/jira/browse/SPARK-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173541#comment-14173541
]
qiaohaijun commented on SPARK-2706:
---
sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests
-Dhadoop.version=2.3.0-cdh5.0.0 -Phive -Dhive.version=0.13.1
I get the same error output, and I execute the command git pull today that
get the update code .
Enable Spark to support Hive 0.13
-
Key: SPARK-2706
URL: https://issues.apache.org/jira/browse/SPARK-2706
Project: Spark
Issue Type: Dependency upgrade
Components: SQL
Affects Versions: 1.0.1
Reporter: Chunjun Xiao
Assignee: Zhan Zhang
Attachments: hive.diff, spark-2706-v1.txt, spark-2706-v2.txt,
spark-hive.err, v1.0.2.diff
It seems Spark cannot work with Hive 0.13 well.
When I compiled Spark with Hive 0.13.1, I got some error messages, as
attached below.
So, when can Spark be enabled to support Hive 0.13?
Compiling Error:
{quote}
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala:180:
type mismatch;
found : String
required: Array[String]
[ERROR] val proc: CommandProcessor =
CommandProcessorFactory.get(tokens(0), hiveconf)
[ERROR] ^
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala:264:
overloaded method constructor TableDesc with alternatives:
(x$1: Class[_ : org.apache.hadoop.mapred.InputFormat[_, _]],x$2:
Class[_],x$3: java.util.Properties)org.apache.hadoop.hive.ql.plan.TableDesc
and
()org.apache.hadoop.hive.ql.plan.TableDesc
cannot be applied to (Class[org.apache.hadoop.hive.serde2.Deserializer],
Class[(some other)?0(in value tableDesc)(in value tableDesc)], Class[?0(in
value tableDesc)(in value tableDesc)], java.util.Properties)
[ERROR] val tableDesc = new TableDesc(
[ERROR] ^
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala:140:
value getPartitionPath is not a member of
org.apache.hadoop.hive.ql.metadata.Partition
[ERROR] val partPath = partition.getPartitionPath
[ERROR]^
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScan.scala:132:
value appendReadColumnNames is not a member of object
org.apache.hadoop.hive.serde2.ColumnProjectionUtils
[ERROR] ColumnProjectionUtils.appendReadColumnNames(hiveConf,
attributes.map(_.name))
[ERROR] ^
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala:79:
org.apache.hadoop.hive.common.type.HiveDecimal does not have a constructor
[ERROR] new HiveDecimal(bd.underlying())
[ERROR] ^
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala:132:
type mismatch;
found : org.apache.hadoop.fs.Path
required: String
[ERROR]
SparkHiveHadoopWriter.createPathFromString(fileSinkConf.getDirName, conf))
[ERROR] ^
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala:179:
value getExternalTmpFileURI is not a member of
org.apache.hadoop.hive.ql.Context
[ERROR] val tmpLocation = hiveContext.getExternalTmpFileURI(tableLocation)
[ERROR] ^
[ERROR]
/ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala:209:
org.apache.hadoop.hive.common.type.HiveDecimal does not have a constructor
[ERROR] case bd: BigDecimal = new HiveDecimal(bd.underlying())
[ERROR] ^
[ERROR] 8 errors found
[DEBUG] Compilation failed (CompilerInterface)
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .. SUCCESS [2.579s]
[INFO] Spark Project Core SUCCESS [2:39.805s]
[INFO] Spark Project Bagel ... SUCCESS [21.148s]
[INFO] Spark Project GraphX .. SUCCESS [59.950s]
[INFO] Spark Project ML Library .. SUCCESS [1:08.771s]
[INFO] Spark Project Streaming ... SUCCESS [1:17.759s]
[INFO] Spark Project Tools ... SUCCESS [15.405s]
[INFO] Spark Project Catalyst SUCCESS [1:17.405s]
[INFO] Spark Project SQL . SUCCESS [1:11.094s]
[INFO] Spark Project Hive FAILURE [11.121s]
[INFO] Spark Project REPL
[jira] [Commented] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
[ https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173543#comment-14173543 ] qiaohaijun commented on SPARK-1479: --- sh make-distribution.sh --tgz -Phadoop-provided -Pyarn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phive -Dhive.verison=0.13.1 get the same error output building spark on 2.0.0-cdh4.4.0 failed --- Key: SPARK-1479 URL: https://issues.apache.org/jira/browse/SPARK-1479 Project: Spark Issue Type: Question Environment: 2.0.0-cdh4.4.0 Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL spark 0.9.1 java version 1.6.0_32 Reporter: jackielihf Attachments: mvn.log [INFO] [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Caused by: Compilation failed at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101) at sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22) at sbt.inc.Incremental$.cycle(Incremental.scala:40) at sbt.inc.Incremental$.compile(Incremental.scala:25) at sbt.inc.IncrementalCompile$.apply(Compile.scala:20) at
