[jira] [Created] (HIVE-13010) partitions autogenerated predicates broken
Stanilovsky Evgeny created HIVE-13010: - Summary: partitions autogenerated predicates broken Key: HIVE-13010 URL: https://issues.apache.org/jira/browse/HIVE-13010 Project: Hive Issue Type: Bug Affects Versions: 1.1.0 Reporter: Stanilovsky Evgeny Priority: Trivial hi, i`m looking for simalar problem but found only: https://issues.apache.org/jira/browse/HIVE-9630 it`s looks like the same but you can easily repeat it on testing i hope. I have two simalar requests , the difference in autogenerated data predicates, in first case explain show ful lscan. ''' set hive.optimize.constant.propagation=true; explain select * from logs.weather_forecasts where dt between from_unixtime(unix_timestamp() - 3600*24*3, '-MM-dd') and from_unixtime(unix_timestamp() - 3600*24*1, '-MM-dd') and provider_id = 100 STAGE PLANS: 5 Stage: Stage-1 6 Map Reduce 7 Map Operator Tree: 8 TableScan 9 alias: weather_forecasts 10 Statistics: Num rows: 36124837607 Data size: 47395787122046 Basic stats: PARTIAL Column stats: NONE '''' and '''' set hive.optimize.constant.propagation=true; explain select * from logs.redir_log where dt between '2016-02-02' and '2016-02-04' and pid = 100 0 STAGE DEPENDENCIES: 1 Stage-1 is a root stage 2 Stage-0 depends on stages: Stage-1 3 4 STAGE PLANS: 5 Stage: Stage-1 6 Map Reduce 7 Map Operator Tree: 8 TableScan 9 alias: redir_log 10 Statistics: Num rows: 2798358420 Data size: 5761819991150 Basic stats: COMPLETE Column stats: NONE '''' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12582) fix hiveserver2.sh script like metastore.sh is.
Stanilovsky Evgeny created HIVE-12582: - Summary: fix hiveserver2.sh script like metastore.sh is. Key: HIVE-12582 URL: https://issues.apache.org/jira/browse/HIVE-12582 Project: Hive Issue Type: Improvement Components: HiveServer2 Environment: linux Reporter: Stanilovsky Evgeny Assignee: Vaibhav Gumashta Priority: Trivial Fix For: 1.0.0 in case of altering startup jvm properties, it would be great to fix hiveserver2.sh, appending : export HADOOP_OPTS="$HIVE_METASTORE_HADOOP_OPTS $HADOOP_OPTS" like it contains in metastore.sh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Description: with hive-site.xml config like: hive.metastore.uris thrift://mashine1:9083,thrift://mashine2:9083 log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, try to refresh database list through hue\beeline : show tables, restore old connection iptables -D OUTPUT 1, and split ones more : iptables -I INPUT -d mashine2 -j REJECT ; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) was: with hive-site.xml config like: hive.metastore.uris thrift://mashine1:9083,thrift://mashine2:9083 log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Status: Patch Available (was: Open) > HiveServer2 can`t correct re-establish metastore communication > -- > > Key: HIVE-8483 > URL: https://issues.apache.org/jira/browse/HIVE-8483 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 > Environment: unbuntu 14.04 java 1.7 > Reporter: Stanilovsky Evgeny > Labels: patch > Attachments: patch1.patch > > > with hive-site.xml config like: > > hive.metastore.uris > thrift://mashine1:9083,thrift://mashine1:9083 > > log: > 2014-10-16 14:04:04,502 INFO hive.metastore > (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with > URI thrift://mashine1:9083 > 2014-10-16 14:04:04,515 INFO hive.metastore > (HiveMetaStoreClient.java:open(345)) - Connected to metastore. > if split current connection, for example with command: iptables -I INPUT -d > mashine1 -j REJECT, and try to refresh database list through hue\beeline : > show tables; we have an stack like : > 2014-10-07 10:14:33,898 ERROR hive.log > (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: > org.apache.thrift.transport.TTransportException java.net.SocketException: > Broken pipe > org.apache.thrift.transport.TTransportException: java.net.SocketException: > Broken pipe > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) > at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) > at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) > at > org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) > at > org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) > at > org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) > at > org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) > at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) > at > org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Description: with hive-site.xml config like: hive.metastore.uris thrift://mashine1:9083,thrift://mashine2:9083 log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) was: with hive-site.xml config like: hive.metastore.uris thrift://mashine1:9083,thrift://mashine1:9083 log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Description: with hive-site.xml config like: hive.metastore.uris thrift://mashine1:9083,thrift://mashine1:9083 log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) was: with hive-site.xml config like: hive.metastore.uris thrift://mashine1:9083,thrift://mashine1:9083 log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Attachment: patch1.patch > HiveServer2 can`t correct re-establish metastore communication > -- > > Key: HIVE-8483 > URL: https://issues.apache.org/jira/browse/HIVE-8483 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 > Environment: unbuntu 14.04 java 1.7 > Reporter: Stanilovsky Evgeny > Labels: patch > Attachments: patch1.patch > > > with hive-site.xml config like: > > hive.metastore.uris > thrift://mashine1:9083,thrift://mashine1:9083 > > log: > 2014-10-16 14:04:04,502 INFO hive.metastore > (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with > URI thrift://mashine1:9083 > 2014-10-16 14:04:04,515 INFO hive.metastore > (HiveMetaStoreClient.java:open(345)) - Connected to metastore. > if split current connection, for example with command: iptables -I INPUT -d > mashine1 -j REJECT, and try to refresh database list through beeline : show > tables; we have an stack like : > 2014-10-07 10:14:33,898 ERROR hive.log > (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: > org.apache.thrift.transport.TTransportException java.net.SocketException: > Broken pipe > org.apache.thrift.transport.TTransportException: java.net.SocketException: > Broken pipe > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) > at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) > at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) > at > org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) > at > org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) > at > org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) > at > org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) > at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) > at > org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
Stanilovsky Evgeny created HIVE-8483: Summary: HiveServer2 can`t correct re-establish metastore communication Key: HIVE-8483 URL: https://issues.apache.org/jira/browse/HIVE-8483 Project: Hive Issue Type: Improvement Affects Versions: 0.12.0 Environment: unbuntu 14.04 java 1.7 Reporter: Stanilovsky Evgeny with hive-site.xml config like: hive.metastore.uris thrift://mashine1:9083,thrift://mashine1:9083 log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.util.HostUtil for Hive 0.10.0
thats why mapreduce.util.HostUtil is from mr2, and you use (i think) mr1,
hive has dummy code :
common/java/org/apache/hadoop/hive/shims/ShimLoader.java ->
> String getMajorVersion() {
> String vers = VersionInfo.getVersion();
>
thus you gets something like : 2.0.0-cdh4.1.0, instead you use old hadoop
for example : 2.0.0-mr1-cdh4.1.0
i will write to cloudera to fix it, but simple fix is to fake:
HADOOP_SHIM_CLASSES.put("0.23",
"org.apache.hadoop.hive.shims.Hadoop20SShims");
not
HADOOP_SHIM_CLASSES.put("0.23",
"org.apache.hadoop.hive.shims.Hadoop23Shims");
>
>
среда, 27 февраля 2013 г., 6:39:36 UTC+4 пользователь Eric Chu написал:
>
> (+hue-user since this issue prevents me from successfully installing Hue
> from source)
>
> Hi,
>
> I recently did the following with both the Hive-0.10 and Hive-0.9, and had
> a problem with 0.10 that I didn't see with 0.9
>
>- Checked out the respective branch from github
>- Did an "ant package"
>- Copied the dist folder to /usr/lib/hive on the right machine
>- Copied mysql-connector-java-5.1.22-bin.jar to /usr/lib/hive/lib
>- Configured /etc/hive/conf (so same for both versions)
>
> The problem is that when I use Hive-0.10, doing a "select count(1) from
> table" (or anything that requires MR) would return a NoClassDefFound error
> (see *Error Msg *below), whereas when I use Hive-0.9, the job would run
> fine. Has anyone run into this problem? I can't use Hive-0.9 b/c it has
> libthrift-0.7.0.jar, while Hue-2.2 (which I'm also using) requires
> libthrift-0.9.0.jar (available in Hive-10).
>
> Any insights would be much appreciated. Googling on this error doesn't get
> very far. Thanks!
>
> *Error Msg:*
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=
> Starting Job = job_201302201756_0009, Tracking URL =
> http://master-hadoop.pww-arp-dev.rfiserve.net:50030/jobdetails.jsp?jobid=job_201302201756_0009
> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201302201756_0009
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2013-02-26 21:05:01,060 Stage-1 map = 0%, reduce = 0%
> 2013-02-26 21:05:32,286 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201302201756_0009 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://master-hadoop.pww-arp-dev.rfiserve.net:50030/jobdetails.jsp?jobid=job_201302201756_0009
> Examining task ID: task_201302201756_0009_m_02 (and more) from job
> job_201302201756_0009
> Exception in thread "Thread-29" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/util/HostUtil
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:53)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.util.HostUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> ... 4 more
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
>
>
>
