date:20170713

Impala build issue

2017-07-13 Thread sky

Hi all,
I use this command( ./buildall.sh -fe_only -notests -so) to compile 
impala,in the fe module will be given the error(Connection timed out), But on 
the host using the "telnet" command to connect the repository.cloudera.com:443 
is success. why?
The following is a detailed error
Scanning dependencies of target fe

Running mvn install -DskipTests
Directory: /home/dreambase/dreambase_deploy/fe

[WARNING] Could not transfer metadata 
com.cloudera.cdh:cdh-root:5.13.0-SNAPSHOT/maven-metadata.xml from/to 
${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}): Cannot access 
${distMgmtSnapshotsUrl} with type default using the available connector 
factories: BasicRepositoryConnectorFactory


[WARNING] Could not transfer metadata 
org.apache.sentry:sentry-binding-hive-conf:1.5.1-cdh5.13.0-SNAPSHOT/maven-metadata.xml
 from/to cdh.snapshots.repo 
(https://repository.cloudera.com/content/repositories/snapshots): Connect to 
repository.cloudera.com:443 [repository.cloudera.com/34.197.206.76, 
repository.cloudera.com/34.201.234.30] failed: Connection timed out
[WARNING] Failure to transfer 
org.apache.sentry:sentry-binding-hive-conf:1.5.1-cdh5.13.0-SNAPSHOT/maven-metadata.xml
 from https://repository.cloudera.com/content/repositories/snapshots was cached 
in the local repository, resolution will not be reattempted until the update 
interval of cdh.snapshots.repo has elapsed or updates are forced. Original 
error: Could not transfer metadata 
org.apache.sentry:sentry-binding-hive-conf:1.5.1-cdh5.13.0-SNAPSHOT/maven-metadata.xml
 from/to cdh.snapshots.repo 
(https://repository.cloudera.com/content/repositories/snapshots): Connect to 
repository.cloudera.com:443 [repository.cloudera.com/34.197.206.76, 
repository.cloudera.com/34.201.234.30] failed: Connection timed out
[WARNING] Could not transfer metadata 
org.apache.sentry:sentry-binding:1.5.1-cdh5.13.0-SNAPSHOT/maven-metadata.xml 
from/to cdh.rcs.releases.repo 
(https://repository.cloudera.com/content/groups/cdh-releases-rcs): Connect to 
repository.cloudera.com:443 [repository.cloudera.com/34.201.234.30, 
repository.cloudera.com/34.197.206.76] failed: Connection timed out

Re: Loading tpc-ds

2017-07-13 Thread Tim Armstrong

I'm not sure exactly what is going on, but I can confirm that I was able to
load data on Ubuntu 16.04 with OpenJDK 8 a while back.

On Thu, Jul 13, 2017 at 2:58 PM, Jim Apple  wrote:

> I also see this with the Oracle JDK. I have also now checked I am not
> running out of memory.
>
> Oracle JDK7 is harder to get one's hands on, and OpenJDK7 isn't packaged by
> canonical for Ubuntu 16.04.
>
> On Wed, Jul 12, 2017 at 11:20 PM, Jim Apple  wrote:
>
> > I'm getting data loading errors on Ubuntu 16.04 in TPC-DS. The terminal
> > shows:
> >
> > ERROR : FAILED: Execution Error, return code 2 from
> > org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> >
> > logs/cluster/hive/hive.log shows the error below, which previous bugs
> have
> > called an issue with the disk being out of space, but my disk has at
> least
> > 45GB left on it
> >
> > IMPALA-3246, IMPALA-2856, IMPALA-2617
> >
> > I see this with openJDK8. I haven't tried Oracle's JDK yet.
> >
> > Has anyone else seen this and been able to diagnose it as something that
> > doesn't mean a full disk?
> >
> >
> > FATAL ExecReducer (ExecReducer.java:reduce(264)) -
> > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> > while processing row (tag=0) {"key":{},"value":{"_col0":
> > 48147,"_col1":17805,"_col2":27944,"_col3":606992,"_col4":
> > 3193,"_col5":16641,"_col6":10,"_col7":209,"_col8":44757,"_
> > col9":20,"_col10":5.51,"_col11":9.36,"_col12":9.17,"_
> > col13":0,"_col14":183.4,"_col15":110.2,"_col16":187.2,"_
> > col17":3.66,"_col18":0,"_col19":183.4,"_col20":187.06,"
> > _col21":73.2,"_col22":2452013}}
> > at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(
> > ExecReducer.java:253)
> > at org.apache.hadoop.mapred.ReduceTask.runOldReducer(
> > ReduceTask.java:444)
> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> > at org.apache.hadoop.mapred.LocalJobRunner$Job$
> > ReduceTaskRunnable.run(LocalJobRunner.java:346)
> > at java.util.concurrent.Executors$RunnableAdapter.
> > call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> > /test-warehouse/tpcds.store_sales/.hive-staging_hive_2017-
> > 07-12_22-51-18_139_3687815919405186455-760/_task_
> > tmp.-ext-1/ss_sold_date_sk=2452013/_tmp.01_0 could only be
> > replicated to 0 nodes instead of minReplication (=1).  There are 3
> > datanode(s) running and no node(s) are excluded in this operation.
> > at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.
> > chooseTarget4NewBlock(BlockManager.java:1724)
> > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getAdditionalBlock(FSNamesystem.java:3385)
> > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> > addBlock(NameNodeRpcServer.java:683)
> > at org.apache.hadoop.hdfs.server.namenode.
> > AuthorizationProviderProxyClientProtocol.addBlock(
> > AuthorizationProviderProxyClientProtocol.java:214)
> > at org.apache.hadoop.hdfs.protocolPB.
> > ClientNamenodeProtocolServerSideTranslatorPB.addBlock(
> > ClientNamenodeProtocolServerSideTranslatorPB.java:495)
> > at org.apache.hadoop.hdfs.protocol.proto.
> > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.
> callBlockingMethod(
> > ClientNamenodeProtocolProtos.java)
> > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1917)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> >
> > at org.apache.hadoop.hive.ql.exec.FileSinkOperator.
> > processOp(FileSinkOperator.java:751)
> > at org.apache.hadoop.hive.ql.exec.Operator.forward(
> > Operator.java:815)
> > at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(
> > SelectOperator.java:84)
> > at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(
> > ExecReducer.java:244)
> >
>

Re: Loading tpc-ds

2017-07-13 Thread Jim Apple

I also see this with the Oracle JDK. I have also now checked I am not
running out of memory.

Oracle JDK7 is harder to get one's hands on, and OpenJDK7 isn't packaged by
canonical for Ubuntu 16.04.

On Wed, Jul 12, 2017 at 11:20 PM, Jim Apple  wrote:

> I'm getting data loading errors on Ubuntu 16.04 in TPC-DS. The terminal
> shows:
>
> ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>
> logs/cluster/hive/hive.log shows the error below, which previous bugs have
> called an issue with the disk being out of space, but my disk has at least
> 45GB left on it
>
> IMPALA-3246, IMPALA-2856, IMPALA-2617
>
> I see this with openJDK8. I haven't tried Oracle's JDK yet.
>
> Has anyone else seen this and been able to diagnose it as something that
> doesn't mean a full disk?
>
>
> FATAL ExecReducer (ExecReducer.java:reduce(264)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row (tag=0) {"key":{},"value":{"_col0":
> 48147,"_col1":17805,"_col2":27944,"_col3":606992,"_col4":
> 3193,"_col5":16641,"_col6":10,"_col7":209,"_col8":44757,"_
> col9":20,"_col10":5.51,"_col11":9.36,"_col12":9.17,"_
> col13":0,"_col14":183.4,"_col15":110.2,"_col16":187.2,"_
> col17":3.66,"_col18":0,"_col19":183.4,"_col20":187.06,"
> _col21":73.2,"_col22":2452013}}
> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(
> ExecReducer.java:253)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(
> ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.LocalJobRunner$Job$
> ReduceTaskRunnable.run(LocalJobRunner.java:346)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test-warehouse/tpcds.store_sales/.hive-staging_hive_2017-
> 07-12_22-51-18_139_3687815919405186455-760/_task_
> tmp.-ext-1/ss_sold_date_sk=2452013/_tmp.01_0 could only be
> replicated to 0 nodes instead of minReplication (=1).  There are 3
> datanode(s) running and no node(s) are excluded in this operation.
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.
> chooseTarget4NewBlock(BlockManager.java:1724)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getAdditionalBlock(FSNamesystem.java:3385)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> addBlock(NameNodeRpcServer.java:683)
> at org.apache.hadoop.hdfs.server.namenode.
> AuthorizationProviderProxyClientProtocol.addBlock(
> AuthorizationProviderProxyClientProtocol.java:214)
> at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolServerSideTranslatorPB.addBlock(
> ClientNamenodeProtocolServerSideTranslatorPB.java:495)
> at org.apache.hadoop.hdfs.protocol.proto.
> ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1917)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
>
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.
> processOp(FileSinkOperator.java:751)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(
> Operator.java:815)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(
> SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(
> ExecReducer.java:244)
>

Re: Disabling all clang-tidy checks

2017-07-13 Thread Henry Robinson

To close this loop - I took the blunt instrument approach and removed the
directory (which is the imported kudu code) from tidying consideration the
same way we do gutil in run_clang_tidy.sh. Nothing else seemed to work as
one might think it should.

On 12 July 2017 at 17:46, Jim Apple  wrote:

> The clang-diagnostics are, IIRC, also enabled by the -W flags. You could
> try turning all warnings off via compiler flags.
>
> There is also a tool that auto-fixes clang-tidy warnings, but only some of
> them, and I never got even that much to work :-/
>
> On Wed, Jul 12, 2017 at 5:24 PM, Henry Robinson  wrote:
>
> > That does not, for whatever reason, actually disable clang-diagnostic-*.
> I
> > don't know why either :/
> >
> > On 12 July 2017 at 17:15, Jim Apple  wrote:
> >
> > > What about "diagnostic-henry-thinks-will-
> never-fire,-*,-clang-diagnosti
> > > c-*"?
> > >
> > >
> > >
> > > On Wed, Jul 12, 2017 at 5:01 PM, Henry Robinson 
> > wrote:
> > >
> > > > Has anyone found a way to disable all clang-tidy checks for a
> > directory?
> > > >
> > > > I've tried a directory-specific .clang-tidy file with
> > > >
> > > > ---
> > > > Checks: "-*"
> > > >
> > > > but that causes clang-tidy to exit with an error (because I didn't
> > > > configure any checks). So I tried adding one check that I thought
> would
> > > > never fire. But that silently re-enables a bunch of clang-diagnostic*
> > > > checks that I don't want.
> > > >
> > > > This happens when running:
> > > >
> > > > git diff HEAD~1 |
> > > >  "${IMPALA_TOOLCHAIN}/llvm-${IMPALA_LLVM_VERSION}/share/clan
> > > > g/clang-tidy-diff.py"
> > > > -clang-tidy-binary
> > > > "${IMPALA_TOOLCHAIN}/llvm-${IMPALA_LLVM_VERSION}/bin/clang-tidy" -p
> 1
> > > >
> > > > per
> > > > https://cwiki.apache.org/confluence/pages/viewpage.action?
> > > pageId=65868536
> > > >
> > > > Any ideas? Am I running clang-tidy wrong?
> > > >
> > >
> >
>



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Re: impala debug

2017-07-13 Thread Tim Armstrong

Removing the user list...

The Oracle docs say "

The -Xcheck:jni option causes the VM to do additional validation on the
arguments passed to JNI functions"


I guessthat means there's a buginhow we're using JNI?

On Thu, Jul 13, 2017 at 5:44 AM, Wang Lei(北京研发中心.112456) <
alaleiw...@sohu-inc.com> wrote:

> Hi all:
>   I followed https://cwiki.apache.org/confluence/display/IMPALA/
> Impala+Debugging+Tips to debug impalad fe
>   My step:
>  JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,
> address=localhost:9009,server=y,suspend=y -Xcheck:jni"
>
>   When using idea attaching,impalad process exited,with following error
> msg:
> I0713 16:49:02.511322 32715 JniFrontend.java:637] checkConfiguration
> called.
> I0713 16:49:02.538199 32715 JniFrontend.java:638] fs.defaultFS=hdfs://
> nameservice1
> I0713 16:49:02.538336 32715 JniFrontend.java:639]
> dfs.nameservices=nameservice1
> I0713 16:49:02.538432 32715 JniFrontend.java:640] foo=null
> I0713 16:49:05.230675 32715 JniFrontend.java:679] Short-circuit reads are
> not enabled.
> I0713 16:49:05.231148 32715 tmp-file-mgr.cc:109] Using scratch directory
> /opt/impala/impalad/impala-scratch on disk 1
> I0713 16:49:05.231283 32715 simple-logger.cc:83] Logging to:
> /tmp/profiles//impala_profile_log_1.1-1499935745231
> I0713 16:49:05.233969 32715 impala-server.cc:492] Event logging is disabled
> I0713 16:49:05.234060 32715 simple-logger.cc:83] Logging to:
> /opt/var/log/impalad/lineage/impala_lineage_log_1.0-1499935745234
> I0713 16:49:05.284077#
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  Internal Error (javaCalls.cpp:550), pid=32715, tid=140494292248832
> #  guarantee(method->size_of_parameters() == size_of_parameters())
> failed: wrong no. of arguments pushed
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build
> 1.7.0_67-b01)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode
> linux-amd64 compressed oops)
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /run/cloudera-scm-agent/process/5532-impala-IMPALAD/
> impala-conf/hs_err_pid32715.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.sun.com/bugreport/crash.jsp
> #
>  32715 impala-server.cc:1856] Impala Beeswax Service listening on 21000
> I0713 16:49:05.285711 32715 impala-server.cc:1878] Impala HiveServer2
> Service listening on 21050
> I0713 16:49:05.287331 32715 impala-server.cc:1897] ImpalaInternalService
> listening on 22000
> I0713 16:49:05.290220 32715 thrift-server.cc:449] ThriftServer 'backend'
> started on port: 22000
> I0713 16:49:05.504369 32715 thrift-server.cc:449] ThriftServer
> 'beeswax-frontend' started on port: 21000
> I0713 16:49:05.878450 32715 thrift-server.cc:449] ThriftServer
> 'hiveserver2-frontend' started on port: 21050
> I0713 16:49:05.878495 32715 exec-env.cc:241] Starting global services
>
> And /run/cloudera-scm-agent/process/5532-impala-IMPALAD/
> impala-conf/hs_err_pid32715.log showed:
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  Internal Error (javaCalls.cpp:550), pid=32715, tid=140494292248832
> #  guarantee(method->size_of_parameters() == size_of_parameters())
> failed: wrong no. of arguments pushed
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build
> 1.7.0_67-b01)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode
> linux-amd64 compressed oops)
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.sun.com/bugreport/crash.jsp
> #
>
> ---  T H R E A D  ---
>
> Current thread (0x0662e000):  JavaThread "main" [_thread_in_vm,
> id=32715, stack(0x7ffe33568000,0x7ffe33668000)]
>
> Stack: [0x7ffe33568000,0x7ffe33668000],  sp=0x7ffe33660020,
> free space=992k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V  [libjvm.so+0x99eb8a]  VMError::report_and_die()+0x2ea
> V  [libjvm.so+0x497282]  report_vm_error(char const*, int, char const*,
> char const*)+0x62
> V  [libjvm.so+0x5ff69a]  JavaCallArguments::verify(methodHandle,
> BasicType, Thread*)+0x5a
> V  [libjvm.so+0x5ff9ed]  JavaCalls::call_helper(JavaValue*,
> methodHandle*, JavaCallArguments*, Thread*)+0x1ed
> V  [libjvm.so+0x5fe5c8]  JavaCalls::call(JavaValue*, methodHandle,
> JavaCallArguments*, Thread*)+0x28
> V  [libjvm.so+0x638dd4]  jni_invoke_nonstatic(JNIEnv_*, JavaValue*,
> _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x2b4
> V  [libjvm.so+0x649ab9]  jni_CallObjectMethodV+0xe9
> V  [libjvm.so+0x67293e]  checked_jni_CallObjectMethodV+0x15e
> C  [impalad+0xdb9799]

Re: Rollback Mechanism

2017-07-13 Thread Matthew Jacobs

No, Kudu does not support multi-row transactions yet which will be
needed to support "rolling back" failed data loads.

Best,
Matt

On Wed, Jul 12, 2017 at 11:52 PM, sky  wrote:
> Hi all,
> Does impala and kudu have rollback mechanism when load data ?
> Thanks.

impala debug

2017-07-13 Thread 北京研发中心.112456

Hi all:
  I followed 
https://cwiki.apache.org/confluence/display/IMPALA/Impala+Debugging+Tips to 
debug impalad fe
  My step:
 
JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y
 -Xcheck:jni"

  When using idea attaching,impalad process exited,with following error msg:
I0713 16:49:02.511322 32715 JniFrontend.java:637] checkConfiguration called.
I0713 16:49:02.538199 32715 JniFrontend.java:638] 
fs.defaultFS=hdfs://nameservice1
I0713 16:49:02.538336 32715 JniFrontend.java:639] dfs.nameservices=nameservice1
I0713 16:49:02.538432 32715 JniFrontend.java:640] foo=null
I0713 16:49:05.230675 32715 JniFrontend.java:679] Short-circuit reads are not 
enabled.
I0713 16:49:05.231148 32715 tmp-file-mgr.cc:109] Using scratch directory 
/opt/impala/impalad/impala-scratch on disk 1
I0713 16:49:05.231283 32715 simple-logger.cc:83] Logging to: 
/tmp/profiles//impala_profile_log_1.1-1499935745231
I0713 16:49:05.233969 32715 impala-server.cc:492] Event logging is disabled
I0713 16:49:05.234060 32715 simple-logger.cc:83] Logging to: 
/opt/var/log/impalad/lineage/impala_lineage_log_1.0-1499935745234
I0713 16:49:05.284077#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (javaCalls.cpp:550), pid=32715, tid=140494292248832
#  guarantee(method->size_of_parameters() == size_of_parameters()) failed: 
wrong no. of arguments pushed
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 
compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# 
/run/cloudera-scm-agent/process/5532-impala-IMPALAD/impala-conf/hs_err_pid32715.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#
 32715 impala-server.cc:1856] Impala Beeswax Service listening on 21000
I0713 16:49:05.285711 32715 impala-server.cc:1878] Impala HiveServer2 Service 
listening on 21050
I0713 16:49:05.287331 32715 impala-server.cc:1897] ImpalaInternalService 
listening on 22000
I0713 16:49:05.290220 32715 thrift-server.cc:449] ThriftServer 'backend' 
started on port: 22000
I0713 16:49:05.504369 32715 thrift-server.cc:449] ThriftServer 
'beeswax-frontend' started on port: 21000
I0713 16:49:05.878450 32715 thrift-server.cc:449] ThriftServer 
'hiveserver2-frontend' started on port: 21050
I0713 16:49:05.878495 32715 exec-env.cc:241] Starting global services

And 
/run/cloudera-scm-agent/process/5532-impala-IMPALAD/impala-conf/hs_err_pid32715.log
 showed:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (javaCalls.cpp:550), pid=32715, tid=140494292248832
#  guarantee(method->size_of_parameters() == size_of_parameters()) failed: 
wrong no. of arguments pushed
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 
compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---  T H R E A D  ---

Current thread (0x0662e000):  JavaThread "main" [_thread_in_vm, 
id=32715, stack(0x7ffe33568000,0x7ffe33668000)]

Stack: [0x7ffe33568000,0x7ffe33668000],  sp=0x7ffe33660020,  free 
space=992k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x99eb8a]  VMError::report_and_die()+0x2ea
V  [libjvm.so+0x497282]  report_vm_error(char const*, int, char const*, char 
const*)+0x62
V  [libjvm.so+0x5ff69a]  JavaCallArguments::verify(methodHandle, BasicType, 
Thread*)+0x5a
V  [libjvm.so+0x5ff9ed]  JavaCalls::call_helper(JavaValue*, methodHandle*, 
JavaCallArguments*, Thread*)+0x1ed
V  [libjvm.so+0x5fe5c8]  JavaCalls::call(JavaValue*, methodHandle, 
JavaCallArguments*, Thread*)+0x28
V  [libjvm.so+0x638dd4]  jni_invoke_nonstatic(JNIEnv_*, JavaValue*, _jobject*, 
JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x2b4
V  [libjvm.so+0x649ab9]  jni_CallObjectMethodV+0xe9
V  [libjvm.so+0x67293e]  checked_jni_CallObjectMethodV+0x15e
C  [impalad+0xdb9799]  JNIEnv_::CallObjectMethod(_jobject*, _jmethodID*, 
...)+0xb5
C  [impalad+0x11a1335]  impala::Status 
impala::JniUtil::CallJniMethod(_jobject* const&, _jmethodID* cons
t&, impala::TGetJvmMetricsRequest const&, impala::TGetJvmMetricsResponse*)+0x18b
C  [impalad+0x11a0f68]  
impala::JniUtil::GetJvmMetrics(impala::TGetJvmMetricsRequest const&, 
impala::TGetJvmMetricsResponse*)+0x42
C  [impalad+0x11a5249]

Rollback Mechanism

2017-07-13 Thread sky

Hi all,
Does impala and kudu have rollback mechanism when load data ?
Thanks.

Loading tpc-ds

2017-07-13 Thread Jim Apple

I'm getting data loading errors on Ubuntu 16.04 in TPC-DS. The terminal
shows:

ERROR : FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask

logs/cluster/hive/hive.log shows the error below, which previous bugs have
called an issue with the disk being out of space, but my disk has at least
45GB left on it

IMPALA-3246, IMPALA-2856, IMPALA-2617

I see this with openJDK8. I haven't tried Oracle's JDK yet.

Has anyone else seen this and been able to diagnose it as something that
doesn't mean a full disk?


FATAL ExecReducer (ExecReducer.java:reduce(264)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{},"value":{"_col0":48147,"_col1":17805,"_col2":27944,"_col3":606992,"_col4":3193,"_col5":16641,"_col6":10,"_col7":209,"_col8":44757,"_col9":20,"_col10":5.51,"_col11":9.36,"_col12":9.17,"_col13":0,"_col14":183.4,"_col15":110.2,"_col16":187.2,"_col17":3.66,"_col18":0,"_col19":183.4,"_col20":187.06,"_col21":73.2,"_col22":2452013}}
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:253)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:346)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/test-warehouse/tpcds.store_sales/.hive-staging_hive_2017-07-12_22-51-18_139_3687815919405186455-760/_task_tmp.-ext-1/ss_sold_date_sk=2452013/_tmp.01_0
could only be replicated to 0 nodes instead of minReplication (=1).  There
are 3 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1724)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3385)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:683)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:495)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)

at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:751)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)

Impala build issue

Re: Loading tpc-ds

Re: Loading tpc-ds

Re: Disabling all clang-tidy checks

Re: impala debug

Re: Rollback Mechanism

impala debug

Rollback Mechanism

Loading tpc-ds

9 matches

Site Navigation

Mail list logo

Footer information