Are you sure you had restarted RS process? you can check
"phoenix-server.jar" whether exists in the classpath of HBase by "jinfo"
command
----------------------------------------
Jaanai Zhang
Best regards!
William Shen <wills...@marinsoftware.com> 于2018年9月20日周四 上午6:01写道：
> For anyone else interested: we ended up identifying one of the RS actually
> failed to load the UngroupedAggregateRegionObserver because of a strange
> XML parsing issue that was not occurring prior to this incident and not
> happening on the any other RS.
>
> Failed to load coprocessor
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
> java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId:
> jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml;
> lineNumber: 196; columnNumber: 47; The string "--" is not permitted within
> comments.
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2656)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at
> org.apache.phoenix.util.PropertiesUtil.cloneConfig(PropertiesUtil.java:81)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.start(UngroupedAggregateRegionObserver.java:219)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:414)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:255)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:208)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:364)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:226)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:723)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631)
> at sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6145)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6449)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6421)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6377)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6328)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.xml.sax.SAXParseException; systemId:
> jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml;
> lineNumber: 196; columnNumber: 47; The string "--" is not permitted within
> comments.
> at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2491)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550)
> ... 27 more
>
>
>
> On Wed, Sep 19, 2018 at 2:15 PM William Shen <wills...@marinsoftware.com>
> wrote:
>
>> Hi there,
>>
>> I have encountered the following exception while trying to query from
>> Phoenix (was able to generate the exception doing a simple SELECT
>> count(1)). I have verified (MD5) that each region server has the correct
>> phoenix jars. Would appreciate any guidance on how to proceed further in
>> troubleshooting this (or what could've caused this). Thank you!
>>
>> java.lang.IllegalStateException: Expected single, aggregated KeyValue
>> from coprocessor, but instead received
>> keyvalues={\x03\x80\x00\x00\x00\x00\x8D\xB8Y\x80\x00\x00\x00\x01c$\xE7\x00\x04\x80\x00\x00\x00\x01\x0C\x95N\x80\x00\x00\x00\x01\xCCU\xF1/SL:_0/1525817954352/Put/vlen=1/seqid=0/value=x}
>>
>> . Ensure aggregating coprocessors are loaded correctly on server
>>
>> at org.apache.phoenix.util.TupleUtil.getAggregateValue(TupleUtil.java:88)
>>
>> at
>> org.apache.phoenix.expression.aggregator.ClientAggregators.aggregate(ClientAggregators.java:54)
>>
>> at
>> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:74)
>>
>> at
>> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
>>
>> at
>> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
>>
>> at sqlline.BufferedRows.<init>(BufferedRows.java:37)
>>
>> at sqlline.SqlLine.print(SqlLine.java:1660)
>>
>> at sqlline.Commands.execute(Commands.java:833)
>>
>> at sqlline.Commands.sql(Commands.java:732)
>>
>> at sqlline.SqlLine.dispatch(SqlLine.java:813)
>>
>> at sqlline.SqlLine.begin(SqlLine.java:686)
>>
>> at sqlline.SqlLine.start(SqlLine.java:398)
>>
>> at sqlline.SqlLine.main(SqlLine.java:291)
>>
>> - Will
>>
>
