For anyone else interested: we ended up identifying one of the RS actually
failed to load the UngroupedAggregateRegionObserver because of a strange
XML parsing issue that was not occurring prior to this incident and not
happening on the any other RS.
Failed to load coprocessor
org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId:
jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml;
lineNumber: 196; columnNumber: 47; The string "--" is not permitted
within comments.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2656)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at
org.apache.phoenix.util.PropertiesUtil.cloneConfig(PropertiesUtil.java:81)
at
org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.start(UngroupedAggregateRegionObserver.java:219)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:414)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:255)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:208)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:364)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:226)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631)
at sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6145)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6449)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6421)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6377)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6328)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException; systemId:
jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml;
lineNumber: 196; columnNumber: 47; The string "--" is not permitted
within comments.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2491)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550)
... 27 more
On Wed, Sep 19, 2018 at 2:15 PM William Shen <wills...@marinsoftware.com>
wrote:
> Hi there,
>
> I have encountered the following exception while trying to query from
> Phoenix (was able to generate the exception doing a simple SELECT
> count(1)). I have verified (MD5) that each region server has the correct
> phoenix jars. Would appreciate any guidance on how to proceed further in
> troubleshooting this (or what could've caused this). Thank you!
>
> java.lang.IllegalStateException: Expected single, aggregated KeyValue from
> coprocessor, but instead received
> keyvalues={\x03\x80\x00\x00\x00\x00\x8D\xB8Y\x80\x00\x00\x00\x01c$\xE7\x00\x04\x80\x00\x00\x00\x01\x0C\x95N\x80\x00\x00\x00\x01\xCCU\xF1/SL:_0/1525817954352/Put/vlen=1/seqid=0/value=x}
>
> . Ensure aggregating coprocessors are loaded correctly on server
>
> at org.apache.phoenix.util.TupleUtil.getAggregateValue(TupleUtil.java:88)
>
> at
> org.apache.phoenix.expression.aggregator.ClientAggregators.aggregate(ClientAggregators.java:54)
>
> at
> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:74)
>
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
>
> at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
>
> at sqlline.BufferedRows.<init>(BufferedRows.java:37)
>
> at sqlline.SqlLine.print(SqlLine.java:1660)
>
> at sqlline.Commands.execute(Commands.java:833)
>
> at sqlline.Commands.sql(Commands.java:732)
>
> at sqlline.SqlLine.dispatch(SqlLine.java:813)
>
> at sqlline.SqlLine.begin(SqlLine.java:686)
>
> at sqlline.SqlLine.start(SqlLine.java:398)
>
> at sqlline.SqlLine.main(SqlLine.java:291)
>
> - Will
>
