Creating soft link from /usr/phd to /usr/hdp makes pxf-service start
successfully.
Just curious what’s the overhead of using PXF?
postgres=# select * from hbase_member;
recordkey | address:city | address:contry | address:province | info:age | info
:birthday | info:company
------------+--------------+----------------+------------------+----------+-----
----------+--------------
scutshuxue | hangzhou | china | zhejiang | 99 | 1987
-06-17 | alibaba
xiaofeng | jieyang | china | guangdong | | 1987
-4-17 | alibaba
(2 rows)
Time: 434.412 ms
hbase(main):004:0* scan 'member'
ROW COLUMN+CELL
scutshuxue column=address:city, timestamp=1446104911726,
value=hangzhou
scutshuxue column=address:contry,
timestamp=1446104910743, value=china
scutshuxue column=address:province,
timestamp=1446104910775, value=zhejiang
scutshuxue column=info:age, timestamp=1446104987420,
value=99
scutshuxue column=info:birthday,
timestamp=1446104910674, value=1987-06-17
scutshuxue column=info:company, timestamp=1446104910715,
value=alibaba
xiaofeng column=address:city, timestamp=1446104920523,
value=jieyang
xiaofeng column=address:contry,
timestamp=1446104920461, value=china
xiaofeng column=address:province,
timestamp=1446104920486, value=guangdong
xiaofeng column=address:town, timestamp=1446104921802,
value=xianqiao
xiaofeng column=info:birthday,
timestamp=1446104920358, value=1987-4-17
xiaofeng column=info:company, timestamp=1446104920423,
value=alibaba
xiaofeng column=info:favorite,
timestamp=1446104920397, value=movie
2 row(s) in 0.0540 seconds
It’s very slow comparing running in hbase shell.
> 在 2015年10月29日，下午8:33，Noa Horn <[email protected]> 写道：
>
> The problem is probably because the jars that are required by PXF are not
> found.
>
> In the attached log file, this error for example shows that hadoop-auth.jar
> is not found:
> 29-Oct-2015 16:37:33.405 WARNING [localhost-startStop-1]
> com.pivotal.pxf.service.utilities.CustomWebappLoader.addRepositories Failed
> to load entry /usr/phd/current/hadoop-client/hadoop-auth.jar:
> java.nio.file.NoSuchFileException: /usr/phd/current/hadoop-client
>
> Have a look at /etc/conf/gphd/pxf (old version) or /etc/conf/pxf (open source
> version), at the file pxf-private.classpath.
> Every source specified there is required by PXF.
> The default paths for these resources is under /usr/phd/... (Pivotal
> distribution) while your system is hdp so the path is different. Luckily, we
> also provide the paths for hdp distribution - in pxf-privatehdp.classpath. If
> you copy the content of that file into pxf-private.classpath and run init and
> start again, it should work.
>
> As an aside, it's highly recommended to compile and use the open source
> version, because we made a few changes in the rpms.
> From the pxf directory, run 'make tomcat' to generate a tomcat rpm (required
> by PXF) and 'make rpm' to compile and create PXF rpms.
>
> Noa
>
>
> On Wed, Oct 28, 2015 at 11:38 PM, mailing-list-recv
> <[email protected] <mailto:[email protected]>>
> wrote:
> Thanks guys,
>
> Not sure if mailing list supports attachment, let me try anyway.
>
> Status command shows following:
> [root@cent61 ~]# service pxf-service status
>
> Checking if tcServer is up and running...
>
> Checking if PXF webapp is up and running...
>
> ERROR: PXF webapp is inaccessible but tcServer is up. Check logs for more
> information
>
> I was using the binary version downloaded from the site. I haven't tried to
> compile from open source yet.
>
> The port 51200 is opened
> [root@cent61 logs]# cat tcserver.pid
>
> 8385
>
> [root@cent61 logs]# ps -elf | grep 8385
>
> 0 S pxf 8385 1 0 80 0 - 312017 futex_ Oct29 ? 00:00:40
> /usr/jdk64/jdk1.7.0_67/bin/java
> -Djava.util.logging.config.file=/var/gphd/pxf/pxf-service/conf/logging.properties
>
> -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager
> -Xmx512M -Xss256K
> -Djava.endorsed.dirs=/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/endorsed
> -classpath
> /opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/bootstrap.jar:/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/tomcat-juli.jar
> -Dcatalina.base=/var/gphd/pxf/pxf-service
> -Dcatalina.home=/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE
> -Djava.io.tmpdir=/var/gphd/pxf/pxf-service/temp
> org.apache.catalina.startup.Bootstrap start
>
> 4 S root 23247 22386 0 80 0 - 25813 pipe_w 14:35 pts/2 00:00:00
> grep 8385
>
> [root@cent61 logs]# netstat -anp | grep 8385
>
> tcp 0 0 ::ffff:127.0.0.1:6969 <http://127.0.0.1:6969/> :::*
> LISTEN 8385/java
>
> tcp 0 0 :::51200 :::*
> LISTEN 8385/java
>
> unix 2 [ ] STREAM CONNECTED 2344585 8385/java
>
> unix 2 [ ] STREAM CONNECTED 2344417 8385/java
>
>
>
> Cheers
>
>
>
>
> 在 2015-10-29 03:22:48，"Jimmy Da" <[email protected]
> <mailto:[email protected]>> 写道：
> So it seems that Tomcat server is up, but the pxf servlet has not started. To
> confirm this, you can run "pxf-service status" to double check that pxf
> service is running.
>
> One guess on what the problem is that the Java libraries were not loaded
> correctly. I am looking at this line
> Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.Log
>
> Can you double check that you can find all the jar files at the locations in
> this file?
> https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-service/src/main/resources/pxf-privatehdp.classpath
>
> <https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-service/src/main/resources/pxf-privatehdp.classpath>
>
> Jimmy Da
> That’s what people do, they leap, and hoping to God they can fly.
>
> On Wed, Oct 28, 2015 at 12:03 PM, Ting(Goden) Yao <[email protected]
> <mailto:[email protected]>> wrote:
> Hi sequoiadb,
>
> which hawq/pxf version are you using (did you just compile the open source
> version or it's former pivotal released hawq versions)?
>
> Can you also attach pxf logs for investigation?
> it's at var/log/gphd/
>
> -Goden
>
> On Wed, Oct 28, 2015 at 1:51 AM sequoiadb <[email protected]
> <mailto:[email protected]>> wrote:
> Hi guys,
>
> I’m trying to setup PXF for HBase and got the following error:
> tpch=# create external table hbase_member ( recordkey bytea, "address:city"
> varchar, "address:contry" varchar, "address:province" varchar, "info:age"
> int, "info:birthday" varchar, "info:company" varchar ) location (
> 'pxf://cent61:50070/member?PROFILE=HBase' <>) FORMAT 'CUSTOM'(
> FORMATTER='pxfwritable_import');
> CREATE EXTERNAL TABLE
> tpch=# select * from hbase_member;
> ERROR: remote component error (0) from '192.168.31.205:51200
> <http://192.168.31.205:51200/>': couldn't connect to host (libchurl.c:852)
>
> I could successfully create regular tables and perform queries, but when I
> try to create pxf tables I’m keep getting error on connecting to port 51200.
>
> So I tried to start pxf-service and got
> [root@cent61 profile.d]# service pxf-service init
> Creating instance 'pxf-service' ...
> Using separate layout
> Creating bin/setenv.sh
> Applying template 'base'
> Copying template's contents
> Applying fragment 'context-fragment.xml' to 'conf/context.xml'
> Applying fragment 'server-fragment.xml' to 'conf/server.xml'
> Applying fragment 'web-fragment.xml' to 'conf/web.xml'
> Applying fragment 'tomcat-users-fragment.xml' to 'conf/tomcat-users.xml'
> Applying fragment 'catalina-fragment.properties' to
> 'conf/catalina.properties'
> Applying template 'base-tomcat-7'
> Copying template's contents
> Applying fragment 'server-fragment.xml' to 'conf/server.xml'
> Applying fragment 'web-fragment.xml' to 'conf/web.xml'
> Applying fragment 'catalina-fragment.properties' to
> 'conf/catalina.properties'
> Applying template 'bio'
> Copying template's contents
> Applying fragment 'server-fragment.xml' to 'conf/server.xml'
> Configuring instance 'pxf-service' to use Tomcat version 7.0.55.A.RELEASE
> Setting permissions
> Instance created
> Connector summary
> Port: 51200 Type: Blocking IO Secure: false
> [root@cent61 profile.d]# service pxf-service start
> /var/gphd/pxf /
> Creating home directory for pxf.
> Using CATALINA_BASE: /var/gphd/pxf/pxf-service
> Using CATALINA_HOME:
> /opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE
> Using CATALINA_TMPDIR: /var/gphd/pxf/pxf-service/temp
> Using JRE_HOME: /usr/jdk64/jdk1.7.0_67
> Using CLASSPATH:
> /opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/bootstrap.jar:/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/tomcat-juli.jar
> Using CATALINA_PID: /var/gphd/pxf/pxf-service/logs/tcserver.pid
> Tomcat started.
> Status: RUNNING as PID=8385
> /
> Checking if tcServer is up and running...
> tcServer not responding, re-trying after 1 second (attempt number 1)
> tcServer not responding, re-trying after 1 second (attempt number 2)
> Checking if PXF webapp is up and running...
> ERROR: PXF webapp is inaccessible but tcServer is up. Check logs for more
> information
>
> Now the select statement showing another error:
> tpch=# select * from base_member;
> ERROR: GPHD component not found (libchurl.c:1058)
>
> Looks like hit this error:
> bool handle_special_error(long response)
> {
> switch (response)
> {
> case 404:
> elog(ERROR, "GPHD component not found");
> break;
> default:
> return false;
> }
> return true;
> }
>
> Now do I need some sort of web service running, in order to make it work?
> Is it caused by PXF web app was not able to run? Which log do I supposed to
> look?
> catalina log showing this and I’m not sure if it’s the right one to look:
> 29-Oct-2015 16:37:34.923 SEVERE [localhost-startStop-1]
> org.apache.catalina.core.ContainerBase.addChildInternal
> ContainerBase.addChild: start:
> org.apache.catalina.LifecycleException: Failed to start component
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/pxf]]
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
> at
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1083)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1880)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError: Lorg/apache/commons/logging/Log;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2436)
> at java.lang.Class.getDeclaredFields(Class.java:1806)
> at
> org.apache.catalina.util.Introspection.getDeclaredFields(Introspection.java:106)
> at
> org.apache.catalina.startup.WebAnnotationSet.loadFieldsAnnotation(WebAnnotationSet.java:270)
> at
> org.apache.catalina.startup.WebAnnotationSet.loadApplicationListenerAnnotations(WebAnnotationSet.java:89)
> at
> org.apache.catalina.startup.WebAnnotationSet.loadApplicationAnnotations(WebAnnotationSet.java:63)
> at
> org.apache.catalina.startup.ContextConfig.applicationAnnotationsConfig(ContextConfig.java:403)
> at
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:879)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:374)
> at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5378)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> ... 10 more
> Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.Log
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> ... 24 more
>
> 29-Oct-2015 16:37:34.924 SEVERE [localhost-startStop-1]
> org.apache.catalina.startup.HostConfig.deployWAR Error deploying web
> application archive /var/gphd/pxf/pxf-service/webapps/pxf.war
> java.lang.IllegalStateException: ContainerBase.addChild: start:
> org.apache.catalina.LifecycleException: Failed to start component
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/pxf]]
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:904)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
> at
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1083)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1880)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> I’m running on a previously built HDP 2.2.8 and performed manual HAWQ
> installation. I got most parts done but stuck at PXF component, any help
> would be appreciate.
>
> Thanks
>
>
>
>
