[
https://issues.apache.org/jira/browse/HAWQ-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866266#comment-15866266
]
Kyle R Dunn edited comment on HAWQ-1234 at 2/14/17 6:03 PM:
------------------------------------------------------------
Additionally, I observed all this using the following command, on the HAWQ
master. First, start the {{tcpdump}} trace, then invoke a SELECT from a
previously defined PXF table, either using HCatalog directly, or manually
defining a PXF external table. A similar approach could be used to observe
datanode traffic during read/write operations.
Watch all traffic on port 51200 (PXF default)
{code}
$ tcpdump -n port 51200 -A
{code}
Initiate a PXF query via HCatalog
{code}
# select * from hcatalog.default.kdtest ;
key | value
---------+-----------
somekey | somevalue
1234 | 56789
hello | world
aloha | mondays
(4 rows)
{code}
Here is the conversation output:
{code}
21:20:14.632410 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [S], seq
3498721498, win 65483, options [mss 65495,sackOK,TS val 1901602390 ecr
1901547904,nop,wscale 9], length 0
E..<f.@.@..-........O.....@.........._.........
qX&VqWQ....
21:20:14.632421 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [S.], seq
3752275736, ack 3498721499, win 65483, options [mss 65495,sackOK,TS val
1901602390 ecr 1901602390,nop,wscale 9], length 0
E..<..@.@.<...........O.../...@................
qX&VqX&V...
21:20:14.632428 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 1, win
128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0
E..4f.@.@..4........O.....@.../............
qX&VqX&V
21:20:14.632602 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [P.], seq 1:318,
ack 1, win 128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 317
E..qf.@.@...........O.....@.../......e.....
qX&VqX&VGET /pxf/v14/Metadata/getMetadata?profile=Hive&pattern=default.kdtest
HTTP/1.1
Host: localhost:51200
Accept: application/json
X-GP-SEGMENT-ID: -100005432
X-GP-SEGMENT-COUNT: 0
X-GP-XID: 2725021
X-GP-ALIGNMENT: 8
X-GP-URL-HOST: localhost
X-GP-URL-PORT: 51200
X-GP-URI: localhost:51200/
X-GP-HAS-FILTER: 0
21:20:14.632607 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [.], ack 318, win
130, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0
E..4 O@.@.3s..........O.../...B............
qX&VqX&V
21:20:15.084890 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [P.], seq 1:269,
ack 318, win 130, options [nop,nop,TS val 1901602843 ecr 1901602390], length 268
E..@ P@.@.2f..........O.../...B......4.....
qX(.qX&VHTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Content-Length: 132
Date: Tue, 14 Feb 2017 05:20:15 GMT
{"PXFMetadata":[{"item":{"path":"default","name":"kdtest"},"fields":[{"name":"key","type":"text"},{"name":"value","type":"text"}]}]}
21:20:15.084900 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 269, win
130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4f.@.@..2........O.....B...0%.....!.....
qX(.qX(.
21:20:15.085229 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [F.], seq 318, ack
269, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4f.@.@..1........O.....B...0%..... .....
qX(.qX(.
21:20:15.085286 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [F.], seq 269, ack
319, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4 Q@.@.3q..........O...0%..B............
qX(.qX(.
21:20:15.085294 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 270, win
130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4f.@.@..0........O.....B...0&...........
qX(.qX(.
21:20:15.112739 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [S], seq 222439143,
win 65483, options [mss 65495,sackOK,TS val 1901602870 ecr
1901602843,nop,wscale 9], length 0
B&......................
qX(6qX(....
21:20:15.112765 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [S.], seq
2606634976, ack 222439144, win 65483, options [mss 65495,sackOK,TS val
1901602871 ecr 1901602870,nop,wscale 9], length 0
B&....................O..^..
qX(7qX(6...
21:20:15.112773 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1, win
128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0
B&..^......E........O...
qX(7qX(7
21:20:15.112809 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [P.], seq 1:581,
ack 1, win 128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 580
B&..^.......m.......O...
qX(7qX(7GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1
Host: localhost:51200
Accept: application/json
X-GP-FORMAT: GPDBWritable
X-GP-ATTRS: 2
X-GP-ATTR-NAME0: key
X-GP-ATTR-TYPECODE0: 25
X-GP-ATTR-TYPENAME0: text
X-GP-ATTR-NAME1: value
X-GP-ATTR-TYPECODE1: 25
X-GP-ATTR-TYPENAME1: text
X-GP-SEGMENT-ID: -100005432
X-GP-SEGMENT-COUNT: 0
X-GP-XID: 2725021
X-GP-ALIGNMENT: 8
X-GP-URL-HOST: localhost
X-GP-URL-PORT: 51200
X-GP-DATA-DIR: default.kdtest
X-GP-Profile: Hive
X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive
X-GP-HAS-FILTER: 0
21:20:15.112813 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [.], ack 581, win
131, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0
B),....C..............O..^..
qX(7qX(7
21:20:15.305723 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [P.], seq 1:1443,
ack 581, win 131, options [nop,nop,TS val 1901603063 ecr 1901602871], length
1442
B),...................O..^..
qX(.qX(7HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Content-Length: 1305
Date: Tue, 14 Feb 2017 05:20:15 GMT
{"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]}
21:20:15.305733 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1443, win
134, options [nop,nop,TS val 1901603063 ecr 1901603063], length 0
B),.^......<........O...
qX(.qX(.
21:20:15.305897 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [F.], seq 581, ack
1443, win 134, options [nop,nop,TS val 1901603064 ecr 1901603063], length 0
B),.^......<}.......O...
qX(.qX(.
21:20:15.305961 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [F.], seq 1443, ack
582, win 131, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0
B)-....<~.............O..^..
qX(.qX(.
21:20:15.305968 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1444, win
134, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0
B)-.^......<{.......O...
qX(.qX(.
21:20:15.348058 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [S], seq
1316022209, win 65483, options [mss 65495,sackOK,TS val 1901603106 ecr
1901603064,nop,wscale 9], length 0
E..<..@.@.m.........O...Np...........=.........
qX)"qX(....
21:20:15.348069 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [S.], seq
124605876, ack 1316022210, win 65483, options [mss 65495,sackOK,TS val
1901603106 ecr 1901603106,nop,wscale 9], length 0
E..<..@.@.<...........O..mU.Np......o..........
qX)"qX)"...
21:20:15.348078 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1, win
128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0
E..4..@.@.m.........O...Np...mU............
qX)"qX)"
21:20:15.348138 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [P.], seq 1:581,
ack 1, win 128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 580
E..x..@.@.k.........O...Np...mU......m.....
qX)"qX)"GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1
Host: localhost:51200
Accept: application/json
X-GP-FORMAT: GPDBWritable
X-GP-ATTRS: 2
X-GP-ATTR-NAME0: key
X-GP-ATTR-TYPECODE0: 25
X-GP-ATTR-TYPENAME0: text
X-GP-ATTR-NAME1: value
X-GP-ATTR-TYPECODE1: 25
X-GP-ATTR-TYPENAME1: text
X-GP-SEGMENT-ID: -100005432
X-GP-SEGMENT-COUNT: 0
X-GP-XID: 2725021
X-GP-ALIGNMENT: 8
X-GP-URL-HOST: localhost
X-GP-URL-PORT: 51200
X-GP-DATA-DIR: default.kdtest
X-GP-Profile: Hive
X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive
X-GP-HAS-FILTER: 0
21:20:15.348144 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [.], ack 581, win
131, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0
E..4|.@.@..*..........O..mU.Np.............
qX)"qX)"
21:20:15.523187 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [P.], seq 1:1443,
ack 581, win 131, options [nop,nop,TS val 1901603281 ecr 1901603106], length
1442
E...|.@.@.............O..mU.Np.............
qX).qX)"HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Content-Length: 1305
Date: Tue, 14 Feb 2017 05:20:15 GMT
{"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]}
21:20:15.523197 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1443, win
134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4..@.@.m.........O...Np...m[W...........
qX).qX).
21:20:15.523402 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [F.], seq 581, ack
1443, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4..@.@.m.........O...Np...m[W...........
qX).qX).
21:20:15.523457 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [F.], seq 1443, ack
582, win 131, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4|.@.@..(..........O..m[WNp.............
qX).qX).
21:20:15.523464 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1444, win
134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4..@.@.m.........O...Np...m[X...........
qX).qX).
{code}
was (Author: kdunn926):
Additionally, I observed all this using the following command, on the HAWQ
master. First, start the {{tcpdump}} trace, then invoke a SELECT from a
previously defined PXF table, either using HCatalog directly, or manually
defining a PXF external table. A similar approach could be used to observe
datanode traffic during read/write operations.
{code}
$ tcpdump -n port 51200 -A
{code}
Here is the conversation output:
{code}
21:20:14.632410 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [S], seq
3498721498, win 65483, options [mss 65495,sackOK,TS val 1901602390 ecr
1901547904,nop,wscale 9], length 0
E..<f.@.@..-........O.....@.........._.........
qX&VqWQ....
21:20:14.632421 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [S.], seq
3752275736, ack 3498721499, win 65483, options [mss 65495,sackOK,TS val
1901602390 ecr 1901602390,nop,wscale 9], length 0
E..<..@.@.<...........O.../...@................
qX&VqX&V...
21:20:14.632428 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 1, win
128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0
E..4f.@.@..4........O.....@.../............
qX&VqX&V
21:20:14.632602 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [P.], seq 1:318,
ack 1, win 128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 317
E..qf.@.@...........O.....@.../......e.....
qX&VqX&VGET /pxf/v14/Metadata/getMetadata?profile=Hive&pattern=default.kdtest
HTTP/1.1
Host: localhost:51200
Accept: application/json
X-GP-SEGMENT-ID: -100005432
X-GP-SEGMENT-COUNT: 0
X-GP-XID: 2725021
X-GP-ALIGNMENT: 8
X-GP-URL-HOST: localhost
X-GP-URL-PORT: 51200
X-GP-URI: localhost:51200/
X-GP-HAS-FILTER: 0
21:20:14.632607 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [.], ack 318, win
130, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0
E..4 O@.@.3s..........O.../...B............
qX&VqX&V
21:20:15.084890 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [P.], seq 1:269,
ack 318, win 130, options [nop,nop,TS val 1901602843 ecr 1901602390], length 268
E..@ P@.@.2f..........O.../...B......4.....
qX(.qX&VHTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Content-Length: 132
Date: Tue, 14 Feb 2017 05:20:15 GMT
{"PXFMetadata":[{"item":{"path":"default","name":"kdtest"},"fields":[{"name":"key","type":"text"},{"name":"value","type":"text"}]}]}
21:20:15.084900 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 269, win
130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4f.@.@..2........O.....B...0%.....!.....
qX(.qX(.
21:20:15.085229 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [F.], seq 318, ack
269, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4f.@.@..1........O.....B...0%..... .....
qX(.qX(.
21:20:15.085286 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [F.], seq 269, ack
319, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4 Q@.@.3q..........O...0%..B............
qX(.qX(.
21:20:15.085294 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 270, win
130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0
E..4f.@.@..0........O.....B...0&...........
qX(.qX(.
21:20:15.112739 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [S], seq 222439143,
win 65483, options [mss 65495,sackOK,TS val 1901602870 ecr
1901602843,nop,wscale 9], length 0
B&......................
qX(6qX(....
21:20:15.112765 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [S.], seq
2606634976, ack 222439144, win 65483, options [mss 65495,sackOK,TS val
1901602871 ecr 1901602870,nop,wscale 9], length 0
B&....................O..^..
qX(7qX(6...
21:20:15.112773 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1, win
128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0
B&..^......E........O...
qX(7qX(7
21:20:15.112809 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [P.], seq 1:581,
ack 1, win 128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 580
B&..^.......m.......O...
qX(7qX(7GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1
Host: localhost:51200
Accept: application/json
X-GP-FORMAT: GPDBWritable
X-GP-ATTRS: 2
X-GP-ATTR-NAME0: key
X-GP-ATTR-TYPECODE0: 25
X-GP-ATTR-TYPENAME0: text
X-GP-ATTR-NAME1: value
X-GP-ATTR-TYPECODE1: 25
X-GP-ATTR-TYPENAME1: text
X-GP-SEGMENT-ID: -100005432
X-GP-SEGMENT-COUNT: 0
X-GP-XID: 2725021
X-GP-ALIGNMENT: 8
X-GP-URL-HOST: localhost
X-GP-URL-PORT: 51200
X-GP-DATA-DIR: default.kdtest
X-GP-Profile: Hive
X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive
X-GP-HAS-FILTER: 0
21:20:15.112813 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [.], ack 581, win
131, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0
B),....C..............O..^..
qX(7qX(7
21:20:15.305723 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [P.], seq 1:1443,
ack 581, win 131, options [nop,nop,TS val 1901603063 ecr 1901602871], length
1442
B),...................O..^..
qX(.qX(7HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Content-Length: 1305
Date: Tue, 14 Feb 2017 05:20:15 GMT
{"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]}
21:20:15.305733 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1443, win
134, options [nop,nop,TS val 1901603063 ecr 1901603063], length 0
B),.^......<........O...
qX(.qX(.
21:20:15.305897 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [F.], seq 581, ack
1443, win 134, options [nop,nop,TS val 1901603064 ecr 1901603063], length 0
B),.^......<}.......O...
qX(.qX(.
21:20:15.305961 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [F.], seq 1443, ack
582, win 131, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0
B)-....<~.............O..^..
qX(.qX(.
21:20:15.305968 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1444, win
134, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0
B)-.^......<{.......O...
qX(.qX(.
21:20:15.348058 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [S], seq
1316022209, win 65483, options [mss 65495,sackOK,TS val 1901603106 ecr
1901603064,nop,wscale 9], length 0
E..<..@.@.m.........O...Np...........=.........
qX)"qX(....
21:20:15.348069 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [S.], seq
124605876, ack 1316022210, win 65483, options [mss 65495,sackOK,TS val
1901603106 ecr 1901603106,nop,wscale 9], length 0
E..<..@.@.<...........O..mU.Np......o..........
qX)"qX)"...
21:20:15.348078 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1, win
128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0
E..4..@.@.m.........O...Np...mU............
qX)"qX)"
21:20:15.348138 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [P.], seq 1:581,
ack 1, win 128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 580
E..x..@.@.k.........O...Np...mU......m.....
qX)"qX)"GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1
Host: localhost:51200
Accept: application/json
X-GP-FORMAT: GPDBWritable
X-GP-ATTRS: 2
X-GP-ATTR-NAME0: key
X-GP-ATTR-TYPECODE0: 25
X-GP-ATTR-TYPENAME0: text
X-GP-ATTR-NAME1: value
X-GP-ATTR-TYPECODE1: 25
X-GP-ATTR-TYPENAME1: text
X-GP-SEGMENT-ID: -100005432
X-GP-SEGMENT-COUNT: 0
X-GP-XID: 2725021
X-GP-ALIGNMENT: 8
X-GP-URL-HOST: localhost
X-GP-URL-PORT: 51200
X-GP-DATA-DIR: default.kdtest
X-GP-Profile: Hive
X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive
X-GP-HAS-FILTER: 0
21:20:15.348144 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [.], ack 581, win
131, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0
E..4|.@.@..*..........O..mU.Np.............
qX)"qX)"
21:20:15.523187 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [P.], seq 1:1443,
ack 581, win 131, options [nop,nop,TS val 1901603281 ecr 1901603106], length
1442
E...|.@.@.............O..mU.Np.............
qX).qX)"HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Content-Length: 1305
Date: Tue, 14 Feb 2017 05:20:15 GMT
{"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]}
21:20:15.523197 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1443, win
134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4..@.@.m.........O...Np...m[W...........
qX).qX).
21:20:15.523402 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [F.], seq 581, ack
1443, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4..@.@.m.........O...Np...m[W...........
qX).qX).
21:20:15.523457 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [F.], seq 1443, ack
582, win 131, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4|.@.@..(..........O..m[WNp.............
qX).qX).
21:20:15.523464 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1444, win
134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0
E..4..@.@.m.........O...Np...m[X...........
qX).qX).
{code}
> Document HAWQ to PXF APIs
> -------------------------
>
> Key: HAWQ-1234
> URL: https://issues.apache.org/jira/browse/HAWQ-1234
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: PXF
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Attachments: PXFAdvancedStatsplan.pdf
>
>
> It would be very useful to start documenting HAWQ to PXF APIs. The right
> places to start are:
> * libcurl (a thin wrapper for making HAWQ C code be able to do REST calls):
> https://github.com/apache/incubator-hawq/blob/master/src/include/access/libchurl.h
> https://github.com/apache/incubator-hawq/blob/master/src/backend/access/external/libchurl.c
> * pxfmasterapi (mostly metadata calls that master is doing):
> https://github.com/apache/incubator-hawq/blob/master/src/backend/access/external/pxfmasterapi.c
> Here you will find how HAWQ via PXF pulls using a REST API to get external
> metadata and some logic to parse the JSON response.
> * gpbridgeapi (segment calls to PXF):
> https://github.com/apache/incubator-hawq/blob/master/src/bin/gpfusion/gpbridgeapi.c
> Here you will find other examples of (read and write calls) used to fetch
> external data.
> Design doc on PXF's support for analyze (pxf's analyzer) is attached
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
