Hi,
I'm using Pig to analyze some log files. We would like to find the last
time a URL has been accessed. I've pulled out the path and the time, but
I'm having difficulty create a relation of paths and latest access. My
thought was to group the relation by the path and then order the bags on
the time. The code looks like,
paths = group formatted by path;
final = foreach paths {
-- Pull out the bag and sort the tuples by the date field.
sorted = order formatted by lastAccess DESC;
biggest = limit sorted 1;
-- Return the first tuple, it should have the largest timestamp.
generate group, biggest;
};
When I illustrate this though, the result is the group and then a bag of
the tuples, not a single tuple,
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| raw | line:chararray
|
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090
images.meez.com[09/Oct/2010:23:29:30 +0000] 68.96.191.249 Anonymous
90B50123876665F7 REST.GET.OBJECT
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png "GET
/resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png HTTP/1.1" 200
- 54023 54023 15 11 "
http://www.meez.com/roomz.dm?act=hangoutz&hood_id=arcadia"; "Mozilla/5.0
(Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko)
Chrome/6.0.472.63 Safari/534.3" - |
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090
images.meez.com[09/Oct/2010:23:28:46 +0000] 75.16.59.201 Anonymous
B539D577B740FDEA REST.GET.OBJECT
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png "GET
/resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png HTTP/1.1" 304
- - 54023 5 - "http://www.meez.com/roomz.dm?act=hangoutz&hood_id=arcadia";
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705;
Media Center PC 3.1; .NET CLR 1.1.4322)" - |
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090
images.meez.com[09/Oct/2010:23:29:30 +0000] 68.96.191.249 Anonymous
90B50123876665F7 REST.GET.OBJECT
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png "GET
/resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png HTTP/1.1" 200
- 54023 54023 15 11 "
http://www.meez.com/roomz.dm?act=hangoutz&hood_id=arcadia"; "Mozilla/5.0
(Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko)
Chrome/6.0.472.63 Safari/534.3" - |
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090
images.meez.com[09/Oct/2010:23:29:30 +0000] 68.96.191.249 Anonymous
90B50123876665F7 REST.GET.OBJECT
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png "GET
/resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png HTTP/1.1" 200
- 54023 54023 15 11 "
http://www.meez.com/roomz.dm?act=hangoutz&hood_id=arcadia"; "Mozilla/5.0
(Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko)
Chrome/6.0.472.63 Safari/534.3" - |
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090
images.meez.com[09/Oct/2010:23:28:46 +0000] 75.16.59.201 Anonymous
B539D577B740FDEA REST.GET.OBJECT
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png "GET
/resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png HTTP/1.1" 304
- - 54023 5 - "http://www.meez.com/roomz.dm?act=hangoutz&hood_id=arcadia";
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705;
Media Center PC 3.1; .NET CLR 1.1.4322)" - |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| pieces | garbage:chararray
| host:chararray | date:chararray | ip:chararray |
user:chararray | garbage2:chararray | operation:chararray |
path:chararray |
command:chararray |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090 |
images.meez.com | 09/Oct/2010:23:29:30 +0000 | 68.96.191.249 |
Anonymous | 90B50123876665F7 | REST.GET.OBJECT |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png | "GET
|
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090 |
images.meez.com | 09/Oct/2010:23:28:46 +0000 | 75.16.59.201 |
Anonymous | B539D577B740FDEA | REST.GET.OBJECT |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png | "GET
|
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090 |
images.meez.com | 09/Oct/2010:23:29:30 +0000 | 68.96.191.249 |
Anonymous | 90B50123876665F7 | REST.GET.OBJECT |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png | "GET
|
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090 |
images.meez.com | 09/Oct/2010:23:29:30 +0000 | 68.96.191.249 |
Anonymous | 90B50123876665F7 | REST.GET.OBJECT |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png | "GET
|
| |
5f49db3521a7480e2a3ff2da407bdae72dbd22a6d9db4c81377182bedb68d090 |
images.meez.com | 09/Oct/2010:23:28:46 +0000 | 75.16.59.201 |
Anonymous | B539D577B740FDEA | REST.GET.OBJECT |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png | "GET
|
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
| formatted | path:chararray
| lastAccess:long |
------------------------------------------------------------------------------------------------------
| |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png |
1286666970000 |
| |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png |
1286666926000 |
| |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png |
1286666970000 |
| |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png |
1286666970000 |
| |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png |
1286666926000 |
------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| paths | group:chararray
| formatted:bag{:tuple(path:chararray,lastAccess:long)}
|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png
| {(resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png,
1286666970000), ...,
(resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png,
1286666926000)} |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------
| final.sorted | path:chararray
| lastAccess:long |
---------------------------------------------------------------------------------------------------------
| |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png |
1286666970000 |
---------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------
| final.biggest | path:chararray
| lastAccess:long |
----------------------------------------------------------------------------------------------------------
| |
resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png |
1286666970000 |
----------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| final | group:chararray
| biggest:bag{:tuple(path:chararray,lastAccess:long)}
|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png
| {(resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png,
1286666970000), ...,
(resources/world/spaces-1.0/Icons/Arcadia/graveyardchapel.png,
1286666926000)} |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
