What does agent-type add? In the sense that if we're pre-parsing the user agent, surely the difference is between "WHERE agent_type != 'spider'" and "WHERE user_agent_map['device_family'] != 'Spider'"? Does agent_type include the isCrawler UDF results?
On 10 April 2015 at 16:47, Joseph Allemandou <[email protected]> wrote: > And I forgot one field : > > is_zero - True if a request is made on a zero provider. > > > On Fri, Apr 10, 2015 at 10:36 PM, Leila Zia <[email protected]> wrote: >> >> Hi Joseph, >> >> Thanks for the update, and for doing this. These three items make the >> analysis of the data much easier on our end. We've had many requests in the >> past that required agent_type and access_method information and having them >> readily available is awesome! :-) >> >> Have a great weekend! >> >> Leila >> >> On Fri, Apr 10, 2015 at 1:21 PM, Joseph Allemandou >> <[email protected]> wrote: >>> >>> Hi Analytics people, >>> >>> Today happens another bunch of addition to the refined webrequest table >>> in hive. >>> Now the table contains: >>> >>> ts - The unix timestamp (milliseconds) version of the dt date >>> access_method - The method used to access the site, being one of the >>> three [mobile app | mobile web | desktop] >>> agent_type - To differentiate easily between spiders and users (more >>> values may be added later). >>> >>> These additions are based on the "tags", as defined here: >>> https://meta.wikimedia.org/wiki/Research:Page_view >>> >>> Have a good weekend ! >>> >>> -- >>> Joseph Allemandou >>> Data Engineer @ Wikimedia Foundation >>> IRC: joal >>> >>> _______________________________________________ >>> Analytics mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/analytics >>> >> >> >> _______________________________________________ >> Analytics mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/analytics >> > > > > -- > Joseph Allemandou > Data Engineer @ Wikimedia Foundation > IRC: joal > > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics > -- Oliver Keyes Research Analyst Wikimedia Foundation _______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
