Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/195285
Change subject: Correct concurrency bug on mobile_apps uniques computation
..
Correct concurrency bug on mobile_apps uniques computation
Joal has submitted this change and it was merged.
Change subject: Adapt bin/refinery-drop-webrequest-partitions to work with
refined table
..
Adapt bin/refinery-drop-webrequest-partitions to work with refined table
I made a
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/195952
Change subject: Move UAParser wrapper to refinery-core and update refinery-hive
accordingly.
..
Move UAParser wrapper to refinery-core and
Joal has submitted this change and it was merged.
Change subject: Add 0.0.8 refinery jars and update symlinks
..
Add 0.0.8 refinery jars and update symlinks
Change-Id: I9fabc1ffc0ef9aa4fb2c58d592f94b0c2276b23a
---
A
Joal has submitted this change and it was merged.
Change subject: Bump refinery version to 0.0.8 for mediacounts
..
Bump refinery version to 0.0.8 for mediacounts
This bump brings in the relevant fixes:
* Make custom file
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/200623
Change subject: Correct refined webrequest to have null x_analytics_map instead
of not meaningfull one.
..
Correct refined webrequest to have
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/194199
Change subject: Add mobile monthly uniques job in oozie.
..
Add mobile monthly uniques job in oozie.
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/194400
Change subject: Refactor mobile_apps_uniques_daily to match newly baked monthly.
..
Refactor mobile_apps_uniques_daily to match newly baked
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/192363
Change subject: Add client ip and geocoded data to refined webrequest table.
..
Add client ip and geocoded data to refined webrequest table.
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/192824
Change subject: Add record_version field to refined webrequest table.
..
Add record_version field to refined webrequest table.
Update oozie
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/192891
Change subject: Add refinery_hive_jar_version as a parameter in oozie bundle
properties.
..
Add refinery_hive_jar_version as a parameter in
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/198789
Change subject: Add ua_map and x_analytics_map fields to refine table.
..
Add ua_map and x_analytics_map fields to refine table.
Use
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/205895
Change subject: Correct mysql handler to correctly use optional replace
parameter.
..
Correct mysql handler to correctly use optional replace
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/205277
Change subject: [WIP] Add spark utility to parse wikidumps
..
[WIP] Add spark utility to parse wikidumps
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/205833
Change subject: Patch pageview definition
..
Patch pageview definition
Modify maven pom.xml files not to build uber jars and to take advantage
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/207814
Change subject: Correct webrequest refinement bug about spiders not labelled
correctly
..
Correct webrequest refinement bug about spiders not
Joal has submitted this change and it was merged.
Change subject: Use wmfuuid from x_analytics_map for app install id if
available else default to the query parameter
..
Use wmfuuid from x_analytics_map for app install id if
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/209532
Change subject: Bump refinery-core and refinery-hive jars version to 0.11.0,
add referer_class to webrequest refine table and update webrequest refined
recordVersion
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/210701
Change subject: Modify forwarder to accept multiple outputs using writers and
adapt some writer function to accept raw data.
..
Modify
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/210869
Change subject: Modify EventLogging forwarder configuration
..
Modify EventLogging forwarder configuration
EventLogging forwarder now accepts
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/202914
Change subject: Add access_method, client_type and is_zero fields to refined
webrequest table.
..
Add access_method, client_type and is_zero
Joal has submitted this change and it was merged.
Change subject: Modify oozie parameters at workflow level
..
Modify oozie parameters at workflow level
Remove queue_name for mark_data_set_done subworkflow (not a map-recude
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216930
Change subject: Modify oozie parameters at workflow level
..
Modify oozie parameters at workflow level
Remove queue_name for mark_data_set_done
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215628
Change subject: Add host normalization udf and underlying functions
..
Add host normalization udf and underlying functions
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217229
Change subject: Add 2 fields and 1 update to refine table
..
Add 2 fields and 1 update to refine table
Add pageview_info and normalized_host
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217232
Change subject: Add oozie job for pageview_hourly aggregation
..
Add oozie job for pageview_hourly aggregation
Add hive creation table file.
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217226
Change subject: Correct UAParser not to fail for malformed wmf_app
..
Correct UAParser not to fail for malformed wmf_app
Previous commit had a
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/219862
Change subject: Fix 2 projectview_hourly aggregation bugs.
..
Fix 2 projectview_hourly aggregation bugs.
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/219181
Change subject: Prepare changelog for version 0.0.14
..
Prepare changelog for version 0.0.14
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/220752
Change subject: Add new projectview to projectcounts aggregation
..
Add new projectview to projectcounts aggregation
Remove projectcounts data
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/220426
Change subject: Add webstatcollector projectview transformation
..
Add webstatcollector projectview transformation
Add oozie job files
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/219195
Change subject: Add maven-scala-plugin version for release to work
..
Add maven-scala-plugin version for release to work
Change-Id:
Joal has submitted this change and it was merged.
Change subject: Add maven-scala-plugin version for release to work
..
Add maven-scala-plugin version for release to work
Change-Id: I200d09dd4e6c5213bb8c64800ec3484029f8d861
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/219246
Change subject: Add refinery-job jar version 0.0.14
..
Add refinery-job jar version 0.0.14
Change-Id: I11acf2843b628be268960839c90220dc7bdcd950
Joal has submitted this change and it was merged.
Change subject: Correct pageview and projectview bugs
..
Correct pageview and projectview bugs
Change count to view_count.
Add missing comma in table creation script
Joal has submitted this change and it was merged.
Change subject: Add oozie job to schedule mobile app session metrics spark job.
..
Add oozie job to schedule mobile app session metrics spark job.
Includes - Add hive external
Joal has submitted this change and it was merged.
Change subject: Add driver memory as a configurable property to Spark job
..
Add driver memory as a configurable property to Spark job
Bug fix for app session metrics job not
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/221185
Change subject: Correct bug in projectview archive workflow
..
Correct bug in projectview archive workflow
Archive sub-worflow needs directory
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217468
Change subject: Correct bug in refine table about pageview_info
..
Correct bug in refine table about pageview_info
Pageview_info was computed
Joal has submitted this change and it was merged.
Change subject: Correct bug in refine table about pageview_info
..
Correct bug in refine table about pageview_info
Pageview_info was computed for every webrequest while it
Joal has submitted this change and it was merged.
Change subject: Add user parameter in pageview hourly properties
..
Add user parameter in pageview hourly properties
Change-Id: Ied29ae30722391b1d5da19510d52254cf1169d81
---
M
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217515
Change subject: Correct typo in pageview_hourly table creation
..
Correct typo in pageview_hourly table creation
Change-Id:
Joal has submitted this change and it was merged.
Change subject: Add oozie job for pageview_hourly aggregation
..
Add oozie job for pageview_hourly aggregation
Add hive creation table file.
Add oozie job files.
Bug: T99931
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217338
Change subject: Add oozie job for projectview_hourly aggregation
..
Add oozie job for projectview_hourly aggregation
Add hive creation table
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217335
Change subject: Add 0.0.12 refinery core and hive jars.
..
Add 0.0.12 refinery core and hive jars.
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217337
Change subject: Add oozie job for pageview_hourly aggregation
..
Add oozie job for pageview_hourly aggregation
Add hive creation table file.
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217322
Change subject: Rename project dialect to language_variant
..
Rename project dialect to language_variant
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217805
Change subject: Correct bug in host normalization and add test
..
Correct bug in host normalization and add test
Previous version of host
Joal has submitted this change and it was merged.
Change subject: Productionize app session metrics
..
Productionize app session metrics
- Parse args using scopt
- Move away from HiveContext to reading Parquet files directly
-
Joal has submitted this change and it was merged.
Change subject: Add Apps session metrics job
..
Add Apps session metrics job
Uses Twitter's algebird to compute percentiles
Removed geometric mean computation (not doable)
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217835
Change subject: Set number of reducers for hive pageview job
..
Set number of reducers for hive pageview job
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/217873
Change subject: Bump jar version to 0.0.13
..
Bump jar version to 0.0.13
Add core and hive 0.0.13 jars
Bump refinery hive jar version number to
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/214349
Change subject: Add get pageview_info udf and underlying functions
..
Add get pageview_info udf and underlying functions
Update
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/222127
Change subject: Correct error in wikimetrics projectview simlink
..
Correct error in wikimetrics projectview simlink
Link was set to
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/221611
Change subject: Add projectview to metrics website
..
Add projectview to metrics website
Modify aggregator-data setup to be
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/212541
Change subject: Add pageview aggregation and parquet merger.
..
Add pageview aggregation and parquet merger.
Pageview aggregation is done
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/212542
Change subject: Add pageview aggregation oozie job and hive table.
..
Add pageview aggregation oozie job and hive table.
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231010
Change subject: Remove webrequest bits and add webrequest maps
..
Remove webrequest bits and add webrequest maps
Remove webrequest bits data
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231241
Change subject: Bump refinery-core and refinery-hive to 0.0.15
..
Bump refinery-core and refinery-hive to 0.0.15
Add jar files.
Bump version in
Joal has submitted this change and it was merged.
Change subject: Remove webrequest bits and add webrequest maps
..
Remove webrequest bits and add webrequest maps
Remove webrequest bits data (camus, partition loading,
Joal has submitted this change and it was merged.
Change subject: Bump refinery-core and refinery-hive to 0.0.15
..
Bump refinery-core and refinery-hive to 0.0.15
Add jar files.
Bump version in oozie refine job, as well as
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231285
Change subject: Update dataset check dump script
..
Update dataset check dump script
Remove webrequest bits and legacy_tsv_5xx_bits
Add
Joal has submitted this change and it was merged.
Change subject: Upgrade camus to version wmf6
..
Upgrade camus to version wmf6
Bug: T109860
Change-Id: I7ac7f08b898f186115e087879b41896bc9235dff
---
M artifacts/camus-wmf.jar
D
Joal has submitted this change and it was merged.
Change subject: [WIP] Generate hourly aggregate statistics about webrequest
sequence stats
..
[WIP] Generate hourly aggregate statistics about webrequest sequence stats
This
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/232448
Change subject: Add CassandraXSVLoader to refinery-job
..
Add CassandraXSVLoader to refinery-job
The CassandraXSVLoader runs a map-reduce job
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/232177
Change subject: Correct bug in pageview definition
..
Correct bug in pageview definition
prevent outreach.wikimedia.org and
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/228247
Change subject: Remove empty projects from intermediate pageviews
..
Remove empty projects from intermediate pageviews
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/229369
Change subject: Correct Pageview host regexps and tests
..
Correct Pageview host regexps and tests
Pageview host regexps were incorrect, as
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/229372
Change subject: Prevent failure in case of wrong line format
..
Prevent failure in case of wrong line format
Instead of raising a RuntimeError,
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/228010
Change subject: Add user_agent_map to intermediate pageview
..
Add user_agent_map to intermediate pageview
user_agent_map field added to table
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/228011
Change subject: Remove empty projects from intermediate pageviews
..
Remove empty projects from intermediate pageviews
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231042
Change subject: Update changelog.md before deployment.
..
Update changelog.md before deployment.
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231545
Change subject: Correct pageview title normalization bug
..
Correct pageview title normalization bug
Pageview title normalization had a bug
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/247847
Change subject: Correct camus-partition-checker to use hdfs conf
..
Correct camus-partition-checker to use hdfs conf
Change-Id:
Joal has submitted this change and it was merged.
Change subject: Archive hourly pageviews in legacy format
..
Archive hourly pageviews in legacy format
Legacy format, also called webstatcollector format in the past, is
used
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/248991
Change subject: Correct two bugs in refinery oozie jobs
..
Correct two bugs in refinery oozie jobs
send_error_email workflow file was
Joal has submitted this change and it was merged.
Change subject: Correct bug in parameter definition
..
Correct bug in parameter definition
output-projectviews parameter was not in the parsed parameters list, generating
an
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249088
Change subject: Correct bug in parameter definition
..
Correct bug in parameter definition
output-projectviews parameter was not in the parsed
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249100
Change subject: Update camus runs
..
Update camus runs
Add run parameter to match script update
Add check parameter to webrequest job
Bug:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249378
Change subject: Correct pageview to dumps synchro
..
Correct pageview to dumps synchro
The path given for synchronization was not correct.
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249407
Change subject: Update changelog.md for v0.0.22
..
Update changelog.md for v0.0.22
Change-Id: I167acaca897352d007a9ee3cf5e969337a0b3a6a
---
M
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249140
Change subject: Update changelog.md preparing for release 0.0.21
..
Update changelog.md preparing for release 0.0.21
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249180
Change subject: Upgrade refinery-camus and refinery-job to 0.0.21
..
Upgrade refinery-camus and refinery-job to 0.0.21
Change-Id:
Joal has submitted this change and it was merged.
Change subject: Add oozie job to compute browser usage reports
..
Add oozie job to compute browser usage reports
It computes 2 reports: mobile-web and desktop+mobile-web.
As
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249368
Change subject: Correct CamusPartitionChecker bug
..
Correct CamusPartitionChecker bug
CamusPartitionChecker was flagging currently importing
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249425
Change subject: Upgrade refinery-job.jar to v0.0.22
..
Upgrade refinery-job.jar to v0.0.22
Change-Id: If3580b0f988b1ff62ed919f204a393edc85ff2ed
Joal has submitted this change and it was merged.
Change subject: Upgrade refinery-job.jar to v0.0.22
..
Upgrade refinery-job.jar to v0.0.22
Change-Id: If3580b0f988b1ff62ed919f204a393edc85ff2ed
---
A
Joal has submitted this change and it was merged.
Change subject: Compress output of pageview legacy files (gzip)
..
Compress output of pageview legacy files (gzip)
Change-Id: I90ef510d1a9eb03d7b64c3b02112facca8e1b040
---
M
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249698
Change subject: Compress output of pageview legacy files (gzip)
..
Compress output of pageview legacy files (gzip)
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249715
Change subject: Correct bug in gzipped legacy pageview filename
..
Correct bug in gzipped legacy pageview filename
Change-Id:
Joal has submitted this change and it was merged.
Change subject: Correct bug in gzipped legacy pageview filename
..
Correct bug in gzipped legacy pageview filename
Change-Id: I1e08171c553e4d7243eac9dfeadb9bf25ee38764
---
M
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/248041
Change subject: Update oozie diagram to reflect current status
..
Update oozie diagram to reflect current status
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/248048
Change subject: Update bin/camus to include CamusPartitionChecker
..
Update bin/camus to include CamusPartitionChecker
Bug:T113252
Change-Id:
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249373
Change subject: Update oozie load job to use _IMPORTED flag
..
Update oozie load job to use _IMPORTED flag
Change-Id:
Joal has submitted this change and it was merged.
Change subject: Fix external table path for session metrics job output
..
Fix external table path for session metrics job output
Change-Id:
Joal has submitted this change and it was merged.
Change subject: Include avro schema in refinery-camus jar
..
Include avro schema in refinery-camus jar
Bug: T117885
Change-Id: I0f47b586f380f10d8c68b5dd0f45fb5a17aae974
---
M
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/251311
Change subject: Improve sorted-json job
..
Improve sorted-json job
Revamp the InputFormat to be parameterized by WikiMediaObjectFactory
Add
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/251497
Change subject: Update mediawiki_CirrusSearchREquestSet camus cron
..
Update mediawiki_CirrusSearchREquestSet camus cron
Add --check parameter
Joal has uploaded a new change for review.
https://gerrit.wikimedia.org/r/244465
Change subject: Update bot regexp to catch http UA
..
Update bot regexp to catch http UA
Change-Id: Ib64061f2eb48c174d084219c627d54bc7dc00896
Joal has submitted this change and it was merged.
Change subject: Add percent loss to refinery-dump-status script
..
Add percent loss to refinery-dump-status script
Adds --percent-loss argument. When used, instead of showing
1 - 100 of 636 matches
Mail list logo