Milimetric has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/403484 )
Change subject: Point data and success flags to the same directory
......................................................................
Point data and success flags to the same directory
While working on re-starting this job, I noticed the output data was
actually going to a different directory than the success flags. I
missed this in manual oozie testing because it has to do with the table
definition, and I was using a different test table in my own db. I'm
not sure how to test this in production, so a sanity check from someone
doing the code review would be nice.
Bug: T170764
Change-Id: I54e5d2017cd6baeb5ec2c1ba0764107b97b3afeb
---
M hive/interlanguage/create_interlanguage_navigation_table.hql
M oozie/interlanguage/daily/README.md
M oozie/interlanguage/datasets.xml
3 files changed, 3 insertions(+), 11 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/refinery
refs/changes/84/403484/1
diff --git a/hive/interlanguage/create_interlanguage_navigation_table.hql
b/hive/interlanguage/create_interlanguage_navigation_table.hql
index 939f8fa..a54629d 100644
--- a/hive/interlanguage/create_interlanguage_navigation_table.hql
+++ b/hive/interlanguage/create_interlanguage_navigation_table.hql
@@ -18,5 +18,5 @@
`date` string COMMENT 'Date in YYYY-MM-DD format'
)
STORED AS PARQUET
-LOCATION '/wmf/data/wmf/interlanguage/navigation'
+LOCATION '/wmf/data/wmf/interlanguage/navigation/daily'
;
diff --git a/oozie/interlanguage/daily/README.md
b/oozie/interlanguage/daily/README.md
index 5cc51cf..6f56798 100644
--- a/oozie/interlanguage/daily/README.md
+++ b/oozie/interlanguage/daily/README.md
@@ -4,4 +4,4 @@
from one wiki project to another. Only counting desktop site browsing.
Output is appended into (yyyy-mm-dd) daily partitions
-in /wmf/data/wmf/interlanguage/navigation
+in /wmf/data/wmf/interlanguage/navigation/daily
diff --git a/oozie/interlanguage/datasets.xml b/oozie/interlanguage/datasets.xml
index 18875b5..fb1d59b 100644
--- a/oozie/interlanguage/datasets.xml
+++ b/oozie/interlanguage/datasets.xml
@@ -16,20 +16,12 @@
that are referred from another wiki within the same project family but
in a different language. This allows the tracking of inter-language user
navigation.
-
- Note that we do not use “${...}” but “${"$"}{...}", as dataset files are
- passed to EL twice in cascade, and in the first EL level, ${MONTH}
- evaluates to the string “${MONTH}”. Hence, we escape the dollar sign in
- “${....}" to “${"$"}{...}”. At the first EL level, “${"$"}” gets turned
- into a dollar sign, and “{...}” is just passed along. Hence, we arrive
- at “${...}” as input for the second EL level. There, the variables hold
- their expected values, and we can start unpadding them.
-->
<dataset name="interlanguage_navigation_daily"
frequency="${coord:days(1)}"
initial-instance="${start_time}"
timezone="Universal">
-
<uri-template>${interlanguage_data_directory}/navigation/daily/date=${YEAR}-${"$"}{MONTH
+ 0}-${"$"}{DAY + 0}</uri-template>
+
<uri-template>${interlanguage_data_directory}/navigation/daily/date=${YEAR}-${MONTH}-${DAY}</uri-template>
<done-flag>_SUCCESS</done-flag>
</dataset>
</datasets>
--
To view, visit https://gerrit.wikimedia.org/r/403484
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I54e5d2017cd6baeb5ec2c1ba0764107b97b3afeb
Gerrit-PatchSet: 1
Gerrit-Project: analytics/refinery
Gerrit-Branch: master
Gerrit-Owner: Milimetric <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
