[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: 09/06/2017 Tobi, Adam, Goran

2017-09-06 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/376217 )

Change subject: 09/06/2017 Tobi, Adam, Goran
..

09/06/2017 Tobi, Adam, Goran

Change-Id: Ibff2581a256284bcbb7ce2bc9c5e8f5bef434f03
---
M README.md
A WDCM.Rproj
M WDCM_Collect_Items.R
A WDCM_Dahsboard/server.R
A WDCM_Dahsboard/ui.R
A WDCM_OverviewDashboard/server.R
A WDCM_OverviewDashboard/ui.R
M WDCM_Pre-Process.R
A WDCM_Process_v2.R
M WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
A WDC_Aux.R
A _misc/.directory
A _misc/SqoopRuntime.png
A _misc/WDCM_Navigate.html
A _misc/WDCM_OverviewDashboard_Description.html
A _misc/clienttables.tsv
A _misc/databasesTest.tsv
A _misc/hiveQLQuery.hql
A _misc/hiveQLQueryTest.hql
A _misc/projectsTracking.csv
A _misc/wdcmSearchReport_2017-08-01 08:18:50.csv
A _misc/wdcmSqoopReport_2017-07-21 10:31:45.csv
A creds.csv
A currentStats.csv
M server.R
M ui.R
26 files changed, 7,008 insertions(+), 56 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/17/376217/1


-- 
To view, visit https://gerrit.wikimedia.org/r/376217
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibff2581a256284bcbb7ce2bc9c5e8f5bef434f03
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: 09/06/2017 Tobi, Adam, Goran [correction]

2017-09-06 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/376217 )

Change subject: 09/06/2017 Tobi, Adam, Goran [correction]
..


09/06/2017 Tobi, Adam, Goran [correction]

Change-Id: Ibff2581a256284bcbb7ce2bc9c5e8f5bef434f03
---
M README.md
A WDCM.Rproj
M WDCM_Collect_Items.R
A WDCM_Dahsboard/server.R
A WDCM_Dahsboard/ui.R
A WDCM_OverviewDashboard/server.R
A WDCM_OverviewDashboard/ui.R
M WDCM_Pre-Process.R
A WDCM_Process_v2.R
M WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
A WDC_Aux.R
A _misc/.directory
A _misc/WDCM_Navigate.html
A _misc/WDCM_OverviewDashboard_Description.html
A _misc/clienttables.tsv
A _misc/databasesTest.tsv
A _misc/hiveQLQuery.hql
A _misc/hiveQLQueryTest.hql
A _misc/projectsTracking.csv
A _misc/wdcmSearchReport_2017-08-01 08:18:50.csv
A _misc/wdcmSqoopReport_2017-07-21 10:31:45.csv
A currentStats.csv
M server.R
M ui.R
24 files changed, 7,006 insertions(+), 56 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved




-- 
To view, visit https://gerrit.wikimedia.org/r/376217
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibff2581a256284bcbb7ce2bc9c5e8f5bef434f03
Gerrit-PatchSet: 2
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: WDCM Usage Dashboard - Crosstabs

2017-09-25 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/380652 )

Change subject: WDCM Usage Dashboard - Crosstabs
..

WDCM Usage Dashboard - Crosstabs

Change-Id: I1ee02278bfa7cba91b79c0c9b7d873cda949cbf1
---
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
2 files changed, 253 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/52/380652/1

diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
index 1f40e6a..57d4b41 100644
--- a/WDCM_UsageDashboard/server.R
+++ b/WDCM_UsageDashboard/server.R
@@ -151,6 +151,13 @@
 totalCategories <- length(wdcmCategory$Category)
 totalProjectTypes <- length(wdcmProjectType$`Project Type`)
 
+### --- prepare search constants for Tabs/Crosstabs
+search_projectTypes <- paste("_", projectTypes, sep = "")
+unzip_projectTypes <- lapply(projectTypes, function(x) {
+  wdcmProject$Project[which(wdcmProject$`Project Type` %in% x)]
+})
+names(unzip_projectTypes) <- search_projectTypes
+
 ### --- shinyServer
 shinyServer(function(input, output, session) {
   
@@ -531,6 +538,178 @@
   return(NULL)
 }
   })
+  
+  ### --
+  ### --- TABS AND CROSSTABS
+  ### --
+  
+  ### --- SELECT: update select 'selectProject'
+  updateSelectizeInput(session,
+   'selectProject',
+   choices = c(projects, paste("_", projectTypes, sep="")),
+   selected = projects[round(runif(5, 1, 
length(projects)))],
+   server = TRUE)
+  
+  ### --- SELECT: update select 'selectCategories'
+  updateSelectizeInput(session,
+   'selectCategories',
+   choices = categories,
+   selected = categories[round(runif(3, 1, 
length(categories)))],
+   server = TRUE)
+  
+  # - OBSERVE: input$applySelection
+  observeEvent(input$applySelection, {
+
+### --- selected projects:
+selectedProjects <- character()
+wUnzip <- which(names(unzip_projectTypes) %in% 
isolate(input$selectProject))
+if (length(wUnzip > 0)) {
+  selectedProjects <- unname(do.call(c, unzip_projectTypes[wUnzip]))
+}
+wSel <- which(projects %in% isolate(input$selectProject))
+if (length(wSel > 0)) {
+  selectedProjects <- c(selectedProjects, projects[wSel])
+}
+selectedProjects <- unique(selectedProjects)
+output$testSelectedProjects <- renderText({
+  paste(selectedProjects, collapse = ", ", sep = "")
+})
+
+### --- selected categories:
+selectedCategories <- isolate(input$selectCategories)
+  
+ ---  Chart: tabulations_projectsChart
+output$tabulations_projectsChart <- renderPlot({
+  # - Chart Frame for output$tabulations_projectsChart
+  plotFrame <- wdcmProjectCategory %>%
+filter(Project %in% selectedProjects & Category %in% 
selectedCategories) %>%
+group_by(Project) %>% 
+summarise(Usage = sum(Usage)) %>%
+arrange(desc(Usage))
+  # - top 25 projects:
+  if (dim(plotFrame)[1] > 25) {
+plotFrame <- plotFrame[1:25, ]
+  }
+  plotFrame$Project <- factor(plotFrame$Project, 
+  levels = 
plotFrame$Project[order(-plotFrame$Usage)])
+  # - express labels as K, M:
+  plotFrame$Label <- sapply(plotFrame$Usage, function(x) {
+if (x >= 1e+03 & x < 1e+06) {
+  out <- paste(round(x/1e+03, 1), "K", sep = "")
+} else if (x > 1e+06) {
+  out <- paste(round(x/1e+06, 1), "M", sep = "")
+} else {
+  out <- as.character(x)
+}
+return(out)
+  })
+  # - Plot
+  ggplot(plotFrame,
+ aes(x = Project, y = Usage, label = Label)) +
+geom_bar(stat = "identity", width = .6, fill = "#4c8cff") +
+xlab('Projects') + ylab('Entity Usage') +
+ylim(0, max(plotFrame$Usage) + .1*max(plotFrame$Usage)) +
+scale_y_continuous(labels = comma) + 
+geom_label(size = 3, vjust = -.1) +
+theme_minimal() +
+theme(axis.text.x = element_text(angle = 90, size = 12, hjust = 1)) +
+theme(axis.title.x = element_text(size = 12)) +
+theme(axis.title.y = element_text(size = 12)) +
+theme(plot.title = element_text(size = 15)) %>%
+withProgress(message = 'Generating plot',
+ min = 0,
+ max = 1,
+ value = 1, {incProgress(amount = 0)})
+})
+# - Download Frame: tabulations_projectsChart
+tabulations_projectsDownload_Frame <- reactive({
+  plotFrame <- wdcmProjectCategory %>%
+filter(Project %in% selectedProjects & Category %in% 
selectedCategories) %>%
+group_by(Project) %>% 
+summarise(Usage = sum(Usage)) %>%
+arrange(desc(Usage))
+

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Fix Tabs/Crosstabs Init

2017-09-26 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/380892 )

Change subject: Fix Tabs/Crosstabs Init
..

Fix Tabs/Crosstabs Init

Change-Id: Ifa7a7b7bf44fd26543d89bab38f8f7fabbd70d34
---
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
2 files changed, 341 insertions(+), 346 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/92/380892/1

diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
index 62fe121..74f8a9f 100644
--- a/WDCM_UsageDashboard/server.R
+++ b/WDCM_UsageDashboard/server.R
@@ -547,26 +547,24 @@
   updateSelectizeInput(session,
'selectProject',
choices = c(projects, paste("_", projectTypes, sep="")),
-   selected = projects[round(runif(5, 1, 
length(projects)))],
+   selected = c("_Wikipedia", "_Wikinews", "_Wiktionary"),
server = TRUE)
   
   ### --- SELECT: update select 'selectCategories'
   updateSelectizeInput(session,
'selectCategories',
choices = categories,
-   selected = categories[round(runif(3, 1, 
length(categories)))],
+   selected = categories[round(runif(6, 1, 
length(categories)))],
server = TRUE)
   
-  # - OBSERVE: input$applySelection
-  observeEvent(input$applySelection, {
-
+  tabsDataset <- reactive({
 ### --- selected projects:
 selectedProjects <- character()
-wUnzip <- which(names(unzip_projectTypes) %in% 
isolate(input$selectProject))
+wUnzip <- which(names(unzip_projectTypes) %in% input$selectProject)
 if (length(wUnzip > 0)) {
   selectedProjects <- unname(do.call(c, unzip_projectTypes[wUnzip]))
 }
-wSel <- which(projects %in% isolate(input$selectProject))
+wSel <- which(projects %in% input$selectProject)
 if (length(wSel > 0)) {
   selectedProjects <- c(selectedProjects, projects[wSel])
 }
@@ -574,343 +572,340 @@
 output$testSelectedProjects <- renderText({
   paste(selectedProjects, collapse = ", ", sep = "")
 })
-
 ### --- selected categories:
-selectedCategories <- isolate(input$selectCategories)
-  
- ---  Chart: tabulations_projectsChart
-output$tabulations_projectsChart <- renderPlot({
-  # - Chart Frame for output$tabulations_projectsChart
-  plotFrame <- wdcmProjectCategory %>%
-filter(Project %in% selectedProjects & Category %in% 
selectedCategories) %>%
-group_by(Project) %>% 
-summarise(Usage = sum(Usage)) %>%
-arrange(desc(Usage))
-  # - top 25 projects:
-  if (dim(plotFrame)[1] > 25) {
-plotFrame <- plotFrame[1:25, ]
-  }
-  plotFrame$Project <- factor(plotFrame$Project, 
-  levels = 
plotFrame$Project[order(-plotFrame$Usage)])
-  # - express labels as K, M:
-  plotFrame$Label <- sapply(plotFrame$Usage, function(x) {
-if (x >= 1e+03 & x < 1e+06) {
-  out <- paste(round(x/1e+03, 1), "K", sep = "")
-} else if (x > 1e+06) {
-  out <- paste(round(x/1e+06, 1), "M", sep = "")
-} else {
-  out <- as.character(x)
-}
-return(out)
-  })
-  # - Plot
-  ggplot(plotFrame,
- aes(x = Project, y = Usage, label = Label)) +
-geom_bar(stat = "identity", width = .6, fill = "#4c8cff") +
-xlab('Projects') + ylab('Entity Usage') +
-ylim(0, max(plotFrame$Usage) + .1*max(plotFrame$Usage)) +
-scale_y_continuous(labels = comma) + 
-geom_label(size = 3, vjust = -.1) +
-theme_minimal() +
-theme(axis.text.x = element_text(angle = 90, size = 12, hjust = 1)) +
-theme(axis.title.x = element_text(size = 12)) +
-theme(axis.title.y = element_text(size = 12)) +
-theme(plot.title = element_text(size = 15)) %>%
-withProgress(message = 'Generating plot',
- min = 0,
- max = 1,
- value = 1, {incProgress(amount = 0)})
-})
-# - Download Frame: tabulations_projectsChart
-tabulations_projectsDownload_Frame <- reactive({
-  plotFrame <- wdcmProjectCategory %>%
-filter(Project %in% selectedProjects & Category %in% 
selectedCategories) %>%
-group_by(Project) %>% 
-summarise(Usage = sum(Usage)) %>%
-arrange(desc(Usage))
-  plotFrame
-})
-# - Download: tabulations_projectsChart
-output$tabulations_projectsDownload_Frame <- downloadHandler(
-  filename = function() {
-'WDCM_Data.csv'},
-  content = function(file) {
-write.csv(tabulations_projectsDownload_Frame(),
-  file,
-  quote = FALSE,
-  row.names = FALSE)
-  },
-  contentType =

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: 09/05/2016 commit to Gerrit from current WDCM official

2017-09-24 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/380427 )

Change subject: 09/05/2016 commit to Gerrit from current WDCM official
..

09/05/2016 commit to Gerrit from current WDCM official

Change-Id: Ib5f3c040ea9ff44a4c72d489d63e9e6c58520307
---
M README.md
A WDCM.Rproj
M WDCM_Collect_Items.R
A WDCM_Dahsboard/server.R
A WDCM_Dahsboard/ui.R
A WDCM_OverviewDashboard/server.R
A WDCM_OverviewDashboard/ui.R
M WDCM_Pre-Process.R
A WDCM_Process_v2.R
M WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
A WDC_Aux.R
A _misc/.directory
A _misc/SqoopRuntime.png
A _misc/WDCM_Navigate.html
A _misc/WDCM_OverviewDashboard_Description.html
A _misc/clienttables.tsv
A _misc/databasesTest.tsv
A _misc/hiveQLQuery.hql
A _misc/hiveQLQueryTest.hql
A _misc/projectsTracking.csv
A _misc/wdcmSearchReport_2017-08-01 08:18:50.csv
A _misc/wdcmSqoopReport_2017-07-21 10:31:45.csv
A creds.csv
A currentStats.csv
M server.R
M ui.R
26 files changed, 7,008 insertions(+), 56 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/27/380427/1


-- 
To view, visit https://gerrit.wikimedia.org/r/380427
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib5f3c040ea9ff44a4c72d489d63e9e6c58520307
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: WDCM Usage Dashboard

2017-09-24 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/380428 )

Change subject: WDCM Usage Dashboard
..

WDCM Usage Dashboard

Change-Id: I64588ca4965f855d569733dabcb1eb9178094071
---
A WDCM_UsageDashboard/server.R
A WDCM_UsageDashboard/ui.R
2 files changed, 1,048 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/28/380428/1

diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
new file mode 100644
index 000..1f40e6a
--- /dev/null
+++ b/WDCM_UsageDashboard/server.R
@@ -0,0 +1,621 @@
+### ---
+### --- WDCM Usage Dashboard, v. Beta 0.1
+### --- Script: server.R, v. Beta 0.1
+### ---
+
+### --- Setup
+rm(list = ls())
+### 
+### --- general
+library(shiny)
+library(shinydashboard)
+library(RMySQL)
+library(data.table)
+library(DT)
+library(stringr)
+library(tidyr)
+library(dplyr)
+library(reshape2)
+### --- compute
+library(parallelDist)
+library(smacof)
+### --- visualization
+library(wordcloud)
+library(RColorBrewer)
+library(visNetwork)
+library(rbokeh)
+library(networkD3)
+library(ggplot2)
+library(ggrepel)
+library(scales)
+library(ggvis)
+
+### --- Server (Session) Scope
+### 
+
+### --- Credentials
+# setwd('/home/goransm/WMDE/WDCM/WDCM_RScripts/WDCM_Dashboard/aux')
+setwd('/srv/shiny-server/WDCM_Dashboard/aux')
+
+mySQLCreds <- fread("mySQLCreds.csv", 
+header = T,
+drop = 1)
+
+### -- Connect
+con <- dbConnect(MySQL(), 
+ host = "tools.labsdb", 
+ defult.file = 
"/home/goransm/mySQL_Credentials/replica.my.cnf",
+ dbname = "u16664__wdcm_p",
+ user = mySQLCreds$user,
+ password = mySQLCreds$password)
+
+### --- list existing tables
+q <- "SHOW TABLES;"
+res <- dbSendQuery(con, q)
+st <- fetch(res, -1)
+dbClearResult(res)
+colnames(st) <- "tables"
+
+### --- fetch wdcm2_project
+q <- "SELECT * FROM wdcm2_project;"
+res <- dbSendQuery(con, q)
+wdcmProject <- fetch(res, -1)
+dbClearResult(res)
+colnames(wdcmProject) <- c('Project', 'Usage', 'Project Type')
+
+### --- fetch wdcm2_project_category
+q <- "SELECT * FROM wdcm2_project_category;"
+res <- dbSendQuery(con, q)
+wdcmProjectCategory <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmProjectCategory) <- c('Project', 'Category', 'Usage', 'Project 
Type')
+
+### --- fetch wdcm2_project_item100
+q <- "SELECT * FROM wdcm2_project_item100;"
+res <- dbSendQuery(con, q)
+wdcmProjectItem100 <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmProjectItem100) <- c('Project', 'EntityID', 'Usage', 'Project 
Type', 'Label')
+
+### --- fetch wdcm2_project_category_item100
+q <- "SELECT * FROM wdcm2_project_category_item100;"
+res <- dbSendQuery(con, q)
+wdcmProjectCategoryItem100 <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmProjectCategoryItem100) <- c('Project', 'Category', 'EntityID', 
'Usage', 'Project Type', 'Label')
+
+### --- fetch wdcm2_category
+q <- "SELECT * FROM wdcm2_category;"
+res <- dbSendQuery(con, q)
+wdcmCategory <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmCategory) <- c('Category', 'Usage')
+
+### --- fetch wdcm2_category_item100
+q <- "SELECT * FROM wdcm2_category_item100;"
+res <- dbSendQuery(con, q)
+wdcmCategoryItem100 <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmCategoryItem100) <- c('EntityID', 'Usage', 'Category', 'Label')
+
+### --- Disconnect
+dbDisconnect(con)
+
+### --- Compute per `Project Type` tables
+# - wdcmProjectType
+wdcmProjectType <- wdcmProject %>% 
+  group_by(`Project Type`) %>% 
+  summarise(Usage = sum(Usage)) %>% 
+  arrange(desc(Usage))
+# - wdcmProjectTypeCategory
+wdcmProjectTypeCategory <- wdcmProjectCategory %>% 
+  group_by(`Project Type`, Category) %>% 
+  summarise(Usage = sum(Usage)) %>% 
+  arrange(desc(Usage))
+# - wdcmProjectTypeItem100
+wdcmProjectTypeItem100 <- wdcmProjectItem100 %>% 
+  select(`Project Type`, EntityID, Label, Usage) %>% 
+  group_by(`Project Type`, EntityID, Label) %>% 
+  summarise(Usage = sum(Usage)) %>% 
+  arrange(`Project Type`, desc(Usage))
+
+### --- Compute project similarity structure
+projectSimilarity <- wdcmProjectCategory %>% 
+  dplyr::select(Project, Category, Usage) %>% 
+  tidyr::spread(key = Category,
+ value = Usage,
+ fill = 0)
+projectNames <- projectSimilarity$Project
+projectSimilarity$Project <- NULL
+# - normalize:
+projectSimilarity <- t(apply(projectSimilarity, 1, function(x) {x/sum(x)}))
+# projectSimilarity[projectSimilarity > 0] <- 1
+projectSimilarity <- as.matrix(parDist(as.matrix(projectSimilarity), method = 
"kullback"))
+rownames(projectSimilarity) <- projectNames
+colnames(projectSimilarity) <-

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Sep 26 2017 Add logo + Crosstabs

2017-09-26 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/380770 )

Change subject: Sep 26 2017 Add logo + Crosstabs
..

Sep 26 2017 Add logo + Crosstabs

Change-Id: Ib912ed9dfa5060bbe888714b648be77eb6f69734
---
M WDCM_OverviewDashboard/server.R
M WDCM_OverviewDashboard/ui.R
A WDCM_OverviewDashboard/www/Wikidata-logo-en.png
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
A WDCM_UsageDashboard/www/Wikidata-logo-en.png
6 files changed, 330 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/70/380770/1

diff --git a/WDCM_OverviewDashboard/server.R b/WDCM_OverviewDashboard/server.R
index be744cd..fd1d5fe 100644
--- a/WDCM_OverviewDashboard/server.R
+++ b/WDCM_OverviewDashboard/server.R
@@ -1,4 +1,3 @@
-
 ### ---
 ### --- WDCM Dashboard Module, v. Beta 0.1
 ### --- Script: server.R, v. Beta 0.1
diff --git a/WDCM_OverviewDashboard/ui.R b/WDCM_OverviewDashboard/ui.R
index 7aa6d79..39e1f07 100644
--- a/WDCM_OverviewDashboard/ui.R
+++ b/WDCM_OverviewDashboard/ui.R
@@ -1,4 +1,3 @@
-
 ### ---
 ### --- WDCM Dashboard Module, v. Beta 0.1
 ### --- Script: ui.R, v. Beta 0.1
@@ -28,11 +27,25 @@
 # - fluidRow Title
 fluidRow(
   column(width = 12,
- h2('WDCM Overview Dahsboard'),
- HTML('Wikidata Concepts 
Monitor'),
- hr()
+ h2('WDCM Overview Dashboard'),
+ HTML('Wikidata Concepts 
Monitor')
  )
 ), # - fluidRow Title END
+
+# - fluidRow Logo
+fluidRow(
+  column(width = 12,
+ img(src='Wikidata-logo-en.png', 
+ align = "left")
+  )
+), # - fluidRow END
+
+# - hr()
+fluidRow(
+  column(width = 12,
+ hr()
+  )
+),
 
 fluidRow(
   column(width = 12,
@@ -204,39 +217,61 @@
)
  ), # - tabPanel Overview END
 
-# - tabPanel Usage
+# - tabPanel Description
 tabPanel("Description",
  fluidRow(
-   column(width = 12,
+   column(width = 8,
   HTML('WDCM Overview 
Dashboard
 Description

Introduction

-   This 
Dashboard is a part of the Wikidata Concepts Monitor (WDMC). The WDCM system 
provides analytics on Wikidata usage
+   This 
Dashboard is a part of the Wikidata Concepts Monitor (WDMC). The WDCM 
system provides analytics on Wikidata usage
across the client projects. 
The WDCM Overview Dashboard presents the big picture of Wikidata usage; other 
WDCM dashboards go
-   into more detail.
+   into more detail. The 
Overview Dashboard provides insights into (1) the similarities between 
the client projects in respect to their use of 
+   of Wikidata, as well as 
(2) the volume of Wikidata usage in every client project, (3) 
Wikidata usage tendencies, described by the volume of 
+   Wikidata usage in each of 
the semantic categories of items that are encompassed by the current WDCM 
edition, (4) the similarities between the 
+   Wikidata semantic 
categories of items in respect to their usage across the client projects, 
(5) ranking of client projects in respect to their 
+   Wikidata usage volume, 
(6) the Wikidata usage breakdown across the types of client projects and 
Wikidata semantic categories.

-   Wikidata Item Usage 
Definition
+   Definitions

-   NOTE. The current Wikidata item usage statistic definition is the 
count of the number of pages in a

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Debug Crosstabs ggrepel w. facet_wrap

2017-09-27 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/381058 )

Change subject: Debug Crosstabs ggrepel w. facet_wrap
..

Debug Crosstabs ggrepel w. facet_wrap

Change-Id: I1a26c631900c412e70196533af87be876153e60b
---
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
2 files changed, 21 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/58/381058/1

diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
index 74f8a9f..8cded77 100644
--- a/WDCM_UsageDashboard/server.R
+++ b/WDCM_UsageDashboard/server.R
@@ -18,17 +18,13 @@
 library(reshape2)
 ### --- compute
 library(parallelDist)
-library(smacof)
 ### --- visualization
-library(wordcloud)
 library(RColorBrewer)
 library(visNetwork)
-library(rbokeh)
 library(networkD3)
 library(ggplot2)
 library(ggrepel)
 library(scales)
-library(ggvis)
 
 ### --- Server (Session) Scope
 ### 
@@ -784,8 +780,6 @@
 selProj <- projectOrder$Project[1:25]
 plotFrame <- plotFrame %>% 
   filter(Project %in% selProj)
-plotFrame$Project <- factor(plotFrame$Project, 
-   levels = selProj)
 # - express labels as K, M:
 plotFrame$Label <- sapply(plotFrame$Usage, function(x) {
   if (x >= 1e+03 & x < 1e+06) {
@@ -797,17 +791,19 @@
   }
   return(out)
 })
+plotFrame$Project <- factor(plotFrame$Project,
+levels = selProj)
 # - Plot
 ggplot(plotFrame,
aes(x = Project, y = Usage, label = Label)) +
   geom_line(size = .25, color = "#4c8cff", group = 1) +
   geom_point(size = 1.5, color = "#4c8cff") + 
   geom_point(size = 1, color = "white") + 
-  geom_text_repel(aes(label = plotFrame$Label), 
+  geom_text_repel(data = plotFrame, 
+  aes(x = Project, y = Usage, label = Label), 
   size = 3) +
   facet_wrap(~ Category, ncol = 3, scales = "free_y") +
   xlab('Project') + ylab('Entity Usage') +
-  ylim(0, max(plotFrame$Usage) + .5*max(plotFrame$Usage)) +
   scale_y_continuous(labels = comma) + 
   theme_minimal() +
   theme(axis.text.x = element_text(angle = 90, size = 12, hjust = 1)) +
@@ -868,7 +864,8 @@
   geom_line(size = .25, color = "#4c8cff", group = 1) +
   geom_point(size = 1.5, color = "#4c8cff") + 
   geom_point(size = 1, color = "white") + 
-  geom_text_repel(aes(label = plotFrame$Label), 
+  geom_text_repel(data = plotFrame, 
+  aes(x = `Project Type`, y = Usage, label = Label), 
   size = 3) +
   facet_wrap(~ Category, ncol = 3, scales = "free_y") +
   xlab('Project Type') + ylab('Entity Usage') +
diff --git a/WDCM_UsageDashboard/ui.R b/WDCM_UsageDashboard/ui.R
index af91400..2eea3fb 100644
--- a/WDCM_UsageDashboard/ui.R
+++ b/WDCM_UsageDashboard/ui.R
@@ -134,6 +134,7 @@
  ),
  
fluidRow(

column(width = 12,
+   
   br(), br(),

   htmlOutput('categoryItems_overview_Title'),

   HTML("Note: In the absence of English item label the 
Wikidata item ID 

 is used in place of it."),
@@ -162,6 +163,7 @@
 ),
 fluidRow(
   column(width 
= 12,
+ br(), 
br(),
  
HTML('Wikidata item usage per semantic category in each project type
   
Note: Item usage count is given on a logarithmic 
scale.'),
  br(), 
br(),
@@ -201,6 +203,7 @@

 ),

   fluidRow(

 column(width = 12,
+

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Change dates in Search and Sqoop

2017-09-29 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/381425 )

Change subject: Change dates in Search and Sqoop
..

Change dates in Search and Sqoop

Change-Id: Ie8ea7c9efd3d488deba116d481ab39df0abcf4df
---
M WDCM_Search_Clients.R
M WDCM_Sqoop_Clients.R
2 files changed, 2 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/25/381425/1

diff --git a/WDCM_Search_Clients.R b/WDCM_Search_Clients.R
index 82d1b28..cca103e 100644
--- a/WDCM_Search_Clients.R
+++ b/WDCM_Search_Clients.R
@@ -197,7 +197,7 @@
 # - store report:
 setwd(auxDir)
 write.csv(wdcmSearchReport, 
-  paste("wdcmSearchReport_", Sys.time(), ".csv", sep = ""))
+  paste("wdcmSearchReport_", strsplit(as.character(Sys.time()), split 
= " ")[[1]][1], ".csv", sep = ""))
 
 
 
diff --git a/WDCM_Sqoop_Clients.R b/WDCM_Sqoop_Clients.R
index 1f9e498..a85eda2 100644
--- a/WDCM_Sqoop_Clients.R
+++ b/WDCM_Sqoop_Clients.R
@@ -136,11 +136,6 @@
 }
 
 # - save wdcmSqoopReport
-fileName <- paste("wdcmSqoopReport_", Sys.time(), ".csv", sep = "")
+fileName <- paste("wdcmSqoopReport_", strsplit(as.character(Sys.time()), split 
= " ")[[1]][1], ".csv", sep = "")
 write.csv(wdcmSqoopReport, fileName)
-
-
-
-
-
 

-- 
To view, visit https://gerrit.wikimedia.org/r/381425
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie8ea7c9efd3d488deba116d481ab39df0abcf4df
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: change timestamp in job report filenames

2017-09-29 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/381432 )

Change subject: change timestamp in job report filenames
..

change timestamp in job report filenames

Change-Id: I679ac9832c7afdc489f969c3720368e37785926f
---
R _misc/wdcmSearchReport_2017-08-01.csv
R _misc/wdcmSqoopReport_2017-07-21.csv
2 files changed, 0 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/32/381432/1

diff --git a/_misc/wdcmSearchReport_2017-08-01 08:18:50.csv 
b/_misc/wdcmSearchReport_2017-08-01.csv
similarity index 100%
rename from _misc/wdcmSearchReport_2017-08-01 08:18:50.csv
rename to _misc/wdcmSearchReport_2017-08-01.csv
diff --git a/_misc/wdcmSqoopReport_2017-07-21 10:31:45.csv 
b/_misc/wdcmSqoopReport_2017-07-21.csv
similarity index 100%
rename from _misc/wdcmSqoopReport_2017-07-21 10:31:45.csv
rename to _misc/wdcmSqoopReport_2017-07-21.csv

-- 
To view, visit https://gerrit.wikimedia.org/r/381432
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I679ac9832c7afdc489f969c3720368e37785926f
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Semantics Dashboard

2017-10-23 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/386121 )

Change subject: Semantics Dashboard
..


Semantics Dashboard

Change-Id: I53b8d162a3e729f388992efdb3d5358ab6646565
---
M WDCM_SemanticsDashboard/server.R
M WDCM_SemanticsDashboard/ui.R
A WDCM_ShinyServerFrontPage/SemanticsDashboard.png
M WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
4 files changed, 398 insertions(+), 20 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_SemanticsDashboard/server.R b/WDCM_SemanticsDashboard/server.R
index 1ebb4a8..f07ae5b 100644
--- a/WDCM_SemanticsDashboard/server.R
+++ b/WDCM_SemanticsDashboard/server.R
@@ -57,6 +57,9 @@
 res <- dbSendQuery(con, q)
 dbClearResult(res)
 
+### --- itemTopicTables
+itemTopicTables <- st$tables[which(grepl("wdcm2_itemtopic_", st$tables, fixed 
= T))]
+
 ### --- fetch wdcm2_project
 q <- "SELECT * FROM wdcm2_project;"
 res <- dbSendQuery(con, q)
@@ -96,7 +99,7 @@
 lF <- lF[grepl("wdcm2_projecttopic_", lF, fixed = T)]
 projectTopic <- vector(mode = "list", length = length(lF))
 for (i in 1:length(lF)) {
-  projectTopic[[i]] <- fread(lF[i])
+  projectTopic[[i]] <- fread(lF[i], data.table = F)
 }
 names(projectTopic) <- sapply(lF, function(x) {
   strsplit(strsplit(x, split = ".", fixed = T)[[1]][1],
@@ -157,6 +160,294 @@
 ### --- shinyServer
 shinyServer(function(input, output, session) {
   
+  ### --
+  ### --- TAB: tabPanel Semantic Models
+  ### --
+  
+  ### --- SELECT: update select 'selectCategory'
+  updateSelectizeInput(session,
+   'selectCategory',
+   "Select Semantic Category:",
+   choices = categories,
+   selected = categories[round(runif(1, 1, 
length(categories)))],
+   server = TRUE)
+  
+  ### --- REACTIVE: category specific wdcm_itemtopic data.frame
+  itemTopicsNum <- reactive({
+sC <- gsub(" ", "", input$selectCategory, fixed = T)
+sTable <- itemTopicTables[which(grepl(sC, itemTopicTables, fixed = T))]
+### -- Connect
+con <- dbConnect(MySQL(), 
+ host = "tools.labsdb", 
+ defult.file = 
"/home/goransm/mySQL_Credentials/replica.my.cnf",
+ dbname = "u16664__wdcm_p",
+ user = mySQLCreds$user,
+ password = mySQLCreds$password)
+### --- check the particular table
+q <- paste("DESCRIBE ", sTable, ";", sep = "")
+res <- dbSendQuery(con, q)
+sIT <- fetch(res, -1)
+dbClearResult(res)
+### --- Disconnect
+dbDisconnect(con)
+sum(grepl("topic", sIT$Field))
+  })
+  
+  ### --- SELECT: updateSelectizeInput 'selectCatTopic'
+  output$selectCatTopic <-
+renderUI({
+  if ((is.null(input$selectCategory)) | (length(input$selectCategory) == 
0)) {
+selectInput(inputId = "selectCategoryTopic",
+label = "Select Semantic Topic:",
+choices = NULL,
+selected = NULL)
+  } else {
+cH <- paste("Topic", 1:itemTopicsNum(), sep = " ")
+selectInput(inputId = "selectCategoryTopic",
+label = "Select Semantic Topic:",
+choices = cH,
+selected = cH[1])
+  }
+})
+  
+  ### --- REACTIVE current itemTopic table:
+  itemTopic <- reactive({
+  sC <- gsub(" ", "", input$selectCategory, fixed = T)
+  sTable <- itemTopicTables[which(grepl(sC, itemTopicTables, fixed = T))]
+  cTopic <- tolower(gsub(" ", "", input$selectCategoryTopic))
+  if (!length(cTopic) == 0) {
+### -- Connect
+con <- dbConnect(MySQL(),
+ host = "tools.labsdb",
+ defult.file = 
"/home/goransm/mySQL_Credentials/replica.my.cnf",
+ dbname = "u16664__wdcm_p",
+ user = mySQLCreds$user,
+ password = mySQLCreds$password)
+### --- check the particular table
+q <- 'SET CHARACTER SET utf8;'
+res <- dbSendQuery(con, q)
+q <- paste("SELECT * FROM ", sTable, " ORDER BY ", cTopic, " DESC 
LIMIT 50;", sep = "")
+res <- dbSendQuery(con, q)
+iT <- fetch(res, -1)
+dbClearResult(res)
+### --- Disconnect
+dbDisconnect(con)
+### --- Output:
+return(iT) 
+  } else {return(NULL)}
+  })
+  
+  ### --- OUTPUT output$topItemsTopic
+  output$topItemsTopic <- renderPlot({
+if (!is.null(itemTopic())) {
+  cTopic <- tolower(gsub(" ", "", input$selectCategoryTopic))
+  plotFrame <- itemTopic()
+  plotFrame <- select(plotFrame, 
+  eu_label, eu_entity_id, cTopic)
+  colnames(plotFrame) <- c('Label', 'Id', 'Probability')
+

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Semantics Dashboard

2017-10-23 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/386121 )

Change subject: Semantics Dashboard
..

Semantics Dashboard

Change-Id: I53b8d162a3e729f388992efdb3d5358ab6646565
---
M WDCM_SemanticsDashboard/server.R
M WDCM_SemanticsDashboard/ui.R
A WDCM_ShinyServerFrontPage/SemanticsDashboard.png
M WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
4 files changed, 398 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/21/386121/1

diff --git a/WDCM_SemanticsDashboard/server.R b/WDCM_SemanticsDashboard/server.R
index 1ebb4a8..f07ae5b 100644
--- a/WDCM_SemanticsDashboard/server.R
+++ b/WDCM_SemanticsDashboard/server.R
@@ -57,6 +57,9 @@
 res <- dbSendQuery(con, q)
 dbClearResult(res)
 
+### --- itemTopicTables
+itemTopicTables <- st$tables[which(grepl("wdcm2_itemtopic_", st$tables, fixed 
= T))]
+
 ### --- fetch wdcm2_project
 q <- "SELECT * FROM wdcm2_project;"
 res <- dbSendQuery(con, q)
@@ -96,7 +99,7 @@
 lF <- lF[grepl("wdcm2_projecttopic_", lF, fixed = T)]
 projectTopic <- vector(mode = "list", length = length(lF))
 for (i in 1:length(lF)) {
-  projectTopic[[i]] <- fread(lF[i])
+  projectTopic[[i]] <- fread(lF[i], data.table = F)
 }
 names(projectTopic) <- sapply(lF, function(x) {
   strsplit(strsplit(x, split = ".", fixed = T)[[1]][1],
@@ -157,6 +160,294 @@
 ### --- shinyServer
 shinyServer(function(input, output, session) {
   
+  ### --
+  ### --- TAB: tabPanel Semantic Models
+  ### --
+  
+  ### --- SELECT: update select 'selectCategory'
+  updateSelectizeInput(session,
+   'selectCategory',
+   "Select Semantic Category:",
+   choices = categories,
+   selected = categories[round(runif(1, 1, 
length(categories)))],
+   server = TRUE)
+  
+  ### --- REACTIVE: category specific wdcm_itemtopic data.frame
+  itemTopicsNum <- reactive({
+sC <- gsub(" ", "", input$selectCategory, fixed = T)
+sTable <- itemTopicTables[which(grepl(sC, itemTopicTables, fixed = T))]
+### -- Connect
+con <- dbConnect(MySQL(), 
+ host = "tools.labsdb", 
+ defult.file = 
"/home/goransm/mySQL_Credentials/replica.my.cnf",
+ dbname = "u16664__wdcm_p",
+ user = mySQLCreds$user,
+ password = mySQLCreds$password)
+### --- check the particular table
+q <- paste("DESCRIBE ", sTable, ";", sep = "")
+res <- dbSendQuery(con, q)
+sIT <- fetch(res, -1)
+dbClearResult(res)
+### --- Disconnect
+dbDisconnect(con)
+sum(grepl("topic", sIT$Field))
+  })
+  
+  ### --- SELECT: updateSelectizeInput 'selectCatTopic'
+  output$selectCatTopic <-
+renderUI({
+  if ((is.null(input$selectCategory)) | (length(input$selectCategory) == 
0)) {
+selectInput(inputId = "selectCategoryTopic",
+label = "Select Semantic Topic:",
+choices = NULL,
+selected = NULL)
+  } else {
+cH <- paste("Topic", 1:itemTopicsNum(), sep = " ")
+selectInput(inputId = "selectCategoryTopic",
+label = "Select Semantic Topic:",
+choices = cH,
+selected = cH[1])
+  }
+})
+  
+  ### --- REACTIVE current itemTopic table:
+  itemTopic <- reactive({
+  sC <- gsub(" ", "", input$selectCategory, fixed = T)
+  sTable <- itemTopicTables[which(grepl(sC, itemTopicTables, fixed = T))]
+  cTopic <- tolower(gsub(" ", "", input$selectCategoryTopic))
+  if (!length(cTopic) == 0) {
+### -- Connect
+con <- dbConnect(MySQL(),
+ host = "tools.labsdb",
+ defult.file = 
"/home/goransm/mySQL_Credentials/replica.my.cnf",
+ dbname = "u16664__wdcm_p",
+ user = mySQLCreds$user,
+ password = mySQLCreds$password)
+### --- check the particular table
+q <- 'SET CHARACTER SET utf8;'
+res <- dbSendQuery(con, q)
+q <- paste("SELECT * FROM ", sTable, " ORDER BY ", cTopic, " DESC 
LIMIT 50;", sep = "")
+res <- dbSendQuery(con, q)
+iT <- fetch(res, -1)
+dbClearResult(res)
+### --- Disconnect
+dbDisconnect(con)
+### --- Output:
+return(iT) 
+  } else {return(NULL)}
+  })
+  
+  ### --- OUTPUT output$topItemsTopic
+  output$topItemsTopic <- renderPlot({
+if (!is.null(itemTopic())) {
+  cTopic <- tolower(gsub(" ", "", input$selectCategoryTopic))
+  plotFrame <- itemTopic()
+  plotFrame <- select(plotFrame, 
+  eu_label, eu_entity_id, cTopic)
+  colnames(plotFrame) <- c('Label', 'Id', 'Probability')
+

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: fix Overview Dashboard

2017-10-23 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/386122 )

Change subject: fix Overview Dashboard
..


fix Overview Dashboard

Change-Id: Ia64b3c4548b02c8366adacdddea922aea003dc10
---
M WDCM_OverviewDashboard/server.R
1 file changed, 0 insertions(+), 4 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_OverviewDashboard/server.R b/WDCM_OverviewDashboard/server.R
index 7e05dc3..86149ee 100644
--- a/WDCM_OverviewDashboard/server.R
+++ b/WDCM_OverviewDashboard/server.R
@@ -40,10 +40,6 @@
 header = T,
 drop = 1)
 
-currentStats <- fread("currentStats.csv",
-  header = T,
-  drop = 1)
-
 ### -- Connect
 con <- dbConnect(MySQL(), 
  host = "tools.labsdb", 

-- 
To view, visit https://gerrit.wikimedia.org/r/386122
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia64b3c4548b02c8366adacdddea922aea003dc10
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: fix Overview Dashboard

2017-10-23 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/386122 )

Change subject: fix Overview Dashboard
..

fix Overview Dashboard

Change-Id: Ia64b3c4548b02c8366adacdddea922aea003dc10
---
M WDCM_OverviewDashboard/server.R
1 file changed, 0 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/22/386122/1

diff --git a/WDCM_OverviewDashboard/server.R b/WDCM_OverviewDashboard/server.R
index 7e05dc3..86149ee 100644
--- a/WDCM_OverviewDashboard/server.R
+++ b/WDCM_OverviewDashboard/server.R
@@ -40,10 +40,6 @@
 header = T,
 drop = 1)
 
-currentStats <- fread("currentStats.csv",
-  header = T,
-  drop = 1)
-
 ### -- Connect
 con <- dbConnect(MySQL(), 
  host = "tools.labsdb", 

-- 
To view, visit https://gerrit.wikimedia.org/r/386122
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia64b3c4548b02c8366adacdddea922aea003dc10
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Shiny Server Frontpage

2017-10-22 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385844 )

Change subject: Shiny Server Frontpage
..

Shiny Server Frontpage

Change-Id: I9b0ca3b609ae80c2fd4b452fff3084c1860e7376
---
A WDCM_ShinyServerFrontPage/OverviewDashboard.png
A WDCM_ShinyServerFrontPage/UsageDashboard.png
A WDCM_ShinyServerFrontPage/Wikidata-logo-en.png
A WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
4 files changed, 285 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/44/385844/1

diff --git a/WDCM_ShinyServerFrontPage/OverviewDashboard.png 
b/WDCM_ShinyServerFrontPage/OverviewDashboard.png
new file mode 100644
index 000..5ad8cda
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/OverviewDashboard.png
Binary files differ
diff --git a/WDCM_ShinyServerFrontPage/UsageDashboard.png 
b/WDCM_ShinyServerFrontPage/UsageDashboard.png
new file mode 100644
index 000..bce494b
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/UsageDashboard.png
Binary files differ
diff --git a/WDCM_ShinyServerFrontPage/Wikidata-logo-en.png 
b/WDCM_ShinyServerFrontPage/Wikidata-logo-en.png
new file mode 100644
index 000..5e52bba
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/Wikidata-logo-en.png
Binary files differ
diff --git a/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html 
b/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
new file mode 100644
index 000..d9368c4
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
@@ -0,0 +1,285 @@
+http://www.w3.org/1999/xhtml; lang="en-US">
+
+Wikidata Concepts Monitor (WDCM)
+
+body, html {
+margin: 0;
+padding: 0;
+font-family: Liberation Sans;
+background-color: ##FF;
+color: #00;
+}
+a {
+text-decoration: none;
+}
+a:hover {
+text-decoration: underline;
+}
+#titleBar {
+border-bottom: 1px solid #0d55a2;
+overflow: hidden;
+height: 80px;
+background-color: #0d55a2;
+}
+#titleBar #container {
+margin-top: 14px;
+}
+#titleBar h1 {
+margin: 0 auto 0.5em;
+padding: 0.2em;
+text-align: center;
+color: white;
+font-family: Liberation Sans;
+}
+#intro {
+border: 1px solid #cc;
+margin: 1em 1em 0;
+padding: 0.75em;
+background-color: #B5D7FC;
+text-align: center;
+font-size: 18px;
+}
+#intro p {
+margin: 0.3em 0;
+}
+#outer-content {
+max-width: 910px;
+margin-left: auto;
+margin-right: auto;
+}
+#content {
+margin: 1em auto;
+float: left;
+}
+#main {
+margin-right: 350px;
+float: left;
+line-height: 18px;
+}
+#shiny {
+border-left: 1px solid #aa;
+float: left;
+width: 305px;
+margin-left: -330px;
+padding-left: 20px;
+}
+#shiny iframe {
+margin-top: 30px;
+}
+.caption {
+font-size: 13px;
+}
+code {
+border: 1px solid #aa;
+padding: 0 0.5em;
+background-color: #e5e5e5;
+-moz-border-radius-topleft: 3px;
+-moz-border-radius-topright: 3px;
+-moz-border-radius-bottomright: 3px;
+-moz-border-radius-bottomleft: 3px;
+}
+
+.box{
+width:100%;
+border: 1px solid #cc;
+padding: 5px;
+}
+
+.blue{
+background:#B5D7FC;
+}
+
+
+
+
+
+
+
+
+Wikidata Concepts Monitor
+
+
+
+
+
+How is Wikidata used across the
+Wikimedia sister projects? 
+
+
+
+0. What is this?
+
+Wikidata Concepts Monitor (WDCM)
+is a system of dashboards that monitor the usage of https://www.wikidata.org/wiki/Wikidata:Main_Page; 
target="_blank">Wikidata
+items on WMF sister projects. The dashboards are currently supported by
+(1) analytical overviews of Wikidata item usage that are organized in a
+number of semantic categories, (2) per sister project analytical
+overviews of Wikidata item usage, and (3) https://en.wikipedia.org/wiki/Distributional_semantics;
+target="_blank">distributional semantics
+models of Wikidata usage that offer analytical insights into the
+structure of Wikidata item usage similarity across the sister projects
+and/or semantic categories of Wikidata items.
+
+
+In other
+words, here you can discover how
+much does a particular project use Wikidata, what
+semantic categories of Wikidata items are more popular in a particular
+project or a subset of projects,
+how similar are two or more
+projects in respect to the way they utilize Wikidata, what
+are the most popular Wikidata items in a particular project or a set of
+projects, and similar. 

+
+
+
+1. Getting started
+In order to be able to use the WDCM system in a way it was ment
+and designed to be used, i.e.
+with a clear understanding of what
+is it built for and why it
+was built that way,
+you probably need to get to learn about some important WDCM definitions
+(and the constraints that dictated them) first. You can do that by
+reading through the Definitions section of the WDCM Wikitech Technical
+Documentation [[LINK HERE!!!]].
+Do not panic, please: it is written in a language that a non-technical
+person who does not necessarily care about https://en.wikipedia.org/wiki/Data_science; target="_blank">Data
+Science or https://en.wikipedia.org/wiki/Cognitive_science;

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Shiny Server Frontpage

2017-10-22 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385844 )

Change subject: Shiny Server Frontpage
..


Shiny Server Frontpage

Change-Id: I9b0ca3b609ae80c2fd4b452fff3084c1860e7376
---
A WDCM_ShinyServerFrontPage/OverviewDashboard.png
A WDCM_ShinyServerFrontPage/UsageDashboard.png
A WDCM_ShinyServerFrontPage/Wikidata-logo-en.png
A WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
4 files changed, 285 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_ShinyServerFrontPage/OverviewDashboard.png 
b/WDCM_ShinyServerFrontPage/OverviewDashboard.png
new file mode 100644
index 000..5ad8cda
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/OverviewDashboard.png
Binary files differ
diff --git a/WDCM_ShinyServerFrontPage/UsageDashboard.png 
b/WDCM_ShinyServerFrontPage/UsageDashboard.png
new file mode 100644
index 000..bce494b
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/UsageDashboard.png
Binary files differ
diff --git a/WDCM_ShinyServerFrontPage/Wikidata-logo-en.png 
b/WDCM_ShinyServerFrontPage/Wikidata-logo-en.png
new file mode 100644
index 000..5e52bba
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/Wikidata-logo-en.png
Binary files differ
diff --git a/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html 
b/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
new file mode 100644
index 000..d9368c4
--- /dev/null
+++ b/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
@@ -0,0 +1,285 @@
+http://www.w3.org/1999/xhtml; lang="en-US">
+
+Wikidata Concepts Monitor (WDCM)
+
+body, html {
+margin: 0;
+padding: 0;
+font-family: Liberation Sans;
+background-color: ##FF;
+color: #00;
+}
+a {
+text-decoration: none;
+}
+a:hover {
+text-decoration: underline;
+}
+#titleBar {
+border-bottom: 1px solid #0d55a2;
+overflow: hidden;
+height: 80px;
+background-color: #0d55a2;
+}
+#titleBar #container {
+margin-top: 14px;
+}
+#titleBar h1 {
+margin: 0 auto 0.5em;
+padding: 0.2em;
+text-align: center;
+color: white;
+font-family: Liberation Sans;
+}
+#intro {
+border: 1px solid #cc;
+margin: 1em 1em 0;
+padding: 0.75em;
+background-color: #B5D7FC;
+text-align: center;
+font-size: 18px;
+}
+#intro p {
+margin: 0.3em 0;
+}
+#outer-content {
+max-width: 910px;
+margin-left: auto;
+margin-right: auto;
+}
+#content {
+margin: 1em auto;
+float: left;
+}
+#main {
+margin-right: 350px;
+float: left;
+line-height: 18px;
+}
+#shiny {
+border-left: 1px solid #aa;
+float: left;
+width: 305px;
+margin-left: -330px;
+padding-left: 20px;
+}
+#shiny iframe {
+margin-top: 30px;
+}
+.caption {
+font-size: 13px;
+}
+code {
+border: 1px solid #aa;
+padding: 0 0.5em;
+background-color: #e5e5e5;
+-moz-border-radius-topleft: 3px;
+-moz-border-radius-topright: 3px;
+-moz-border-radius-bottomright: 3px;
+-moz-border-radius-bottomleft: 3px;
+}
+
+.box{
+width:100%;
+border: 1px solid #cc;
+padding: 5px;
+}
+
+.blue{
+background:#B5D7FC;
+}
+
+
+
+
+
+
+
+
+Wikidata Concepts Monitor
+
+
+
+
+
+How is Wikidata used across the
+Wikimedia sister projects? 
+
+
+
+0. What is this?
+
+Wikidata Concepts Monitor (WDCM)
+is a system of dashboards that monitor the usage of https://www.wikidata.org/wiki/Wikidata:Main_Page; 
target="_blank">Wikidata
+items on WMF sister projects. The dashboards are currently supported by
+(1) analytical overviews of Wikidata item usage that are organized in a
+number of semantic categories, (2) per sister project analytical
+overviews of Wikidata item usage, and (3) https://en.wikipedia.org/wiki/Distributional_semantics;
+target="_blank">distributional semantics
+models of Wikidata usage that offer analytical insights into the
+structure of Wikidata item usage similarity across the sister projects
+and/or semantic categories of Wikidata items.
+
+
+In other
+words, here you can discover how
+much does a particular project use Wikidata, what
+semantic categories of Wikidata items are more popular in a particular
+project or a subset of projects,
+how similar are two or more
+projects in respect to the way they utilize Wikidata, what
+are the most popular Wikidata items in a particular project or a set of
+projects, and similar. 

+
+
+
+1. Getting started
+In order to be able to use the WDCM system in a way it was ment
+and designed to be used, i.e.
+with a clear understanding of what
+is it built for and why it
+was built that way,
+you probably need to get to learn about some important WDCM definitions
+(and the constraints that dictated them) first. You can do that by
+reading through the Definitions section of the WDCM Wikitech Technical
+Documentation [[LINK HERE!!!]].
+Do not panic, please: it is written in a language that a non-technical
+person who does not necessarily care about https://en.wikipedia.org/wiki/Data_science; target="_blank">Data
+Science or https://en.wikipedia.org/wiki/Cognitive_science; 
target="_blank">Cognitive

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Non-productionized runs 02 Dec 2017

2017-12-01 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394737 )

Change subject: Non-productionized runs 02 Dec 2017
..

Non-productionized runs 02 Dec 2017

Change-Id: I5cac7822b41c93d5ca60e5bee63e84f053f39758
---
A WDCM_Engine_goransm.R
M WDCM_Sqoop_Clients.R
2 files changed, 885 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/37/394737/1

diff --git a/WDCM_Engine_goransm.R b/WDCM_Engine_goransm.R
new file mode 100644
index 000..d7a2528
--- /dev/null
+++ b/WDCM_Engine_goransm.R
@@ -0,0 +1,874 @@
+#!/usr/bin/env Rscript
+
+### ---
+### --- WDCM Engine, v. Beta 0.1
+### --- Script: WDCM_Engine.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Engine_goransm.R unifies the previous
+### --- three WDCM Engine scripts:
+### --- WDCM_Collect_Items.R
+### --- WDCM_Search_Clients.R
+### --- WDCM_Pre-Process.R
+### --- NOTE: the execution of this WDCM script is always dependent upon the
+### --- previous WDCM_Sqoop_Clients.R run from stat1004 (currently).
+### --- Each section in WDCM_Engine.R provides additional explanation.
+### --- NOTE: WDCM_Engine.R is the only WDCM R script
+### --- that is run from the statboxes (stat1005 currently)
+### --- to produce the WDCM update.
+### ---
+### --- RUN FROM: /home/goransm/RScripts/WDCM_R
+### --- nohup Rscript WDCM_Engine_goransm.R &
+### ---
+
+### ---
+### --- LICENSE:
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### ---
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### ---
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of
+### --- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+### --- GNU General Public License for more details.
+### ---
+### --- You should have received a copy of the GNU General Public License
+### --- along with WDCM. If not, see .
+### ---
+
+### ---
+### --- Script 1: WDCM_Collect_Items.R, WDCM Search Module
+### ---
+
+### ---
+### --- WDCM Search Module, v. Beta 0.1
+### --- Script: WDCM_Collect_Items.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Collect_Items.R takes a list of concepts (categories)
+### --- defined by a given WDCM Ontology (human input) and then
+### --- contacts the Wikidata Query Service to fetch all relevant item IDs.
+### ---
+### --- INPUT:
+### --- the WDCM_Collect_Items.R reads the WDCM Ontology file (csv)
+### --- from /WDCM_Ontology
+### --- on stat1005
+### --- ACTIVE WDCM TAXONOMY: WDCM_Ontology_Berlin_05032017.csv
+### ---
+### --- OUTPUT:
+### --- Results are stored locally as .csv files on stat1005:
+### --- in: /WDCM_CollectedItems
+### ---
+
+# - to nohup.out
+print(paste("--- UPDATE RUN STARTED ON:", Sys.time(), sep = " "))
+
+### --- Setup
+# - contact:
+library(httr)
+library(XML)
+# - wrangling:
+library(stringr)
+library(readr)
+library(data.table)
+library(tidyr)
+# - modeling:
+library(maptpx)
+library(Rtsne)
+library(proxy)
+
+### --- Directories
+ontologyDir <- '/WDCM_Ontology' # - NOTE: starting dir, not '..' 
+logDir <- '../WDCM_Logs'
+itemsDir <- '../WDCM_CollectedItems/'
+dataDir <- '../WDCM_dataOut'
+
+### ---

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Non-productionized runs 02 Dec 2017

2017-12-01 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394737 )

Change subject: Non-productionized runs 02 Dec 2017
..


Non-productionized runs 02 Dec 2017

Change-Id: I5cac7822b41c93d5ca60e5bee63e84f053f39758
---
A WDCM_Engine_goransm.R
M WDCM_Sqoop_Clients.R
2 files changed, 885 insertions(+), 4 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_Engine_goransm.R b/WDCM_Engine_goransm.R
new file mode 100644
index 000..d7a2528
--- /dev/null
+++ b/WDCM_Engine_goransm.R
@@ -0,0 +1,874 @@
+#!/usr/bin/env Rscript
+
+### ---
+### --- WDCM Engine, v. Beta 0.1
+### --- Script: WDCM_Engine.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Engine_goransm.R unifies the previous
+### --- three WDCM Engine scripts:
+### --- WDCM_Collect_Items.R
+### --- WDCM_Search_Clients.R
+### --- WDCM_Pre-Process.R
+### --- NOTE: the execution of this WDCM script is always dependent upon the
+### --- previous WDCM_Sqoop_Clients.R run from stat1004 (currently).
+### --- Each section in WDCM_Engine.R provides additional explanation.
+### --- NOTE: WDCM_Engine.R is the only WDCM R script
+### --- that is run from the statboxes (stat1005 currently)
+### --- to produce the WDCM update.
+### ---
+### --- RUN FROM: /home/goransm/RScripts/WDCM_R
+### --- nohup Rscript WDCM_Engine_goransm.R &
+### ---
+
+### ---
+### --- LICENSE:
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### ---
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### ---
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of
+### --- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+### --- GNU General Public License for more details.
+### ---
+### --- You should have received a copy of the GNU General Public License
+### --- along with WDCM. If not, see .
+### ---
+
+### ---
+### --- Script 1: WDCM_Collect_Items.R, WDCM Search Module
+### ---
+
+### ---
+### --- WDCM Search Module, v. Beta 0.1
+### --- Script: WDCM_Collect_Items.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Collect_Items.R takes a list of concepts (categories)
+### --- defined by a given WDCM Ontology (human input) and then
+### --- contacts the Wikidata Query Service to fetch all relevant item IDs.
+### ---
+### --- INPUT:
+### --- the WDCM_Collect_Items.R reads the WDCM Ontology file (csv)
+### --- from /WDCM_Ontology
+### --- on stat1005
+### --- ACTIVE WDCM TAXONOMY: WDCM_Ontology_Berlin_05032017.csv
+### ---
+### --- OUTPUT:
+### --- Results are stored locally as .csv files on stat1005:
+### --- in: /WDCM_CollectedItems
+### ---
+
+# - to nohup.out
+print(paste("--- UPDATE RUN STARTED ON:", Sys.time(), sep = " "))
+
+### --- Setup
+# - contact:
+library(httr)
+library(XML)
+# - wrangling:
+library(stringr)
+library(readr)
+library(data.table)
+library(tidyr)
+# - modeling:
+library(maptpx)
+library(Rtsne)
+library(proxy)
+
+### --- Directories
+ontologyDir <- '/WDCM_Ontology' # - NOTE: starting dir, not '..' 
+logDir <- '../WDCM_Logs'
+itemsDir <- '../WDCM_CollectedItems/'
+dataDir <- '../WDCM_dataOut'
+
+### --- Set proxy

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: WDCM EngineGeo 17 Dec 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398668 )

Change subject: WDCM EngineGeo 17 Dec 2017
..

WDCM EngineGeo 17 Dec 2017

Change-Id: I6b2239853ea574493b2900729d10eabac5bb08cd
---
M WDCM_EngineGeo_goransm.R
1 file changed, 74 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/68/398668/1

diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 9d9c8df..635606b 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -154,7 +154,10 @@
 rm(item); rm(coordinate); rm(label); rm(rc); gc()
 # - keep unique result set:
 w <- which(duplicated(items$item))
-items <- items[-w, ]
+if (length(w) > 0) {items <- items[-w, ]}
+# - clear possible NAs from coordinates
+w <- which(is.na(items$coordinate) | (items$coordinate == ""))
+if (length(w) > 0) {items <- items[-w, ]}
 # - fix items
 items$item <- gsub("http://www.wikidata.org/entity/;, "", items$item, 
fixed = T)
 # - fix coordinates (lon, lat)
@@ -283,7 +286,64 @@
 }
 
 ### --- join coordinates, items, labels, and usage
+setwd(dataDir)
 
+# - list .tsv files
+lF <- list.files()
+w <- which(grepl("^wdcm_geoitem", lF))
+lF <- lF[w]
+w <- which(grepl(".tsv", lF, fixed = T))
+lF <- lF[w]
+
+# - remove old .csv files:
+rmF <- list.files()
+w <- which(grepl("^wdcm_geoitem", rmF))
+rmF <- rmF[w]
+w <- which(grepl(".csv", rmF, fixed = T))
+rmF <- rmF[w]
+file.remove(rmF)
+
+for (i in 1:length(lF)) {
+  
+  # - to runtime Log:
+  print(paste("Joining dataset: ", lF[i], sep = ""))
+  
+  # - usage data
+  usage <- readLines(lF[i])
+  usage <- usage[-which(grepl("null", usage, fixed = T))]
+  usage <- usage[-which(usage == "")]
+  usageList <- lapply(usage, function(x) {
+udata <- strsplit(x, split = "\t", fixed = T)[[1]]
+udata <- data.frame(item = udata[1], 
+usage = udata[2],
+stringsAsFactors = F)
+  })
+  usage <- rbindlist(usageList)
+  rm(usageList)
+  # - coordinates data and labels
+  setwd(itemsDir)
+  rF <- list.files()
+  # - find coordinates data and labels file
+  catUsage <- strsplit(
+strsplit(lF[i], split = ".", fixed = T)[[1]][1],
+split = "_",
+fixed = T)[[1]][3]
+  rFcheck <- sapply(rF, function(x) {
+gsub(" ", "", strsplit(x, split = "_", fixed = T)[[1]][1], fixed = T)
+  })
+  w <- sapply(rFcheck, function(x) {grepl(catUsage, x)})
+  geo <- read.csv(rF[w], 
+  header = T,
+  check.names = F,
+  stringsAsFactors = F)
+  # - join
+  usage <- left_join(usage, geo, by = 'item')
+  rm(geo)
+  # - write: back to dataDir
+  setwd(dataDir)
+  file.remove(lF[i])
+  write.csv(usage, file = gsub(".tsv", ".csv", lF[i], fixed = T))
+}
 
 ### --- log ETL step:
 # - to runtime Log:
@@ -314,13 +374,20 @@
 ### --- Step 3: toLabsGeoReport
 ### ---
 
+### --- toLabsGeoReport
+toLabsGeoReport <- data.frame(timeStamp = as.character(Sys.time()),
+  statbox = "stat1005",
+  sqoopbox = "stat1004",
+  stringsAsFactors = F)
+write.csv(toLabsGeoReport, "toLabsGeoReport.csv")
 
+# - to runtime Log:
+print(paste("--- UPDATE RUN COMPLETED ON:", Sys.time(), sep = " "))
 
+### --- copy reports to /srv/published-datasets/wdcm:
 
-
-
-
-
-
-
+# - WDCM_MainReport
+system(command = 'cp 
/home/goransm/RScripts/WDCM_R/WDCM_Logs/WDCM_GeoReport.csv 
/srv/published-datasets/wdcm/', wait = T)
+# - toLabsReport
+system(command = 'cp 
/home/goransm/RScripts/WDCM_R/WDCM_Logs/toLabsGeoReport.csv 
/srv/published-datasets/wdcm/', wait = T)
 

-- 
To view, visit https://gerrit.wikimedia.org/r/398668
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6b2239853ea574493b2900729d10eabac5bb08cd
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Labs processing 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398692 )

Change subject: Labs processing 17 Dec 2017
..


Labs processing 17 Dec 2017

Change-Id: I1cfad83ed71be0e046205a5cff09987df0a8ab7f
---
M WDCM_EngineGeo_goransm.R
A WDCM_Process.R
A WDCM_Update_Labs.R
3 files changed, 1,159 insertions(+), 2 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 635606b..4a04518 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -288,14 +288,14 @@
 ### --- join coordinates, items, labels, and usage
 setwd(dataDir)
 
-# - list .tsv files
+# - list .tsv files from dataDir
 lF <- list.files()
 w <- which(grepl("^wdcm_geoitem", lF))
 lF <- lF[w]
 w <- which(grepl(".tsv", lF, fixed = T))
 lF <- lF[w]
 
-# - remove old .csv files:
+# - remove old .csv files from dataDir
 rmF <- list.files()
 w <- which(grepl("^wdcm_geoitem", rmF))
 rmF <- rmF[w]
diff --git a/WDCM_Process.R b/WDCM_Process.R
new file mode 100644
index 000..467892d
--- /dev/null
+++ b/WDCM_Process.R
@@ -0,0 +1,910 @@
+
+### ---
+### --- WDCM Process Module, v. Beta 0.1
+### --- Script: WDCM_Process_v2.R, v. Beta 0.1
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Process_v2.R takes a list of .tsv files that present
+### --- the data from wbc_entity_usage tables accross the client projects
+### --- fetched from production (stat1005) by WDCM_Search_Clients.R and 
+### --- further pre-processed by WDCM_Pre-Process.R (also on production).
+### --- The goal of this WDCM module/script is to produce (or update) 
+### --- the WDCM Stats Dashboard database.
+### ---
+### --- INPUT: 
+### --- the WDCM_Process_v2.R reads the .tsv input files from:
+### --- /home/goransm/WMDE/WDCM/WDCM_DataIN/WDCM_DataIN_ClientUsage_v2/
+### --- on the wikidataconcepts.eqiad.wmflabs Cloud VPS instance
+### --- These files are brought to Labs directly from productio
+### --- (currently the stat1005.eqiad.wmnet statbox)
+### ---
+### --- OUTPUT: the WDCM Dashboards MariaDB database is update
+### ---
+
+### ---
+### --- LICENSE:
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### ---
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### ---
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of
+### --- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+### --- GNU General Public License for more details.
+### ---
+### --- You should have received a copy of the GNU General Public License
+### --- along with WDCM. If not, see .
+### ---
+
+### --- Setup
+library(RMySQL)
+library(httr)
+library(XML)
+library(data.table)
+library(dplyr)
+library(tidyr)
+library(readr)
+library(htmltab)
+library(snowfall)
+library(maptpx)
+library(Rtsne)
+
+# - mysql --defaults-file=/home/goransm/mySQL_Credentials/replica.my.cnf -h 
tools.labsdb u16664__wdcm_p
+# - database: u16664__wdcm_p
+
+### --- functions
+
+# - projectType() to determine project type
+projectType <- function(projectName) {
+  unname(sapply(projectName, function(x) {
+if (grepl("commons", x, fixed = T)) {"Commons"
+} else if (grepl("mediawiki|meta|species|wikidata", x)) {"Other"
+} else if (grepl("wiki$", x)) {"Wikipedia"
+} else if (grepl("quote$", x)) {"Wikiquote"
+} else if (grepl("voyage$", x)) {"Wikivoyage"
+} else if (grepl("news$", x)) {"Wikinews"
+} else if (grepl("source$", x)) {"Wikisource"
+} else if (grepl("wiktionary$", x)) {"Wiktionary"
+} else if (grepl("versity$", x)) {"Wikiversity"
+} else if (grepl("books$", x)) {"Wikibooks"
+} else {"Other"}
+  }))
+}
+
+### ---
+### --- NOTE:
+### --- TABLE NAMING CONVENTION FOR v2 (WDCM Stats Dashboard)
+### --- wdcm2_something
+### ---
+
+# - to nohup.out
+print(paste("WDCM Process.R update started at: ", Sys.time(), sep

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Labs processing 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398692 )

Change subject: Labs processing 17 Dec 2017
..

Labs processing 17 Dec 2017

Change-Id: I1cfad83ed71be0e046205a5cff09987df0a8ab7f
---
M WDCM_EngineGeo_goransm.R
A WDCM_Process.R
A WDCM_Update_Labs.R
3 files changed, 1,159 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/92/398692/1

diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 635606b..4a04518 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -288,14 +288,14 @@
 ### --- join coordinates, items, labels, and usage
 setwd(dataDir)
 
-# - list .tsv files
+# - list .tsv files from dataDir
 lF <- list.files()
 w <- which(grepl("^wdcm_geoitem", lF))
 lF <- lF[w]
 w <- which(grepl(".tsv", lF, fixed = T))
 lF <- lF[w]
 
-# - remove old .csv files:
+# - remove old .csv files from dataDir
 rmF <- list.files()
 w <- which(grepl("^wdcm_geoitem", rmF))
 rmF <- rmF[w]
diff --git a/WDCM_Process.R b/WDCM_Process.R
new file mode 100644
index 000..467892d
--- /dev/null
+++ b/WDCM_Process.R
@@ -0,0 +1,910 @@
+
+### ---
+### --- WDCM Process Module, v. Beta 0.1
+### --- Script: WDCM_Process_v2.R, v. Beta 0.1
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Process_v2.R takes a list of .tsv files that present
+### --- the data from wbc_entity_usage tables accross the client projects
+### --- fetched from production (stat1005) by WDCM_Search_Clients.R and 
+### --- further pre-processed by WDCM_Pre-Process.R (also on production).
+### --- The goal of this WDCM module/script is to produce (or update) 
+### --- the WDCM Stats Dashboard database.
+### ---
+### --- INPUT: 
+### --- the WDCM_Process_v2.R reads the .tsv input files from:
+### --- /home/goransm/WMDE/WDCM/WDCM_DataIN/WDCM_DataIN_ClientUsage_v2/
+### --- on the wikidataconcepts.eqiad.wmflabs Cloud VPS instance
+### --- These files are brought to Labs directly from productio
+### --- (currently the stat1005.eqiad.wmnet statbox)
+### ---
+### --- OUTPUT: the WDCM Dashboards MariaDB database is update
+### ---
+
+### ---
+### --- LICENSE:
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### ---
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### ---
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of
+### --- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+### --- GNU General Public License for more details.
+### ---
+### --- You should have received a copy of the GNU General Public License
+### --- along with WDCM. If not, see .
+### ---
+
+### --- Setup
+library(RMySQL)
+library(httr)
+library(XML)
+library(data.table)
+library(dplyr)
+library(tidyr)
+library(readr)
+library(htmltab)
+library(snowfall)
+library(maptpx)
+library(Rtsne)
+
+# - mysql --defaults-file=/home/goransm/mySQL_Credentials/replica.my.cnf -h 
tools.labsdb u16664__wdcm_p
+# - database: u16664__wdcm_p
+
+### --- functions
+
+# - projectType() to determine project type
+projectType <- function(projectName) {
+  unname(sapply(projectName, function(x) {
+if (grepl("commons", x, fixed = T)) {"Commons"
+} else if (grepl("mediawiki|meta|species|wikidata", x)) {"Other"
+} else if (grepl("wiki$", x)) {"Wikipedia"
+} else if (grepl("quote$", x)) {"Wikiquote"
+} else if (grepl("voyage$", x)) {"Wikivoyage"
+} else if (grepl("news$", x)) {"Wikinews"
+} else if (grepl("source$", x)) {"Wikisource"
+} else if (grepl("wiktionary$", x)) {"Wiktionary"
+} else if (grepl("versity$", x)) {"Wikiversity"
+} else if (grepl("books$", x)) {"Wikibooks"
+} else {"Other"}
+  }))
+}
+
+### ---
+### --- NOTE:
+### --- TABLE NAMING CONVENTION FOR v2 (WDCM Stats Dashboard)
+### --- wdcm2_something
+### ---
+
+# - to nohup.out
+print(paste("WDCM Process.R update started at: ",

[MediaWiki-commits] [Gerrit] analytics...WDCM-Overview-Dashboard[master]: Overview Dashboard 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398693 )

Change subject: Overview Dashboard 17 Dec 2017
..

Overview Dashboard 17 Dec 2017

Change-Id: I4fc52a454f172c578e650a1ea8ce56face3b7c3d
---
M server.R
M ui.R
2 files changed, 37 insertions(+), 11 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Overview-Dashboard 
refs/changes/93/398693/1

diff --git a/server.R b/server.R
index 86149ee..a2ddf2d 100644
--- a/server.R
+++ b/server.R
@@ -129,9 +129,27 @@
 ### --- Disconnect
 dbDisconnect(con)
 
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_OverviewDashboard/update/')
+update <- read.csv('toLabsReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
 ### --- shinyServer
 shinyServer(function(input, output, session) {
   
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
+  
   ### --
   ### --- TAB: Overview
   ### --
diff --git a/ui.R b/ui.R
index 33bbfc8..fa722ef 100644
--- a/ui.R
+++ b/ui.R
@@ -26,19 +26,27 @@
 
 # - fluidRow Title
 fluidRow(
-  column(width = 12,
- h2('WDCM Overview Dashboard'),
- HTML('Wikidata Concepts 
Monitor')
+  column(width = 5,
+ fluidRow(
+   column(width = 3,
+  img(src = 'Wikidata-logo-en.png',
+  align = "left")
+   ),
+   column(width = 1),
+   column(width = 8,
+  h1('WDCM Overview Dashboard'),
+  HTML('Wikidata Concepts 
Monitor')
+   )
  )
-), # - fluidRow Title END
-
-# - fluidRow Logo
-fluidRow(
-  column(width = 12,
- img(src='Wikidata-logo-en.png', 
- align = "left")
+  ),
+  column(width = 7,
+ br(),
+ HTML('Interactive visualizations of 
Wikidata use by other Wikimedia projects.'),
+ HTML('https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor; 
target="_blank">Visit the WDCM wiki page'),
+ HTML('Did you spot a bug, a missing 
label, or wrong data? https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor/UserFeedback; 
target="_blank">Give us feedback here'),
+ htmlOutput('updateInfo')
   )
-), # - fluidRow END
+), # - fluidRow Title END
 
 # - hr()
 fluidRow(

-- 
To view, visit https://gerrit.wikimedia.org/r/398693
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4fc52a454f172c578e650a1ea8ce56face3b7c3d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Overview-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Usage-Dashboard[master]: Usage Dashboard Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398695 )

Change subject: Usage Dashboard Dec 2017
..

Usage Dashboard Dec 2017

Change-Id: I229d45fc96e581da487effce6497720b319b0234
---
M server.R
M ui.R
2 files changed, 37 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Usage-Dashboard 
refs/changes/95/398695/1

diff --git a/server.R b/server.R
index 5bdfdd6..a57f449 100644
--- a/server.R
+++ b/server.R
@@ -159,9 +159,27 @@
 })
 names(unzip_projectTypes) <- search_projectTypes
 
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_UsageDashboard/update/')
+update <- read.csv('toLabsReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
 ### --- shinyServer
 shinyServer(function(input, output, session) {
   
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
+  
   ### --
   ### --- BASIC FACTS
   ### --
diff --git a/ui.R b/ui.R
index 8a8221b..0caf258 100644
--- a/ui.R
+++ b/ui.R
@@ -25,20 +25,27 @@
 
 # - fluidRow Title
 fluidRow(
-  column(width = 12,
- h2('WDCM Usage Dashboard'),
- HTML('Wikidata Concepts 
Monitor')
- 
+  column(width = 5,
+ fluidRow(
+   column(width = 3,
+  img(src = 'Wikidata-logo-en.png',
+  align = "left")
+   ),
+   column(width = 1),
+   column(width = 8,
+  h1('WDCM Usage Dashboard'),
+  HTML('Wikidata Concepts 
Monitor')
+   )
  )
+  ),
+  column(width = 7,
+ br(),
+ HTML('Interactive visualizations of 
Wikidata use by other Wikimedia projects.'),
+ HTML('https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor; 
target="_blank">Visit the WDCM wiki page'),
+ HTML('Did you spot a bug, a missing 
label, or wrong data? https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor/UserFeedback; 
target="_blank">Give us feedback here'),
+ htmlOutput('updateInfo')
+  )
 ), # - fluidRow Title END
-
-# - fluidRow Logo
-fluidRow(
-  column(width = 12,
- img(src='Wikidata-logo-en.png', 
- align = "left")
- )
-), # - fluidRow END
 
 # - hr()
 fluidRow(

-- 
To view, visit https://gerrit.wikimedia.org/r/398695
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I229d45fc96e581da487effce6497720b319b0234
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Usage-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Semantics-Dashboard[master]: Semantics Dashboard 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398694 )

Change subject: Semantics Dashboard 17 Dec 2017
..

Semantics Dashboard 17 Dec 2017

Change-Id: I294b0bbf46dc333c8aa48158f8acf47bb7b46718
---
M server.R
M ui.R
2 files changed, 52 insertions(+), 22 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Semantics-Dashboard 
refs/changes/94/398694/1

diff --git a/server.R b/server.R
index 6954581..fcc3215 100644
--- a/server.R
+++ b/server.R
@@ -4,7 +4,7 @@
 ### ---
 
 ### --- Setup
-rm(list = ls())
+
 ### 
 ### --- general
 library(shiny)
@@ -92,7 +92,7 @@
 dbDisconnect(con)
 
 ### --- Fetch local files
-setwd('/home/goransm/WMDE/WDCM/WDCM_SemanticsDashboard/data/')
+setwd('/srv/shiny-server/WDCM_SemanticsDashboard/data/')
 
 ### --- fetch projecttopic tables
 lF <- list.files()
@@ -133,6 +133,14 @@
fixed = T)[[1]][4]
 })
 
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_SemanticsDashboard/update/')
+update <- read.csv('toLabsReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
 ### - Determine Constants
 # - determine Projects
 projects <- wdcmProject$Project
@@ -159,6 +167,16 @@
 
 ### --- shinyServer
 shinyServer(function(input, output, session) {
+  
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
   
   ### --
   ### --- TAB: tabPanel Semantic Models
@@ -215,7 +233,7 @@
   sC <- gsub(" ", "", input$selectCategory, fixed = T)
   sTable <- itemTopicTables[which(grepl(sC, itemTopicTables, fixed = T))]
   cTopic <- tolower(gsub(" ", "", input$selectCategoryTopic))
-  if (!length(cTopic) == 0) {
+  if (!(length(cTopic) == 0)) {
 ### -- Connect
 con <- dbConnect(MySQL(),
  host = "tools.labsdb",
@@ -274,8 +292,8 @@
 
 if (!is.null(itemTopic())) {
   # - normalization: Luce's choice axiom
-  itemNames <- itemTopic()$eu_label
-  root <- select(itemTopic(), starts_with('topic'))
+  itemNames <- itemTopic()$eu_entity_id
+  root <- dplyr::select(itemTopic(), starts_with('topic'))
   root <- as.matrix(parDist(as.matrix(root), method = "euclidean"))
   rownames(root) <- itemNames
   colnames(root) <- itemNames
@@ -299,6 +317,9 @@
 nodes$id[which(nodes$label %in% x)]
   })
   conceptsStruct$arrows <- rep("to", length(conceptsStruct$to))
+  nodes$label <- sapply(nodes$label, function(x) {
+itemTopic()$eu_label[itemTopic()$eu_entity_id == x]
+  })
   visNetwork(nodes = nodes,
  edges = conceptsStruct,
  width = "100%",
@@ -376,7 +397,7 @@
 if (!is.null(input$selectProject)) {
   wUnzip <- which(names(unzip_projectTypes) %in% input$selectProject)
   if (length(wUnzip > 0)) {
-selectedProjects <- unname(do.call(c, unzip_projectTypes[wUnzip]))
+selectedProjects <- unname(do.call('c', unzip_projectTypes[wUnzip]))
   }
   wSel <- which(projects %in% input$selectProject)
   if (length(wSel > 0)) {
@@ -407,7 +428,6 @@
starts_with('topic'))
   catName <- gsub("([[:lower:]])([[:upper:]])", "\\1 \\2", 
names(projectTopic)[cCategory])
   # - FIX THIS:
-  catName <- gsub("Workof Art", "Work of Art", catName, fixed = T) 
   cProj$Category <- catName
   cProj <- cProj %>% 
 select(Topic, Probability, Category) %>% 
@@ -423,7 +443,10 @@
   projList <- as.data.frame(rbindlist(projList[wEl]))
   # - factor projList$Topic:
   projList$Topic <- str_to_title(gsub("([[:alpha:]]+)", "\\1 ", 
projList$Topic))
-  projList$Topic <- factor(projList$Topic, levels = unique(projList$Topic))
+  topicLevels <- unique(projList$Topic)
+  topicLevelsOrd <- as.numeric(str_extract(topicLevels, "[[:digit:]]+"))
+  topicLevels <- topicLevels[order(topicLevelsOrd)]
+  projList$Topic <- factor(projList$Topic, levels = topicLevels)
   # - visualize w. ggplot2
   ggplot(projList,
  aes(x = Topic, 
diff --git a/ui.R b/ui.R
index 6ddb88d..0f75409 100644
--- a/ui.R
+++ b/ui.R
@@ -25,20 +25,27 @@
 
 # - fluidRow Title
 fluidRow(
-  column(width = 12,
- h2('WDCM Semantics Dashboard'),
-

[MediaWiki-commits] [Gerrit] analytics...WDCM-Usage-Dashboard[master]: Usage Dashboard Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398695 )

Change subject: Usage Dashboard Dec 2017
..


Usage Dashboard Dec 2017

Change-Id: I229d45fc96e581da487effce6497720b319b0234
---
M server.R
M ui.R
2 files changed, 37 insertions(+), 12 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/server.R b/server.R
index 5bdfdd6..a57f449 100644
--- a/server.R
+++ b/server.R
@@ -159,9 +159,27 @@
 })
 names(unzip_projectTypes) <- search_projectTypes
 
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_UsageDashboard/update/')
+update <- read.csv('toLabsReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
 ### --- shinyServer
 shinyServer(function(input, output, session) {
   
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
+  
   ### --
   ### --- BASIC FACTS
   ### --
diff --git a/ui.R b/ui.R
index 8a8221b..0caf258 100644
--- a/ui.R
+++ b/ui.R
@@ -25,20 +25,27 @@
 
 # - fluidRow Title
 fluidRow(
-  column(width = 12,
- h2('WDCM Usage Dashboard'),
- HTML('Wikidata Concepts 
Monitor')
- 
+  column(width = 5,
+ fluidRow(
+   column(width = 3,
+  img(src = 'Wikidata-logo-en.png',
+  align = "left")
+   ),
+   column(width = 1),
+   column(width = 8,
+  h1('WDCM Usage Dashboard'),
+  HTML('Wikidata Concepts 
Monitor')
+   )
  )
+  ),
+  column(width = 7,
+ br(),
+ HTML('Interactive visualizations of 
Wikidata use by other Wikimedia projects.'),
+ HTML('https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor; 
target="_blank">Visit the WDCM wiki page'),
+ HTML('Did you spot a bug, a missing 
label, or wrong data? https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor/UserFeedback; 
target="_blank">Give us feedback here'),
+ htmlOutput('updateInfo')
+  )
 ), # - fluidRow Title END
-
-# - fluidRow Logo
-fluidRow(
-  column(width = 12,
- img(src='Wikidata-logo-en.png', 
- align = "left")
- )
-), # - fluidRow END
 
 # - hr()
 fluidRow(

-- 
To view, visit https://gerrit.wikimedia.org/r/398695
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I229d45fc96e581da487effce6497720b319b0234
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Usage-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Semantics-Dashboard[master]: Semantics Dashboard 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398694 )

Change subject: Semantics Dashboard 17 Dec 2017
..


Semantics Dashboard 17 Dec 2017

Change-Id: I294b0bbf46dc333c8aa48158f8acf47bb7b46718
---
M server.R
M ui.R
2 files changed, 52 insertions(+), 22 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/server.R b/server.R
index 6954581..fcc3215 100644
--- a/server.R
+++ b/server.R
@@ -4,7 +4,7 @@
 ### ---
 
 ### --- Setup
-rm(list = ls())
+
 ### 
 ### --- general
 library(shiny)
@@ -92,7 +92,7 @@
 dbDisconnect(con)
 
 ### --- Fetch local files
-setwd('/home/goransm/WMDE/WDCM/WDCM_SemanticsDashboard/data/')
+setwd('/srv/shiny-server/WDCM_SemanticsDashboard/data/')
 
 ### --- fetch projecttopic tables
 lF <- list.files()
@@ -133,6 +133,14 @@
fixed = T)[[1]][4]
 })
 
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_SemanticsDashboard/update/')
+update <- read.csv('toLabsReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
 ### - Determine Constants
 # - determine Projects
 projects <- wdcmProject$Project
@@ -159,6 +167,16 @@
 
 ### --- shinyServer
 shinyServer(function(input, output, session) {
+  
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
   
   ### --
   ### --- TAB: tabPanel Semantic Models
@@ -215,7 +233,7 @@
   sC <- gsub(" ", "", input$selectCategory, fixed = T)
   sTable <- itemTopicTables[which(grepl(sC, itemTopicTables, fixed = T))]
   cTopic <- tolower(gsub(" ", "", input$selectCategoryTopic))
-  if (!length(cTopic) == 0) {
+  if (!(length(cTopic) == 0)) {
 ### -- Connect
 con <- dbConnect(MySQL(),
  host = "tools.labsdb",
@@ -274,8 +292,8 @@
 
 if (!is.null(itemTopic())) {
   # - normalization: Luce's choice axiom
-  itemNames <- itemTopic()$eu_label
-  root <- select(itemTopic(), starts_with('topic'))
+  itemNames <- itemTopic()$eu_entity_id
+  root <- dplyr::select(itemTopic(), starts_with('topic'))
   root <- as.matrix(parDist(as.matrix(root), method = "euclidean"))
   rownames(root) <- itemNames
   colnames(root) <- itemNames
@@ -299,6 +317,9 @@
 nodes$id[which(nodes$label %in% x)]
   })
   conceptsStruct$arrows <- rep("to", length(conceptsStruct$to))
+  nodes$label <- sapply(nodes$label, function(x) {
+itemTopic()$eu_label[itemTopic()$eu_entity_id == x]
+  })
   visNetwork(nodes = nodes,
  edges = conceptsStruct,
  width = "100%",
@@ -376,7 +397,7 @@
 if (!is.null(input$selectProject)) {
   wUnzip <- which(names(unzip_projectTypes) %in% input$selectProject)
   if (length(wUnzip > 0)) {
-selectedProjects <- unname(do.call(c, unzip_projectTypes[wUnzip]))
+selectedProjects <- unname(do.call('c', unzip_projectTypes[wUnzip]))
   }
   wSel <- which(projects %in% input$selectProject)
   if (length(wSel > 0)) {
@@ -407,7 +428,6 @@
starts_with('topic'))
   catName <- gsub("([[:lower:]])([[:upper:]])", "\\1 \\2", 
names(projectTopic)[cCategory])
   # - FIX THIS:
-  catName <- gsub("Workof Art", "Work of Art", catName, fixed = T) 
   cProj$Category <- catName
   cProj <- cProj %>% 
 select(Topic, Probability, Category) %>% 
@@ -423,7 +443,10 @@
   projList <- as.data.frame(rbindlist(projList[wEl]))
   # - factor projList$Topic:
   projList$Topic <- str_to_title(gsub("([[:alpha:]]+)", "\\1 ", 
projList$Topic))
-  projList$Topic <- factor(projList$Topic, levels = unique(projList$Topic))
+  topicLevels <- unique(projList$Topic)
+  topicLevelsOrd <- as.numeric(str_extract(topicLevels, "[[:digit:]]+"))
+  topicLevels <- topicLevels[order(topicLevelsOrd)]
+  projList$Topic <- factor(projList$Topic, levels = topicLevels)
   # - visualize w. ggplot2
   ggplot(projList,
  aes(x = Topic, 
diff --git a/ui.R b/ui.R
index 6ddb88d..0f75409 100644
--- a/ui.R
+++ b/ui.R
@@ -25,20 +25,27 @@
 
 # - fluidRow Title
 fluidRow(
-  column(width = 12,
- h2('WDCM Semantics Dashboard'),
- HTML('Wikidata

[MediaWiki-commits] [Gerrit] analytics...WDCM-Overview-Dashboard[master]: Overview Dashboard 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398693 )

Change subject: Overview Dashboard 17 Dec 2017
..


Overview Dashboard 17 Dec 2017

Change-Id: I4fc52a454f172c578e650a1ea8ce56face3b7c3d
---
M server.R
M ui.R
2 files changed, 37 insertions(+), 11 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/server.R b/server.R
index 86149ee..a2ddf2d 100644
--- a/server.R
+++ b/server.R
@@ -129,9 +129,27 @@
 ### --- Disconnect
 dbDisconnect(con)
 
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_OverviewDashboard/update/')
+update <- read.csv('toLabsReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
 ### --- shinyServer
 shinyServer(function(input, output, session) {
   
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
+  
   ### --
   ### --- TAB: Overview
   ### --
diff --git a/ui.R b/ui.R
index 33bbfc8..fa722ef 100644
--- a/ui.R
+++ b/ui.R
@@ -26,19 +26,27 @@
 
 # - fluidRow Title
 fluidRow(
-  column(width = 12,
- h2('WDCM Overview Dashboard'),
- HTML('Wikidata Concepts 
Monitor')
+  column(width = 5,
+ fluidRow(
+   column(width = 3,
+  img(src = 'Wikidata-logo-en.png',
+  align = "left")
+   ),
+   column(width = 1),
+   column(width = 8,
+  h1('WDCM Overview Dashboard'),
+  HTML('Wikidata Concepts 
Monitor')
+   )
  )
-), # - fluidRow Title END
-
-# - fluidRow Logo
-fluidRow(
-  column(width = 12,
- img(src='Wikidata-logo-en.png', 
- align = "left")
+  ),
+  column(width = 7,
+ br(),
+ HTML('Interactive visualizations of 
Wikidata use by other Wikimedia projects.'),
+ HTML('https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor; 
target="_blank">Visit the WDCM wiki page'),
+ HTML('Did you spot a bug, a missing 
label, or wrong data? https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor/UserFeedback; 
target="_blank">Give us feedback here'),
+ htmlOutput('updateInfo')
   )
-), # - fluidRow END
+), # - fluidRow Title END
 
 # - hr()
 fluidRow(

-- 
To view, visit https://gerrit.wikimedia.org/r/398693
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4fc52a454f172c578e650a1ea8ce56face3b7c3d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Overview-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Semantics-Dashboard[master]: Semantics ignore 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398696 )

Change subject: Semantics ignore 17 Dec 2017
..

Semantics ignore 17 Dec 2017

Change-Id: Ie38b02a6f9831fefe8c3b655980f5702b1b30f9d
---
A .gitignore
M data/wdcm2_projecttopic_ArchitecturalStructure.csv
M data/wdcm2_projecttopic_AstronomicalObject.csv
M data/wdcm2_projecttopic_Book.csv
M data/wdcm2_projecttopic_ChemicalEntities.csv
M data/wdcm2_projecttopic_Event.csv
M data/wdcm2_projecttopic_Gene.csv
M data/wdcm2_projecttopic_GeographicalObject.csv
M data/wdcm2_projecttopic_Human.csv
M data/wdcm2_projecttopic_Organization.csv
M data/wdcm2_projecttopic_ScientificArticle.csv
M data/wdcm2_projecttopic_Taxon.csv
M data/wdcm2_projecttopic_Thoroughfare.csv
M data/wdcm2_projecttopic_Wikimedia.csv
M data/wdcm2_visNetworkEdges_project_ArchitecturalStructure.csv
M data/wdcm2_visNetworkEdges_project_AstronomicalObject.csv
M data/wdcm2_visNetworkEdges_project_Book.csv
M data/wdcm2_visNetworkEdges_project_ChemicalEntities.csv
M data/wdcm2_visNetworkEdges_project_Event.csv
M data/wdcm2_visNetworkEdges_project_Gene.csv
M data/wdcm2_visNetworkEdges_project_GeographicalObject.csv
M data/wdcm2_visNetworkEdges_project_Human.csv
M data/wdcm2_visNetworkEdges_project_Organization.csv
M data/wdcm2_visNetworkEdges_project_ScientificArticle.csv
M data/wdcm2_visNetworkEdges_project_Taxon.csv
M data/wdcm2_visNetworkEdges_project_Thoroughfare.csv
M data/wdcm2_visNetworkEdges_project_Wikimedia.csv
M data/wdcm2_visNetworkNodes_project_ArchitecturalStructure.csv
M data/wdcm2_visNetworkNodes_project_AstronomicalObject.csv
M data/wdcm2_visNetworkNodes_project_Book.csv
M data/wdcm2_visNetworkNodes_project_ChemicalEntities.csv
M data/wdcm2_visNetworkNodes_project_Event.csv
M data/wdcm2_visNetworkNodes_project_Gene.csv
M data/wdcm2_visNetworkNodes_project_GeographicalObject.csv
M data/wdcm2_visNetworkNodes_project_Human.csv
M data/wdcm2_visNetworkNodes_project_Organization.csv
M data/wdcm2_visNetworkNodes_project_ScientificArticle.csv
M data/wdcm2_visNetworkNodes_project_Taxon.csv
M data/wdcm2_visNetworkNodes_project_Thoroughfare.csv
M data/wdcm2_visNetworkNodes_project_Wikimedia.csv
40 files changed, 13,867 insertions(+), 13,884 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Semantics-Dashboard 
refs/changes/96/398696/1


-- 
To view, visit https://gerrit.wikimedia.org/r/398696
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie38b02a6f9831fefe8c3b655980f5702b1b30f9d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Semantics-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Semantics-Dashboard[master]: Semantics ignore 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398696 )

Change subject: Semantics ignore 17 Dec 2017
..


Semantics ignore 17 Dec 2017

Change-Id: Ie38b02a6f9831fefe8c3b655980f5702b1b30f9d
---
A .gitignore
M data/wdcm2_projecttopic_ArchitecturalStructure.csv
M data/wdcm2_projecttopic_AstronomicalObject.csv
M data/wdcm2_projecttopic_Book.csv
M data/wdcm2_projecttopic_ChemicalEntities.csv
M data/wdcm2_projecttopic_Event.csv
M data/wdcm2_projecttopic_Gene.csv
M data/wdcm2_projecttopic_GeographicalObject.csv
M data/wdcm2_projecttopic_Human.csv
M data/wdcm2_projecttopic_Organization.csv
M data/wdcm2_projecttopic_ScientificArticle.csv
M data/wdcm2_projecttopic_Taxon.csv
M data/wdcm2_projecttopic_Thoroughfare.csv
M data/wdcm2_projecttopic_Wikimedia.csv
M data/wdcm2_visNetworkEdges_project_ArchitecturalStructure.csv
M data/wdcm2_visNetworkEdges_project_AstronomicalObject.csv
M data/wdcm2_visNetworkEdges_project_Book.csv
M data/wdcm2_visNetworkEdges_project_ChemicalEntities.csv
M data/wdcm2_visNetworkEdges_project_Event.csv
M data/wdcm2_visNetworkEdges_project_Gene.csv
M data/wdcm2_visNetworkEdges_project_GeographicalObject.csv
M data/wdcm2_visNetworkEdges_project_Human.csv
M data/wdcm2_visNetworkEdges_project_Organization.csv
M data/wdcm2_visNetworkEdges_project_ScientificArticle.csv
M data/wdcm2_visNetworkEdges_project_Taxon.csv
M data/wdcm2_visNetworkEdges_project_Thoroughfare.csv
M data/wdcm2_visNetworkEdges_project_Wikimedia.csv
M data/wdcm2_visNetworkNodes_project_ArchitecturalStructure.csv
M data/wdcm2_visNetworkNodes_project_AstronomicalObject.csv
M data/wdcm2_visNetworkNodes_project_Book.csv
M data/wdcm2_visNetworkNodes_project_ChemicalEntities.csv
M data/wdcm2_visNetworkNodes_project_Event.csv
M data/wdcm2_visNetworkNodes_project_Gene.csv
M data/wdcm2_visNetworkNodes_project_GeographicalObject.csv
M data/wdcm2_visNetworkNodes_project_Human.csv
M data/wdcm2_visNetworkNodes_project_Organization.csv
M data/wdcm2_visNetworkNodes_project_ScientificArticle.csv
M data/wdcm2_visNetworkNodes_project_Taxon.csv
M data/wdcm2_visNetworkNodes_project_Thoroughfare.csv
M data/wdcm2_visNetworkNodes_project_Wikimedia.csv
40 files changed, 13,867 insertions(+), 13,884 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified




-- 
To view, visit https://gerrit.wikimedia.org/r/398696
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie38b02a6f9831fefe8c3b655980f5702b1b30f9d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Semantics-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Overview-Dashboard[master]: Usage Dashboard ignore 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398697 )

Change subject: Usage Dashboard ignore 17 Dec 2017
..

Usage Dashboard ignore 17 Dec 2017

Change-Id: I11b522022fa4b4a0601c427f7e40522d443d8d6a
---
A .gitignore
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Overview-Dashboard 
refs/changes/97/398697/1

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..3159b93
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,3 @@
+update/
+www/
+

-- 
To view, visit https://gerrit.wikimedia.org/r/398697
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I11b522022fa4b4a0601c427f7e40522d443d8d6a
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Overview-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Overview-Dashboard[master]: Usage Dashboard ignore 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398697 )

Change subject: Usage Dashboard ignore 17 Dec 2017
..


Usage Dashboard ignore 17 Dec 2017

Change-Id: I11b522022fa4b4a0601c427f7e40522d443d8d6a
---
A .gitignore
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..3159b93
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,3 @@
+update/
+www/
+

-- 
To view, visit https://gerrit.wikimedia.org/r/398697
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I11b522022fa4b4a0601c427f7e40522d443d8d6a
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Overview-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Usage-Dashboard[master]: Usage Dashboard ignore 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398698 )

Change subject: Usage Dashboard ignore 17 Dec 2017
..

Usage Dashboard ignore 17 Dec 2017

Change-Id: Ie942db5bcfe8d5af4fc9d0847f42e22ad9beac51
---
A .gitignore
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Usage-Dashboard 
refs/changes/98/398698/1

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..515e514
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,3 @@
+www/
+update/
+

-- 
To view, visit https://gerrit.wikimedia.org/r/398698
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie942db5bcfe8d5af4fc9d0847f42e22ad9beac51
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Usage-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Usage-Dashboard[master]: Usage Dashboard ignore 17 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398698 )

Change subject: Usage Dashboard ignore 17 Dec 2017
..


Usage Dashboard ignore 17 Dec 2017

Change-Id: Ie942db5bcfe8d5af4fc9d0847f42e22ad9beac51
---
A .gitignore
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..515e514
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,3 @@
+www/
+update/
+

-- 
To view, visit https://gerrit.wikimedia.org/r/398698
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie942db5bcfe8d5af4fc9d0847f42e22ad9beac51
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Usage-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Production v2 16 Dec 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398655 )

Change subject: Production v2 16 Dec 2017
..

Production v2 16 Dec 2017

Change-Id: Icb4fa9f89460e8a6290a34e6305370a756d4a706
---
D WDCM_Collect_Items.R
A WDCM_CollectedGeoItems/.directory
A WDCM_CollectedGeoItems/archive_ItemIDs.csv
A WDCM_CollectedGeoItems/art gallery_ItemIDs.csv
A WDCM_CollectedGeoItems/botanical garden_ItemIDs.csv
A WDCM_CollectedGeoItems/city_ItemIDs.csv
A WDCM_CollectedGeoItems/country_ItemIDs.csv
A WDCM_CollectedGeoItems/library_ItemIDs.csv
A WDCM_CollectedGeoItems/museum_ItemIDs.csv
A WDCM_CollectedGeoItems/university_ItemIDs.csv
A WDCM_CollectedGeoItems/zoological garden_ItemIDs.csv
A WDCM_CollectedItems/.directory
A WDCM_CollectedItems/Human_ItemIDs.csv
A WDCM_CollectedItems/Wikimedia_Internal_ItemIDs.csv
M WDCM_Engine_goransm.R
A WDCM_Ontology/WDCM_GeoItems_Belgrade_12152017.csv
A WDCM_Ontology/WDCM_Ontology_Berlin_05032017.csv
D WDCM_Pre-Process.R
D WDCM_Process.R
D WDCM_Search_Clients.R
D WDCM_Search_Clients_MariaDB.R
M WDCM_Sqoop_Clients.R
M _installProduction_analytics-wmde.R
23 files changed, 68,601 insertions(+), 1,920 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/55/398655/1


-- 
To view, visit https://gerrit.wikimedia.org/r/398655
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icb4fa9f89460e8a6290a34e6305370a756d4a706
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: .csv deletions 16 Dec 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398656 )

Change subject: .csv deletions 16 Dec 2017
..


.csv deletions 16 Dec 2017

Change-Id: Ib6689cfad6a7480053aacc8ed5430845c4ebcb76
---
D WDCM_CollectedGeoItems/archive_ItemIDs.csv
D WDCM_CollectedGeoItems/art gallery_ItemIDs.csv
D WDCM_CollectedGeoItems/botanical garden_ItemIDs.csv
D WDCM_CollectedGeoItems/city_ItemIDs.csv
D WDCM_CollectedGeoItems/country_ItemIDs.csv
D WDCM_CollectedGeoItems/library_ItemIDs.csv
D WDCM_CollectedGeoItems/museum_ItemIDs.csv
D WDCM_CollectedGeoItems/university_ItemIDs.csv
D WDCM_CollectedGeoItems/zoological garden_ItemIDs.csv
D WDCM_CollectedItems/Human_ItemIDs.csv
D WDCM_CollectedItems/Wikimedia_Internal_ItemIDs.csv
11 files changed, 0 insertions(+), 68,016 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved
  jenkins-bot: Verified




-- 
To view, visit https://gerrit.wikimedia.org/r/398656
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib6689cfad6a7480053aacc8ed5430845c4ebcb76
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: .csv deletions 16 Dec 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398656 )

Change subject: .csv deletions 16 Dec 2017
..

.csv deletions 16 Dec 2017

Change-Id: Ib6689cfad6a7480053aacc8ed5430845c4ebcb76
---
D WDCM_CollectedGeoItems/archive_ItemIDs.csv
D WDCM_CollectedGeoItems/art gallery_ItemIDs.csv
D WDCM_CollectedGeoItems/botanical garden_ItemIDs.csv
D WDCM_CollectedGeoItems/city_ItemIDs.csv
D WDCM_CollectedGeoItems/country_ItemIDs.csv
D WDCM_CollectedGeoItems/library_ItemIDs.csv
D WDCM_CollectedGeoItems/museum_ItemIDs.csv
D WDCM_CollectedGeoItems/university_ItemIDs.csv
D WDCM_CollectedGeoItems/zoological garden_ItemIDs.csv
D WDCM_CollectedItems/Human_ItemIDs.csv
D WDCM_CollectedItems/Wikimedia_Internal_ItemIDs.csv
11 files changed, 0 insertions(+), 68,016 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/56/398656/1


-- 
To view, visit https://gerrit.wikimedia.org/r/398656
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib6689cfad6a7480053aacc8ed5430845c4ebcb76
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: EngineGeo Dec 16 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398652 )

Change subject: EngineGeo Dec 16 2017
..


EngineGeo Dec 16 2017

Change-Id: I3c5c2bdcd1503c99d5a8e22a2ee849edccae25ed
---
A WDCM_EngineGeo_goransm.R
1 file changed, 1,116 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
new file mode 100644
index 000..7013c3a
--- /dev/null
+++ b/WDCM_EngineGeo_goransm.R
@@ -0,0 +1,1116 @@
+#!/usr/bin/env Rscript
+
+### ---
+### --- WDCM EngineGeo, v. Beta 0.1
+### --- Script: WDCM_EngineGeo.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Engine_Geo contacts the WDQS SPARQL end-point
+### --- and fetches the item IDs for several Wikidata concepts
+### --- that have geographical co-ordinates. 
+### --- WDCM_Pre-Process.R
+### --- The remainder of the script searches the Hive goransm.wdcm_maintable
+### --- for usage data and prepares the export .tsv files
+### --- that migrate to Labs (wikidataconcepts, currently) where they are
+### --- additionaly processed and stored to MariaDB to support
+### --- the WDCM Geo Dashboard.
+### --- NOTE: the execution of this WDCM script is always dependent upon the
+### --- previous WDCM_Sqoop_Clients.R run from stat1004 (currently).
+### ---
+### --- RUN FROM: /home/goransm/RScripts/WDCM_R
+### --- nohup Rscript WDCM_EngineGeo_goransm.R &
+### ---
+
+### ---
+### --- LICENSE:
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### ---
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### ---
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of
+### --- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+### --- GNU General Public License for more details.
+### ---
+### --- You should have received a copy of the GNU General Public License
+### --- along with WDCM. If not, see .
+### ---
+
+### ---
+### --- Step 1: Collect from WQDS
+### ---
+
+### --- Setup
+# - contact:
+library(httr)
+library(XML)
+# - wrangling:
+library(stringr)
+library(readr)
+library(data.table)
+library(tidyr)
+library(jsonlite)
+
+### --- Directories
+# - fPath: where the scripts is run from?
+fPath <- 
'/home/goransm/Work/___DataKolektiv/Projects/WikimediaDEU/_WMDE_Projects/WDCM_Dev/WDCM/'
+# - form paths:
+ontologyDir <- paste(fPath, 'WDCM_Ontology', sep = "")
+logDir <- paste(fPath, 'WDCM_Logs', sep = "")
+itemsDir <- paste(fPath, 'WDCM_CollectedGeoItems', sep = "")
+# - stat1005 published-datasets, maps onto 
+# - https://analytics.wikimedia.org/datasets/wdcm/
+dataDir <- '/srv/published-datasets/wdcm'
+
+# - to runtime Log:
+print(paste("--- UPDATE RUN STARTED ON:", Sys.time(), sep = " "))
+
+### --- Set proxy
+# Sys.setenv(
+#   http_proxy = "http://webproxy.eqiad.wmnet:8080;,
+#   https_proxy = "http://webproxy.eqiad.wmnet:8080;)
+
+### --- Read WDCM_GeoItems
+# - to runtime Log:
+print("--- Reading Ontology.")
+setwd(ontologyDir)
+wdcmGeoItems <- read.csv("WDCM_GeoItems_Belgrade_12152017.csv",
+ header = T,
+ check.names = F,
+ stringsAsFactors = F)
+
+### --- Select all instances accross all sub-classes of searchItems:
+# - endPoint:
+endPointURL <- 
"https://query.wikidata.org/bigdata/namespace/wdq/sparql?format=json=;
+
+# - set itemsDir:
+setwd(itemsDir)
+
+# - clear output dir:
+lF <- list.files()
+rmF <- file.remove(lF)
+
+# - track uncompleted queries:
+qErrors <- character()
+
+# - startTime (WDCM Main Report)
+startTime <- as.character(Sys.time())
+
+for (i in 1:length(wdcmGeoItems$item)) {
+
+  # - to runtime Log:
+  print(paste("--- SPARQL category:",

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: EngineGeo Dec 16 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398652 )

Change subject: EngineGeo Dec 16 2017
..

EngineGeo Dec 16 2017

Change-Id: I3c5c2bdcd1503c99d5a8e22a2ee849edccae25ed
---
A WDCM_EngineGeo_goransm.R
1 file changed, 1,116 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/52/398652/1

diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
new file mode 100644
index 000..7013c3a
--- /dev/null
+++ b/WDCM_EngineGeo_goransm.R
@@ -0,0 +1,1116 @@
+#!/usr/bin/env Rscript
+
+### ---
+### --- WDCM EngineGeo, v. Beta 0.1
+### --- Script: WDCM_EngineGeo.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Engine_Geo contacts the WDQS SPARQL end-point
+### --- and fetches the item IDs for several Wikidata concepts
+### --- that have geographical co-ordinates. 
+### --- WDCM_Pre-Process.R
+### --- The remainder of the script searches the Hive goransm.wdcm_maintable
+### --- for usage data and prepares the export .tsv files
+### --- that migrate to Labs (wikidataconcepts, currently) where they are
+### --- additionaly processed and stored to MariaDB to support
+### --- the WDCM Geo Dashboard.
+### --- NOTE: the execution of this WDCM script is always dependent upon the
+### --- previous WDCM_Sqoop_Clients.R run from stat1004 (currently).
+### ---
+### --- RUN FROM: /home/goransm/RScripts/WDCM_R
+### --- nohup Rscript WDCM_EngineGeo_goransm.R &
+### ---
+
+### ---
+### --- LICENSE:
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### ---
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### ---
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of
+### --- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+### --- GNU General Public License for more details.
+### ---
+### --- You should have received a copy of the GNU General Public License
+### --- along with WDCM. If not, see .
+### ---
+
+### ---
+### --- Step 1: Collect from WQDS
+### ---
+
+### --- Setup
+# - contact:
+library(httr)
+library(XML)
+# - wrangling:
+library(stringr)
+library(readr)
+library(data.table)
+library(tidyr)
+library(jsonlite)
+
+### --- Directories
+# - fPath: where the scripts is run from?
+fPath <- 
'/home/goransm/Work/___DataKolektiv/Projects/WikimediaDEU/_WMDE_Projects/WDCM_Dev/WDCM/'
+# - form paths:
+ontologyDir <- paste(fPath, 'WDCM_Ontology', sep = "")
+logDir <- paste(fPath, 'WDCM_Logs', sep = "")
+itemsDir <- paste(fPath, 'WDCM_CollectedGeoItems', sep = "")
+# - stat1005 published-datasets, maps onto 
+# - https://analytics.wikimedia.org/datasets/wdcm/
+dataDir <- '/srv/published-datasets/wdcm'
+
+# - to runtime Log:
+print(paste("--- UPDATE RUN STARTED ON:", Sys.time(), sep = " "))
+
+### --- Set proxy
+# Sys.setenv(
+#   http_proxy = "http://webproxy.eqiad.wmnet:8080;,
+#   https_proxy = "http://webproxy.eqiad.wmnet:8080;)
+
+### --- Read WDCM_GeoItems
+# - to runtime Log:
+print("--- Reading Ontology.")
+setwd(ontologyDir)
+wdcmGeoItems <- read.csv("WDCM_GeoItems_Belgrade_12152017.csv",
+ header = T,
+ check.names = F,
+ stringsAsFactors = F)
+
+### --- Select all instances accross all sub-classes of searchItems:
+# - endPoint:
+endPointURL <- 
"https://query.wikidata.org/bigdata/namespace/wdq/sparql?format=json=;
+
+# - set itemsDir:
+setwd(itemsDir)
+
+# - clear output dir:
+lF <- list.files()
+rmF <- file.remove(lF)
+
+# - track uncompleted queries:
+qErrors <- character()
+
+# - startTime (WDCM Main Report)
+startTime <- as.character(Sys.time())
+
+for (i in 1:length(wdcmGeoItems$item)) {
+
+  # - to runtime Log:
+  print(paste("--- SPARQL

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Production v2 16 Dec 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398655 )

Change subject: Production v2 16 Dec 2017
..


Production v2 16 Dec 2017

Change-Id: Icb4fa9f89460e8a6290a34e6305370a756d4a706
---
D WDCM_Collect_Items.R
A WDCM_CollectedGeoItems/.directory
A WDCM_CollectedGeoItems/archive_ItemIDs.csv
A WDCM_CollectedGeoItems/art gallery_ItemIDs.csv
A WDCM_CollectedGeoItems/botanical garden_ItemIDs.csv
A WDCM_CollectedGeoItems/city_ItemIDs.csv
A WDCM_CollectedGeoItems/country_ItemIDs.csv
A WDCM_CollectedGeoItems/library_ItemIDs.csv
A WDCM_CollectedGeoItems/museum_ItemIDs.csv
A WDCM_CollectedGeoItems/university_ItemIDs.csv
A WDCM_CollectedGeoItems/zoological garden_ItemIDs.csv
A WDCM_CollectedItems/.directory
A WDCM_CollectedItems/Human_ItemIDs.csv
A WDCM_CollectedItems/Wikimedia_Internal_ItemIDs.csv
M WDCM_Engine_goransm.R
A WDCM_Ontology/WDCM_GeoItems_Belgrade_12152017.csv
A WDCM_Ontology/WDCM_Ontology_Berlin_05032017.csv
D WDCM_Pre-Process.R
D WDCM_Process.R
D WDCM_Search_Clients.R
D WDCM_Search_Clients_MariaDB.R
M WDCM_Sqoop_Clients.R
M _installProduction_analytics-wmde.R
23 files changed, 68,601 insertions(+), 1,920 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved
  jenkins-bot: Verified




-- 
To view, visit https://gerrit.wikimedia.org/r/398655
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icb4fa9f89460e8a6290a34e6305370a756d4a706
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: README.md change

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398657 )

Change subject: README.md change
..


README.md change

Change-Id: I3d136e860ed7a74db7650b84bef4ab53861ae11f
---
M README.md
1 file changed, 10 insertions(+), 3 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/README.md b/README.md
index 42231e7..0114fd7 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,17 @@
 # WDCM
 Wikidata Concept Monitor
 
-Wikidata Concept Monitor (WDCM) is a set of analytical dashboards that track 
the usage of Wikidata across the Wikimedia projects.
-WDCM is currently under development. All R scrips should be considered as test 
versions only.
-Thorough documentation will be provided soon. Stay tuned.
+The Wikidata Concepts Monitor (WDCM) (aka: Q42376073) is an analytical tool 
that enables you to browse and build an understanding of the way Wikidata is 
used across the Wikimedia projects.
 
+While Wikidata itself is a semantic ontology with pre-defined and evolving 
normative rules of description and inference, Wikidata usage is essentialy a 
social, behavioral phenomenon, suitable for study by means of machine learning 
in the field of distributional semantics: the analysis and modeling of 
statistical patterns of occurrence and co-occurence of Wikidata item and 
property usage across the client projects (e.g. enwiki, frwiki, ruwiki, etc). 
+
+WDCM thus employs various statistical methodologies in an attempt to describe 
and provide insights from the observable Wikidata usage statistics (e.g. topic 
modeling, clustering, dimensionality reduction, all beyond providing elementary 
descriptive statistics of Wikidata usage, of course.).
+
+The WDCM is designed to become a path towards discovery: following the 
examples listed here not only that you can learn to work with a system that 
might improve your understanding of Wikidata, but you could also find yourself 
involved in adventurous attempts to learn and discover more of it.
+
+WDCM Wiki project page: 
https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor
+WDCM Wikitech page: 
https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor
+WDCM User Feedback page: 
https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor/UserFeedback
 ---
 
 WDCM is developed by Goran S. Milovanovic, Data Analyst, WMDE, under the 
contract between Goran Milovanovic PR Data Kolektiv and WMDE.
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/398657
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3d136e860ed7a74db7650b84bef4ab53861ae11f
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: README.md change

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398657 )

Change subject: README.md change
..

README.md change

Change-Id: I3d136e860ed7a74db7650b84bef4ab53861ae11f
---
M README.md
1 file changed, 10 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/57/398657/1

diff --git a/README.md b/README.md
index 42231e7..0114fd7 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,17 @@
 # WDCM
 Wikidata Concept Monitor
 
-Wikidata Concept Monitor (WDCM) is a set of analytical dashboards that track 
the usage of Wikidata across the Wikimedia projects.
-WDCM is currently under development. All R scrips should be considered as test 
versions only.
-Thorough documentation will be provided soon. Stay tuned.
+The Wikidata Concepts Monitor (WDCM) (aka: Q42376073) is an analytical tool 
that enables you to browse and build an understanding of the way Wikidata is 
used across the Wikimedia projects.
 
+While Wikidata itself is a semantic ontology with pre-defined and evolving 
normative rules of description and inference, Wikidata usage is essentialy a 
social, behavioral phenomenon, suitable for study by means of machine learning 
in the field of distributional semantics: the analysis and modeling of 
statistical patterns of occurrence and co-occurence of Wikidata item and 
property usage across the client projects (e.g. enwiki, frwiki, ruwiki, etc). 
+
+WDCM thus employs various statistical methodologies in an attempt to describe 
and provide insights from the observable Wikidata usage statistics (e.g. topic 
modeling, clustering, dimensionality reduction, all beyond providing elementary 
descriptive statistics of Wikidata usage, of course.).
+
+The WDCM is designed to become a path towards discovery: following the 
examples listed here not only that you can learn to work with a system that 
might improve your understanding of Wikidata, but you could also find yourself 
involved in adventurous attempts to learn and discover more of it.
+
+WDCM Wiki project page: 
https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor
+WDCM Wikitech page: 
https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor
+WDCM User Feedback page: 
https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor/UserFeedback
 ---
 
 WDCM is developed by Goran S. Milovanovic, Data Analyst, WMDE, under the 
contract between Goran Milovanovic PR Data Kolektiv and WMDE.
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/398657
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3d136e860ed7a74db7650b84bef4ab53861ae11f
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: EngineGeo 16 Dec 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398666 )

Change subject: EngineGeo 16 Dec 2017
..


EngineGeo 16 Dec 2017

Change-Id: I4750665c1789c201d579c425b9231a1f848be3a0
---
M WDCM_EngineGeo_goransm.R
1 file changed, 1 insertion(+), 900 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 7013c3a..25e8a40 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -209,908 +209,9 @@
 }
 
 ### ---
-### --- Script 2: WDCM_Search_Clients.R, WDCM Search Module
+### --- Step 2: ETL: Wikidata usage statistics from WDCM Maintable
 ### ---
 
-### ---
-### --- WDCM Search Module, v. Beta 0.1
-### --- Script: WDCM_Search_Clients.R, v. Beta 0.1
-### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
-### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
-### --- and WMDE.
-### --- Contact: goran.milovanovic_...@wikimedia.de
-### ---
-### --- DESCRIPTION:
-### --- WDCM_Search_Clients.R takes a list of
-### --- item IDs from Wikidata (the list is delivered by
-### --- the WDCM_Collect_Items.R script) and searches for their
-### --- usage across the Wikimedia projects in Hadoop:
-### --- database: goransm
-### --- directory: wdcmsqoop
-### --- table: wdcm_clients_wb_entity_usage
-### --- from production (currently: stat1005.eqiad.wmnet).
-### --- NOTE: wdcm_clients_wb_entity_usage is produced by
-### --- WDCM_Sqoop_Clients.R (currently run from: stat1004.eqiad.wmnet)
-### ---
-### --- INPUT:
-### --- the WDCM_Search_Clients_HiveQL.R reads the list of item IDs
-### --- to search for from /WDCM_CollectedItems
-### --- This folder contains the .csv files that specify the item IDs
-### --- to search for; the files are produced by Scrpt 1: WDCM_Collect_Items.R
-### ---
-### --- OUTPUT:
-### --- wdcm_maintable Hive table on hdfs, database: goransm
-### ---
-
-### --- read item categories:
-setwd(itemsDir)
-idFiles <- list.files()
-idFiles <- idFiles[grepl(".csv$", idFiles)]
-idFilesSize <- file.size(idFiles)/1024^2
-
-### --- Track all categories under processing:
-# - check for the existence of the wdcmSearchReport file
-# - and delete the old file if it's found:
-setwd(logDir)
-lF <- list.files()
-w <- which(grepl("^wdcmSearchReport", lF))
-if (length(w) == 1) {
-  file.remove(lF[w])
-}
-### --- generate wdcmSearchReport data.frame
-wdcmSearchReport <- data.frame(category = idFiles,
-   fileSize = idFilesSize,
-   startTime = character(length(idFiles)),
-   endTime = character(length(idFiles)),
-   stringsAsFactors = F
-)
-wdcmSearchReport <- wdcmSearchReport[order(-wdcmSearchReport$fileSize), ]
-
-### --- PREPARATION: delete goransm.wdcm_maintable if exists,
-### --- delete all from EXTERNAL Hive table from /user/goransm/wdcmtables 
(hdfs path)
-### --- make directory for EXTERNAL Hive table /user/goransm/wdcmtables (hdfs 
path)
-
-### --- check if goransm.wdcm_maintable exists in Hadoop; if yes, drop it:
-# - NOTE: drop wdcm_maintable == erase metastore data:
-# - [query01Err]
-
-# - to runtime Log:
-print("Running query [query01Err].")
-
-query01Err <- system(command = '/usr/local/bin/beeline --silent -e "USE 
goransm; DROP TABLE IF EXISTS wdcm_maintable;"', wait = T)
-if (query01Err != 0) {
-  # - to runtime Log:
-  print("--- (!!!) query01Err failed: waiting for 1h before next attempt...")
-  # - sleep for one hour
-  Sys.sleep(time = 60*60)
-  # - re-run query
-  query01Err <- system(command = '/usr/local/bin/beeline --silent -e "USE 
goransm; DROP TABLE IF EXISTS wdcm_maintable;"', wait = T)
-  # - check errors:
-  if (query01Err != 0) {
-# - to runtime Log:
-print("--- (!!!) query01Err failed AGAIN: quiting.")
-quit()
-  }
-}
-
-### --- delete files for EXTERNAL Hive table from /user/goransm/wdcmtables 
(hdfs path)
-# - [query02Err]
-# - to runtime Log:
-print("--- Running query [query02Err].")
-query02Err <- system(command = 'hdfs dfs -rm -r /user/goransm/wdcmtables', 
wait = T)
-if (query02Err != 0) {
-  # - to runtime Log:
-  print("--- (!!!) query02Err failed: waiting for 1h before next attempt...")
-  # - sleep for one hour
-  Sys.sleep(time = 60*60)
-  # - re-run query
-  query02Err <- system(command = 'hdfs dfs -rm

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: EngineGeo 16 Dec 2017

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398666 )

Change subject: EngineGeo 16 Dec 2017
..

EngineGeo 16 Dec 2017

Change-Id: I4750665c1789c201d579c425b9231a1f848be3a0
---
M WDCM_EngineGeo_goransm.R
1 file changed, 1 insertion(+), 900 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/66/398666/1

diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 7013c3a..25e8a40 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -209,908 +209,9 @@
 }
 
 ### ---
-### --- Script 2: WDCM_Search_Clients.R, WDCM Search Module
+### --- Step 2: ETL: Wikidata usage statistics from WDCM Maintable
 ### ---
 
-### ---
-### --- WDCM Search Module, v. Beta 0.1
-### --- Script: WDCM_Search_Clients.R, v. Beta 0.1
-### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
-### --- Developed under the contract between Goran Milovanovic PR Data Kolektiv
-### --- and WMDE.
-### --- Contact: goran.milovanovic_...@wikimedia.de
-### ---
-### --- DESCRIPTION:
-### --- WDCM_Search_Clients.R takes a list of
-### --- item IDs from Wikidata (the list is delivered by
-### --- the WDCM_Collect_Items.R script) and searches for their
-### --- usage across the Wikimedia projects in Hadoop:
-### --- database: goransm
-### --- directory: wdcmsqoop
-### --- table: wdcm_clients_wb_entity_usage
-### --- from production (currently: stat1005.eqiad.wmnet).
-### --- NOTE: wdcm_clients_wb_entity_usage is produced by
-### --- WDCM_Sqoop_Clients.R (currently run from: stat1004.eqiad.wmnet)
-### ---
-### --- INPUT:
-### --- the WDCM_Search_Clients_HiveQL.R reads the list of item IDs
-### --- to search for from /WDCM_CollectedItems
-### --- This folder contains the .csv files that specify the item IDs
-### --- to search for; the files are produced by Scrpt 1: WDCM_Collect_Items.R
-### ---
-### --- OUTPUT:
-### --- wdcm_maintable Hive table on hdfs, database: goransm
-### ---
-
-### --- read item categories:
-setwd(itemsDir)
-idFiles <- list.files()
-idFiles <- idFiles[grepl(".csv$", idFiles)]
-idFilesSize <- file.size(idFiles)/1024^2
-
-### --- Track all categories under processing:
-# - check for the existence of the wdcmSearchReport file
-# - and delete the old file if it's found:
-setwd(logDir)
-lF <- list.files()
-w <- which(grepl("^wdcmSearchReport", lF))
-if (length(w) == 1) {
-  file.remove(lF[w])
-}
-### --- generate wdcmSearchReport data.frame
-wdcmSearchReport <- data.frame(category = idFiles,
-   fileSize = idFilesSize,
-   startTime = character(length(idFiles)),
-   endTime = character(length(idFiles)),
-   stringsAsFactors = F
-)
-wdcmSearchReport <- wdcmSearchReport[order(-wdcmSearchReport$fileSize), ]
-
-### --- PREPARATION: delete goransm.wdcm_maintable if exists,
-### --- delete all from EXTERNAL Hive table from /user/goransm/wdcmtables 
(hdfs path)
-### --- make directory for EXTERNAL Hive table /user/goransm/wdcmtables (hdfs 
path)
-
-### --- check if goransm.wdcm_maintable exists in Hadoop; if yes, drop it:
-# - NOTE: drop wdcm_maintable == erase metastore data:
-# - [query01Err]
-
-# - to runtime Log:
-print("Running query [query01Err].")
-
-query01Err <- system(command = '/usr/local/bin/beeline --silent -e "USE 
goransm; DROP TABLE IF EXISTS wdcm_maintable;"', wait = T)
-if (query01Err != 0) {
-  # - to runtime Log:
-  print("--- (!!!) query01Err failed: waiting for 1h before next attempt...")
-  # - sleep for one hour
-  Sys.sleep(time = 60*60)
-  # - re-run query
-  query01Err <- system(command = '/usr/local/bin/beeline --silent -e "USE 
goransm; DROP TABLE IF EXISTS wdcm_maintable;"', wait = T)
-  # - check errors:
-  if (query01Err != 0) {
-# - to runtime Log:
-print("--- (!!!) query01Err failed AGAIN: quiting.")
-quit()
-  }
-}
-
-### --- delete files for EXTERNAL Hive table from /user/goransm/wdcmtables 
(hdfs path)
-# - [query02Err]
-# - to runtime Log:
-print("--- Running query [query02Err].")
-query02Err <- system(command = 'hdfs dfs -rm -r /user/goransm/wdcmtables', 
wait = T)
-if (query02Err != 0) {
-  # - to runtime Log:
-  print("--- (!!!) query02Err failed: waiting for 1h before next attempt...")
-  # - sleep for one hour
-  Sys.sleep(time = 60*60)
-  # - re-run query
-  query02Err <- system(command = 'hdfs dfs -rm -r

[MediaWiki-commits] [Gerrit] analytics...WDCM-Semantics-Dashboard[master]: minor

2017-12-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398835 )

Change subject: minor
..


minor

Change-Id: Ic011d084bd028fed23f90fe4c45db4fad1557791
---
M ui.R
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/ui.R b/ui.R
index 0f75409..0b780ef 100644
--- a/ui.R
+++ b/ui.R
@@ -376,6 +376,8 @@
Fine-grained information on Wikidata usage across client projects and 
project types. Cross-tabulations and similar...
WDCM Items
Fine-grained information on particular Wikidata item usage across the 
client projects. (Under development)
+   http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM Geo
+   Wikidata 
items interactive maps.
https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
The WDCM 
Wikitech Page.
'

-- 
To view, visit https://gerrit.wikimedia.org/r/398835
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic011d084bd028fed23f90fe4c45db4fad1557791
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Semantics-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Overview-Dashboard[master]: minor

2017-12-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398833 )

Change subject: minor
..


minor

Change-Id: I6e4fa36b42b427e3d5173376586101f78c899a41
---
M ui.R
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/ui.R b/ui.R
index fa722ef..8c7322f 100644
--- a/ui.R
+++ b/ui.R
@@ -343,6 +343,8 @@
Fine-grained information on Wikidata usage across client projects and 
project types. Cross-tabulations and similar..
WDCM Items
Fine-grained information on particular Wikidata item usage across the 
client projects. (Under development)
+   http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM Geo
+   Wikidata 
items interactive maps.
https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
The WDCM 
Wikitech Page.
'

-- 
To view, visit https://gerrit.wikimedia.org/r/398833
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6e4fa36b42b427e3d5173376586101f78c899a41
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Overview-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Usage-Dashboard[master]: minor

2017-12-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398834 )

Change subject: minor
..


minor

Change-Id: Id50e661a559848e454ac0595d961a16b2b0cdae4
---
M ui.R
1 file changed, 3 insertions(+), 1 deletion(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/ui.R b/ui.R
index 0caf258..3a63c91 100644
--- a/ui.R
+++ b/ui.R
@@ -531,7 +531,9 @@
Fine-grained information on Wikidata usage across client projects and 
project types. Cross-tabulations and similar..
WDCM Items
Fine-grained information on particular Wikidata item usage across the 
client projects. (Under development)
-  https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
+   http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM Geo
+   Wikidata 
items interactive maps.
+   https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
The WDCM 
Wikitech Page.
'
 )

-- 
To view, visit https://gerrit.wikimedia.org/r/398834
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id50e661a559848e454ac0595d961a16b2b0cdae4
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Usage-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Overview-Dashboard[master]: minor

2017-12-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398833 )

Change subject: minor
..

minor

Change-Id: I6e4fa36b42b427e3d5173376586101f78c899a41
---
M ui.R
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Overview-Dashboard 
refs/changes/33/398833/1

diff --git a/ui.R b/ui.R
index fa722ef..8c7322f 100644
--- a/ui.R
+++ b/ui.R
@@ -343,6 +343,8 @@
Fine-grained information on Wikidata usage across client projects and 
project types. Cross-tabulations and similar..
WDCM Items
Fine-grained information on particular Wikidata item usage across the 
client projects. (Under development)
+   http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM Geo
+   Wikidata 
items interactive maps.
https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
The WDCM 
Wikitech Page.
'

-- 
To view, visit https://gerrit.wikimedia.org/r/398833
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6e4fa36b42b427e3d5173376586101f78c899a41
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Overview-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Usage-Dashboard[master]: minor

2017-12-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398834 )

Change subject: minor
..

minor

Change-Id: Id50e661a559848e454ac0595d961a16b2b0cdae4
---
M ui.R
1 file changed, 3 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Usage-Dashboard 
refs/changes/34/398834/1

diff --git a/ui.R b/ui.R
index 0caf258..3a63c91 100644
--- a/ui.R
+++ b/ui.R
@@ -531,7 +531,9 @@
Fine-grained information on Wikidata usage across client projects and 
project types. Cross-tabulations and similar..
WDCM Items
Fine-grained information on particular Wikidata item usage across the 
client projects. (Under development)
-  https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
+   http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM Geo
+   Wikidata 
items interactive maps.
+   https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
The WDCM 
Wikitech Page.
'
 )

-- 
To view, visit https://gerrit.wikimedia.org/r/398834
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id50e661a559848e454ac0595d961a16b2b0cdae4
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Usage-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Semantics-Dashboard[master]: minor

2017-12-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398835 )

Change subject: minor
..

minor

Change-Id: Ic011d084bd028fed23f90fe4c45db4fad1557791
---
M ui.R
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Semantics-Dashboard 
refs/changes/35/398835/1

diff --git a/ui.R b/ui.R
index 0f75409..0b780ef 100644
--- a/ui.R
+++ b/ui.R
@@ -376,6 +376,8 @@
Fine-grained information on Wikidata usage across client projects and 
project types. Cross-tabulations and similar...
WDCM Items
Fine-grained information on particular Wikidata item usage across the 
client projects. (Under development)
+   http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM Geo
+   Wikidata 
items interactive maps.
https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM System 
Technical Documentation
The WDCM 
Wikitech Page.
'

-- 
To view, visit https://gerrit.wikimedia.org/r/398835
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic011d084bd028fed23f90fe4c45db4fad1557791
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Semantics-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: EngineGeo 16 Dec 2017 ed2

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398667 )

Change subject: EngineGeo 16 Dec 2017 ed2
..


EngineGeo 16 Dec 2017 ed2

Change-Id: I0f16705bf73bd0e66a19636321fd438b386f6139
---
M WDCM_EngineGeo_goransm.R
M WDCM_Engine_goransm.R
2 files changed, 122 insertions(+), 14 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 25e8a40..9d9c8df 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -54,6 +54,7 @@
 library(httr)
 library(XML)
 # - wrangling:
+library(dplyr)
 library(stringr)
 library(readr)
 library(data.table)
@@ -62,11 +63,11 @@
 
 ### --- Directories
 # - fPath: where the scripts is run from?
-fPath <- 
'/home/goransm/Work/___DataKolektiv/Projects/WikimediaDEU/_WMDE_Projects/WDCM_Dev/WDCM/'
+fPath <- '/home/goransm/RScripts/WDCM_R'
 # - form paths:
-ontologyDir <- paste(fPath, 'WDCM_Ontology', sep = "")
-logDir <- paste(fPath, 'WDCM_Logs', sep = "")
-itemsDir <- paste(fPath, 'WDCM_CollectedGeoItems', sep = "")
+ontologyDir <- paste(fPath, '/WDCM_Ontology', sep = "")
+logDir <- paste(fPath, '/WDCM_Logs', sep = "")
+itemsDir <- paste(fPath, '/WDCM_CollectedGeoItems', sep = "")
 # - stat1005 published-datasets, maps onto 
 # - https://analytics.wikimedia.org/datasets/wdcm/
 dataDir <- '/srv/published-datasets/wdcm'
@@ -75,9 +76,9 @@
 print(paste("--- UPDATE RUN STARTED ON:", Sys.time(), sep = " "))
 
 ### --- Set proxy
-# Sys.setenv(
-#   http_proxy = "http://webproxy.eqiad.wmnet:8080;,
-#   https_proxy = "http://webproxy.eqiad.wmnet:8080;)
+Sys.setenv(
+  http_proxy = "http://webproxy.eqiad.wmnet:8080;,
+  https_proxy = "http://webproxy.eqiad.wmnet:8080;)
 
 ### --- Read WDCM_GeoItems
 # - to runtime Log:
@@ -114,8 +115,6 @@
   strsplit(wdcmGeoItems$item[i],
split = ",", fixed = T)[[1]],
   "both")
-  
-  itemsOut <- list()
   
   # - Construct Query:
   query <- paste0(
@@ -190,8 +189,8 @@
 write.csv(qErrors, "WDCM_Collect_GeoItems_SPARQL_Errors.csv")
 # - write to WDCM main reporting file:
 lF <- list.files()
-if ('WDCM_MainReport.csv' %in% lF) {
-  mainReport <- read.csv('WDCM_MainReport.csv',
+if ('WDCM_GeoReport.csv' %in% lF) {
+  mainReport <- read.csv('WDCM_GeoReport.csv',
  header = T,
  row.names = 1,
  check.names = F,
@@ -200,18 +199,128 @@
   Time = as.character(Sys.time()),
   stringsAsFactors = F)
   mainReport <- rbind(mainReport, newReport)
-  write.csv(mainReport, 'WDCM_MainReport.csv')
+  write.csv(mainReport, 'WDCM_GeoReport.csv')
 } else {
   newReport <- data.frame(Step = 'CollectItems',
   Time = as.character(Sys.time()),
   stringsAsFactors = F)
-  write.csv(newReport, 'WDCM_MainReport.csv')
+  write.csv(newReport, 'WDCM_GeoReport.csv')
 }
 
 ### ---
 ### --- Step 2: ETL: Wikidata usage statistics from WDCM Maintable
 ### ---
 
+# - to runtime Log:
+print("--- ETL: Wikidata usage statistics from wdcm_maintable")
+
+### --- read item categories:
+setwd(itemsDir)
+
+idFiles <- list.files()
+idFiles <- idFiles[grepl(".csv$", idFiles)]
+categories <- unname(sapply(idFiles, function(x) {
+  strsplit(x, split = "_")[[1]][1]
+}))
+
+for (i in 1:length(categories)) {
+  
+  # - filename
+  filename <- paste("wdcm_geoitem_",
+gsub(" ", "", categories[i], fixed = T), ".tsv",
+sep = "")
+  
+  # - geoitems
+  geoitems <- read.csv(idFiles[i],
+   header = T,
+   check.names = F,
+   stringsAsFactors = F)
+  searchitems <- geoitems$item
+  
+  # - to runtime Log:
+  print(paste("--- processing: ", i, ": ", filename, sep = ""))
+  
+  # - to runtime Log:
+  print("--- RUNNING HiveQL Query to search for geoitems.")
+  hiveQLQuery_1 <- "USE goransm; SET hive.mapred.mode=unstrict;"
+  hiveQLQuery_2 <- paste("SELECT eu_entity_id, SUM(eu_count) AS usage FROM 
wdcm_maintable WHERE eu_entity_id IN (",
+ paste("\"", searchitems, "\"", collapse = ", ", sep = 
""),
+ ") GROUP BY eu_entity_id ORDER BY usage DESC LIMIT 
1000;",
+ sep = "")
+  hiveQLQuery <- paste(hiveQLQuery_1, hiveQLQuery_2, sep = " ")
+  
+  # - write hiveQLQuery locally:
+  setwd(fPath)
+  write(hiveQLQuery, "hiveQL_geoQuery.hql")
+  
+  # - execute HiveQLQuery:
+  hiveQLQueryCommand <- paste("/usr/local/bin/beeline --silent -f ", 
+  getwd(), "/hiveQL_geoQuery.hql", 
+  " > ", dataDir, "/", filename, 
+  sep = "")
+

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: EngineGeo 16 Dec 2017 ed2

2017-12-16 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398667 )

Change subject: EngineGeo 16 Dec 2017 ed2
..

EngineGeo 16 Dec 2017 ed2

Change-Id: I0f16705bf73bd0e66a19636321fd438b386f6139
---
M WDCM_EngineGeo_goransm.R
M WDCM_Engine_goransm.R
2 files changed, 122 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/67/398667/1

diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 25e8a40..9d9c8df 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -54,6 +54,7 @@
 library(httr)
 library(XML)
 # - wrangling:
+library(dplyr)
 library(stringr)
 library(readr)
 library(data.table)
@@ -62,11 +63,11 @@
 
 ### --- Directories
 # - fPath: where the scripts is run from?
-fPath <- 
'/home/goransm/Work/___DataKolektiv/Projects/WikimediaDEU/_WMDE_Projects/WDCM_Dev/WDCM/'
+fPath <- '/home/goransm/RScripts/WDCM_R'
 # - form paths:
-ontologyDir <- paste(fPath, 'WDCM_Ontology', sep = "")
-logDir <- paste(fPath, 'WDCM_Logs', sep = "")
-itemsDir <- paste(fPath, 'WDCM_CollectedGeoItems', sep = "")
+ontologyDir <- paste(fPath, '/WDCM_Ontology', sep = "")
+logDir <- paste(fPath, '/WDCM_Logs', sep = "")
+itemsDir <- paste(fPath, '/WDCM_CollectedGeoItems', sep = "")
 # - stat1005 published-datasets, maps onto 
 # - https://analytics.wikimedia.org/datasets/wdcm/
 dataDir <- '/srv/published-datasets/wdcm'
@@ -75,9 +76,9 @@
 print(paste("--- UPDATE RUN STARTED ON:", Sys.time(), sep = " "))
 
 ### --- Set proxy
-# Sys.setenv(
-#   http_proxy = "http://webproxy.eqiad.wmnet:8080;,
-#   https_proxy = "http://webproxy.eqiad.wmnet:8080;)
+Sys.setenv(
+  http_proxy = "http://webproxy.eqiad.wmnet:8080;,
+  https_proxy = "http://webproxy.eqiad.wmnet:8080;)
 
 ### --- Read WDCM_GeoItems
 # - to runtime Log:
@@ -114,8 +115,6 @@
   strsplit(wdcmGeoItems$item[i],
split = ",", fixed = T)[[1]],
   "both")
-  
-  itemsOut <- list()
   
   # - Construct Query:
   query <- paste0(
@@ -190,8 +189,8 @@
 write.csv(qErrors, "WDCM_Collect_GeoItems_SPARQL_Errors.csv")
 # - write to WDCM main reporting file:
 lF <- list.files()
-if ('WDCM_MainReport.csv' %in% lF) {
-  mainReport <- read.csv('WDCM_MainReport.csv',
+if ('WDCM_GeoReport.csv' %in% lF) {
+  mainReport <- read.csv('WDCM_GeoReport.csv',
  header = T,
  row.names = 1,
  check.names = F,
@@ -200,18 +199,128 @@
   Time = as.character(Sys.time()),
   stringsAsFactors = F)
   mainReport <- rbind(mainReport, newReport)
-  write.csv(mainReport, 'WDCM_MainReport.csv')
+  write.csv(mainReport, 'WDCM_GeoReport.csv')
 } else {
   newReport <- data.frame(Step = 'CollectItems',
   Time = as.character(Sys.time()),
   stringsAsFactors = F)
-  write.csv(newReport, 'WDCM_MainReport.csv')
+  write.csv(newReport, 'WDCM_GeoReport.csv')
 }
 
 ### ---
 ### --- Step 2: ETL: Wikidata usage statistics from WDCM Maintable
 ### ---
 
+# - to runtime Log:
+print("--- ETL: Wikidata usage statistics from wdcm_maintable")
+
+### --- read item categories:
+setwd(itemsDir)
+
+idFiles <- list.files()
+idFiles <- idFiles[grepl(".csv$", idFiles)]
+categories <- unname(sapply(idFiles, function(x) {
+  strsplit(x, split = "_")[[1]][1]
+}))
+
+for (i in 1:length(categories)) {
+  
+  # - filename
+  filename <- paste("wdcm_geoitem_",
+gsub(" ", "", categories[i], fixed = T), ".tsv",
+sep = "")
+  
+  # - geoitems
+  geoitems <- read.csv(idFiles[i],
+   header = T,
+   check.names = F,
+   stringsAsFactors = F)
+  searchitems <- geoitems$item
+  
+  # - to runtime Log:
+  print(paste("--- processing: ", i, ": ", filename, sep = ""))
+  
+  # - to runtime Log:
+  print("--- RUNNING HiveQL Query to search for geoitems.")
+  hiveQLQuery_1 <- "USE goransm; SET hive.mapred.mode=unstrict;"
+  hiveQLQuery_2 <- paste("SELECT eu_entity_id, SUM(eu_count) AS usage FROM 
wdcm_maintable WHERE eu_entity_id IN (",
+ paste("\"", searchitems, "\"", collapse = ", ", sep = 
""),
+ ") GROUP BY eu_entity_id ORDER BY usage DESC LIMIT 
1000;",
+ sep = "")
+  hiveQLQuery <- paste(hiveQLQuery_1, hiveQLQuery_2, sep = " ")
+  
+  # - write hiveQLQuery locally:
+  setwd(fPath)
+  write(hiveQLQuery, "hiveQL_geoQuery.hql")
+  
+  # - execute HiveQLQuery:
+  hiveQLQueryCommand <- paste("/usr/local/bin/beeline --silent -f ", 
+  getwd(), "/hiveQL_geoQuery.hql", 
+  " > ", dataDir, "/", filename, 
+

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: GeoEngine 18 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398774 )

Change subject: GeoEngine 18 Dec 2017
..


GeoEngine 18 Dec 2017

Change-Id: I2fbe58b2ef52f1d1ede0da3528600d34a06632dd
---
M WDCM_EngineGeo_goransm.R
1 file changed, 3 insertions(+), 2 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 4a04518..9f54564 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -219,7 +219,6 @@
 
 ### --- read item categories:
 setwd(itemsDir)
-
 idFiles <- list.files()
 idFiles <- idFiles[grepl(".csv$", idFiles)]
 categories <- unname(sapply(idFiles, function(x) {
@@ -342,7 +341,9 @@
   # - write: back to dataDir
   setwd(dataDir)
   file.remove(lF[i])
-  write.csv(usage, file = gsub(".tsv", ".csv", lF[i], fixed = T))
+  write.csv(usage, file = gsub(".tsv", ".csv", 
+   gsub("_ItemIDs", "", paste("wdcm_geoitem_", 
rF[i], sep = ""), fixed = T)
+   ))
 }
 
 ### --- log ETL step:

-- 
To view, visit https://gerrit.wikimedia.org/r/398774
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2fbe58b2ef52f1d1ede0da3528600d34a06632dd
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: GeoEngine 18 Dec 2017

2017-12-17 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398774 )

Change subject: GeoEngine 18 Dec 2017
..

GeoEngine 18 Dec 2017

Change-Id: I2fbe58b2ef52f1d1ede0da3528600d34a06632dd
---
M WDCM_EngineGeo_goransm.R
1 file changed, 3 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/74/398774/1

diff --git a/WDCM_EngineGeo_goransm.R b/WDCM_EngineGeo_goransm.R
index 4a04518..9f54564 100644
--- a/WDCM_EngineGeo_goransm.R
+++ b/WDCM_EngineGeo_goransm.R
@@ -219,7 +219,6 @@
 
 ### --- read item categories:
 setwd(itemsDir)
-
 idFiles <- list.files()
 idFiles <- idFiles[grepl(".csv$", idFiles)]
 categories <- unname(sapply(idFiles, function(x) {
@@ -342,7 +341,9 @@
   # - write: back to dataDir
   setwd(dataDir)
   file.remove(lF[i])
-  write.csv(usage, file = gsub(".tsv", ".csv", lF[i], fixed = T))
+  write.csv(usage, file = gsub(".tsv", ".csv", 
+   gsub("_ItemIDs", "", paste("wdcm_geoitem_", 
rF[i], sep = ""), fixed = T)
+   ))
 }
 
 ### --- log ETL step:

-- 
To view, visit https://gerrit.wikimedia.org/r/398774
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2fbe58b2ef52f1d1ede0da3528600d34a06632dd
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-ShinyServerFrontPage[master]: Init

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391013 )

Change subject: Init
..

Init

Change-Id: I3a0270fdc47379106fbdd36a67c543cb6ffaee41
---
A OverviewDashboard.png
A SemanticsDashboard.png
A UsageDashboard.png
A Wikidata-logo-en.png
A index.html
A wikitech.png
6 files changed, 296 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-ShinyServerFrontPage 
refs/changes/13/391013/1

diff --git a/OverviewDashboard.png b/OverviewDashboard.png
new file mode 100644
index 000..5ad8cda
--- /dev/null
+++ b/OverviewDashboard.png
Binary files differ
diff --git a/SemanticsDashboard.png b/SemanticsDashboard.png
new file mode 100644
index 000..159803b
--- /dev/null
+++ b/SemanticsDashboard.png
Binary files differ
diff --git a/UsageDashboard.png b/UsageDashboard.png
new file mode 100644
index 000..bce494b
--- /dev/null
+++ b/UsageDashboard.png
Binary files differ
diff --git a/Wikidata-logo-en.png b/Wikidata-logo-en.png
new file mode 100644
index 000..5e52bba
--- /dev/null
+++ b/Wikidata-logo-en.png
Binary files differ
diff --git a/index.html b/index.html
new file mode 100644
index 000..627138d
--- /dev/null
+++ b/index.html
@@ -0,0 +1,296 @@
+http://www.w3.org/1999/xhtml; lang="en-US">
+
+Wikidata Concepts Monitor (WDCM)
+
+body, html {
+margin: 0;
+padding: 0;
+font-family: Liberation Sans;
+background-color: ##FF;
+color: #00;
+}
+a {
+text-decoration: none;
+}
+a:hover {
+text-decoration: underline;
+}
+#titleBar {
+border-bottom: 1px solid #0d55a2;
+overflow: hidden;
+height: 80px;
+background-color: #0d55a2;
+}
+#titleBar #container {
+margin-top: 14px;
+}
+#titleBar h1 {
+margin: 0 auto 0.5em;
+padding: 0.2em;
+text-align: center;
+color: white;
+font-family: Liberation Sans;
+}
+#intro {
+border: 1px solid #cc;
+margin: 1em 1em 0;
+padding: 0.75em;
+background-color: #B5D7FC;
+text-align: center;
+font-size: 18px;
+}
+#intro p {
+margin: 0.3em 0;
+}
+#outer-content {
+max-width: 910px;
+margin-left: auto;
+margin-right: auto;
+}
+#content {
+margin: 1em auto;
+float: left;
+}
+#main {
+margin-right: 350px;
+float: left;
+line-height: 18px;
+}
+#shiny {
+border-left: 1px solid #aa;
+float: left;
+width: 305px;
+margin-left: -330px;
+padding-left: 20px;
+}
+#shiny iframe {
+margin-top: 30px;
+}
+.caption {
+font-size: 13px;
+}
+code {
+border: 1px solid #aa;
+padding: 0 0.5em;
+background-color: #e5e5e5;
+-moz-border-radius-topleft: 3px;
+-moz-border-radius-topright: 3px;
+-moz-border-radius-bottomright: 3px;
+-moz-border-radius-bottomleft: 3px;
+}
+
+.box{
+width:100%;
+border: 1px solid #cc;
+padding: 5px;
+}
+
+.blue{
+background:#B5D7FC;
+}
+
+
+
+
+
+
+
+
+Wikidata Concepts Monitor
+
+
+
+
+
+How is Wikidata used across the
+Wikimedia sister projects? 
+
+
+
+0. What is this?
+
+Wikidata Concepts Monitor (WDCM)
+is a system of dashboards that monitor the usage of https://www.wikidata.org/wiki/Wikidata:Main_Page; 
target="_blank">Wikidata
+items on WMF sister projects. The dashboards are currently supported by
+(1) analytical overviews of Wikidata item usage that are organized in a
+number of semantic categories, (2) per sister project analytical
+overviews of Wikidata item usage, and (3) https://en.wikipedia.org/wiki/Distributional_semantics;
+target="_blank">distributional semantics
+models of Wikidata usage that offer analytical insights into the
+structure of Wikidata item usage similarity across the sister projects
+and/or semantic categories of Wikidata items.
+
+
+In other
+words, here you can discover how
+much does a particular project use Wikidata, what
+semantic categories of Wikidata items are more popular in a particular
+project or a subset of projects,
+how similar are two or more
+projects in respect to the way they utilize Wikidata, what
+are the most popular Wikidata items in a particular project or a set of
+projects, and similar. 

+
+
+
+1. Getting started
+In order to be able to use the WDCM system in a way it was ment
+and designed to be used, i.e.
+with a clear understanding of what
+is it built for and why it
+was built that way,
+you probably need to get to learn about some important WDCM definitions
+(and the constraints that dictated them) first. You can do that by
+reading through the Definitions section of the WDCM Wikitech Technical
+Documentation .
+Do not panic, please: it is written in a language that a non-technical
+person who does not necessarily care about https://en.wikipedia.org/wiki/Data_science; target="_blank">Data
+Science or https://en.wikipedia.org/wiki/Cognitive_science; 
target="_blank">Cognitive
+Science can understand.
+
+Obviously, the current version of the WDCM system focuses on https://www.wikidata.org/wiki/Help:Items; target="_blank">Wikidata
+item usage.
+
+To start browsing the WDCM system,
+a list of currently available dashboards is provided on the

[MediaWiki-commits] [Gerrit] analytics...WDCM-Overview-Dashboard[master]: Initial

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391008 )

Change subject: Initial
..

Initial

Change-Id: I3cb15f8dc8fed47e571dc9884541af48683bb29a
---
A server.R
A ui.R
A www/Wikidata-logo-en.png
3 files changed, 774 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Overview-Dashboard 
refs/changes/08/391008/1

diff --git a/server.R b/server.R
new file mode 100644
index 000..86149ee
--- /dev/null
+++ b/server.R
@@ -0,0 +1,409 @@
+### ---
+### --- WDCM Dashboard Module, v. Beta 0.1
+### --- Script: server.R, v. Beta 0.1
+### ---
+
+### --- Setup
+rm(list = ls())
+### 
+### --- general
+library(shiny)
+library(RMySQL)
+library(data.table)
+library(DT)
+library(stringr)
+library(tidyr)
+library(dplyr)
+library(reshape2)
+### --- compute
+library(parallelDist)
+library(smacof)
+### --- visualization
+library(wordcloud)
+library(RColorBrewer)
+library(visNetwork)
+library(rbokeh)
+library(networkD3)
+library(ggplot2)
+library(ggrepel)
+library(scales)
+library(ggvis)
+
+### --- Server (Session) Scope
+### 
+
+### --- Credentials
+# setwd('/home/goransm/WMDE/WDCM/WDCM_RScripts/WDCM_Dashboard/aux')
+setwd('/srv/shiny-server/aux')
+
+mySQLCreds <- fread("mySQLCreds.csv", 
+header = T,
+drop = 1)
+
+### -- Connect
+con <- dbConnect(MySQL(), 
+ host = "tools.labsdb", 
+ defult.file = 
"/home/goransm/mySQL_Credentials/replica.my.cnf",
+ dbname = "u16664__wdcm_p",
+ user = mySQLCreds$user,
+ password = mySQLCreds$password)
+
+### --- list existing tables
+q <- "SHOW TABLES;"
+res <- dbSendQuery(con, q)
+st <- fetch(res, -1)
+dbClearResult(res)
+colnames(st) <- "tables"
+
+### --- SET CHARACTER SET utf8
+q <- "SET CHARACTER SET utf8;"
+res <- dbSendQuery(con, q)
+dbClearResult(res)
+
+### --- fetch wdcm2_project
+q <- "SELECT * FROM wdcm2_project;"
+res <- dbSendQuery(con, q)
+wdcmProject <- fetch(res, -1)
+dbClearResult(res) 
+
+### --- determine how many project types are present
+### --- and assign Brewer colors
+lengthProjectColor <- length(unique(wdcmProject$projectype))
+projectTypeColor <- brewer.pal(lengthProjectColor, "Set1")
+
+### --- fetch wdcm2_project_category_2dmap
+q <- "SELECT * FROM wdcm2_project_category_2dmap;"
+res <- dbSendQuery(con, q)
+wdcm2_project_category_2dmap <- fetch(res, -1)
+dbClearResult(res)
+colnames(wdcm2_project_category_2dmap)[3] <- "eu_project"
+wdcm2_project_category_2dmap <- left_join(wdcm2_project_category_2dmap,
+  wdcmProject,
+  by = "eu_project")
+labelSet <- unlist(lapply(unique(wdcm2_project_category_2dmap$projecttype), 
function(x){
+  w <- which(wdcm2_project_category_2dmap$projecttype %in% x)
+  lS <- arrange(wdcm2_project_category_2dmap[w, ], desc(eu_count))[1:5, ]
+  lS$eu_project 
+}))
+labelSetSmall1 <- 
unlist(lapply(unique(wdcm2_project_category_2dmap$projecttype), function(x){
+  w <- which(wdcm2_project_category_2dmap$projecttype %in% x)
+  lS <- arrange(wdcm2_project_category_2dmap[w, ], desc(eu_count))[1, ]
+  lS$eu_project 
+}))
+labelSetSmall3 <- 
unlist(lapply(unique(wdcm2_project_category_2dmap$projecttype), function(x){
+  w <- which(wdcm2_project_category_2dmap$projecttype %in% x)
+  lS <- arrange(wdcm2_project_category_2dmap[w, ], desc(eu_count))[1:3, ]
+  lS$eu_project 
+}))
+
+wdcm2_project_category_2dmapReduceLabels <- wdcm2_project_category_2dmap 
+wdcm2_project_category_2dmapReduceLabels$eu_project[which(!(wdcm2_project_category_2dmapReduceLabels$eu_project
 %in% labelSet))] <- ""
+colnames(wdcm2_project_category_2dmap)[c(3, 5,6)] <- c('Project', 'Usage', 
'Project Type')
+wdcm2_project_category_2dmap$projectTypeColor <- 
sapply(wdcm2_project_category_2dmap$`Project Type`, function(x) {
+  projectTypeColor[which(sort(unique(wdcm2_project_category_2dmap$`Project 
Type`)) %in% x)]
+})
+
+### --- fetch wdcm2_category
+q <- "SELECT * FROM wdcm2_category;"
+res <- dbSendQuery(con, q)
+wdcmCategory <- fetch(res, -1)
+dbClearResult(res) 
+
+### ---fetch wdcm2_category_project_2dmap
+q <- "SELECT * FROM wdcm2_category_project_2dmap;"
+res <- dbSendQuery(con, q)
+wdcm2_category_project_2dmap <- fetch(res, -1)
+dbClearResult(res) 
+wdcm2_category_project_2dmap <- left_join(wdcm2_category_project_2dmap,
+  wdcmCategory,
+  by = "category")
+colnames(wdcm2_category_project_2dmap)[3:4] <- c('Category', 'Usage')
+
+### --- fetch wdcm2_project_category
+q <- "SELECT * FROM wdcm2_project_category;"
+res <- dbSendQuery(con,

[MediaWiki-commits] [Gerrit] analytics...WDCM-Semantics-Dashboard[master]: Init

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391012 )

Change subject: Init
..

Init

Change-Id: I6f1438cebb9cc0bde9720581db99e462f7186d19
---
A data/wdcm2_projecttopic_ArchitecturalStructure.csv
A data/wdcm2_projecttopic_AstronomicalObject.csv
A data/wdcm2_projecttopic_Book.csv
A data/wdcm2_projecttopic_ChemicalEntities.csv
A data/wdcm2_projecttopic_Event.csv
A data/wdcm2_projecttopic_Gene.csv
A data/wdcm2_projecttopic_GeographicalObject.csv
A data/wdcm2_projecttopic_Human.csv
A data/wdcm2_projecttopic_Organization.csv
A data/wdcm2_projecttopic_ScientificArticle.csv
A data/wdcm2_projecttopic_Taxon.csv
A data/wdcm2_projecttopic_Thoroughfare.csv
A data/wdcm2_projecttopic_Wikimedia.csv
A data/wdcm2_projecttopic_WorkofArt.csv
A data/wdcm2_visNetworkEdges_project_ArchitecturalStructure.csv
A data/wdcm2_visNetworkEdges_project_AstronomicalObject.csv
A data/wdcm2_visNetworkEdges_project_Book.csv
A data/wdcm2_visNetworkEdges_project_ChemicalEntities.csv
A data/wdcm2_visNetworkEdges_project_Event.csv
A data/wdcm2_visNetworkEdges_project_Gene.csv
A data/wdcm2_visNetworkEdges_project_GeographicalObject.csv
A data/wdcm2_visNetworkEdges_project_Human.csv
A data/wdcm2_visNetworkEdges_project_Organization.csv
A data/wdcm2_visNetworkEdges_project_ScientificArticle.csv
A data/wdcm2_visNetworkEdges_project_Taxon.csv
A data/wdcm2_visNetworkEdges_project_Thoroughfare.csv
A data/wdcm2_visNetworkEdges_project_Wikimedia.csv
A data/wdcm2_visNetworkEdges_project_WorkofArt.csv
A data/wdcm2_visNetworkNodes_project_ArchitecturalStructure.csv
A data/wdcm2_visNetworkNodes_project_AstronomicalObject.csv
A data/wdcm2_visNetworkNodes_project_Book.csv
A data/wdcm2_visNetworkNodes_project_ChemicalEntities.csv
A data/wdcm2_visNetworkNodes_project_Event.csv
A data/wdcm2_visNetworkNodes_project_Gene.csv
A data/wdcm2_visNetworkNodes_project_GeographicalObject.csv
A data/wdcm2_visNetworkNodes_project_Human.csv
A data/wdcm2_visNetworkNodes_project_Organization.csv
A data/wdcm2_visNetworkNodes_project_ScientificArticle.csv
A data/wdcm2_visNetworkNodes_project_Taxon.csv
A data/wdcm2_visNetworkNodes_project_Thoroughfare.csv
A data/wdcm2_visNetworkNodes_project_Wikimedia.csv
A data/wdcm2_visNetworkNodes_project_WorkofArt.csv
A server.R
A ui.R
A www/Wikidata-logo-en.png
45 files changed, 16,775 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Semantics-Dashboard 
refs/changes/12/391012/1


-- 
To view, visit https://gerrit.wikimedia.org/r/391012
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6f1438cebb9cc0bde9720581db99e462f7186d19
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-Semantics-Dashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: _installProduction_analytics-wmde.R puppetization

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391071 )

Change subject: _installProduction_analytics-wmde.R puppetization
..

_installProduction_analytics-wmde.R puppetization

Change-Id: I82747d8fb4d9e3f658cd464692e44820ec2c85e4
---
A _installProduction_analytics-wmde.R
1 file changed, 15 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/71/391071/1

diff --git a/_installProduction_analytics-wmde.R 
b/_installProduction_analytics-wmde.R
new file mode 100644
index 000..1f54867
--- /dev/null
+++ b/_installProduction_analytics-wmde.R
@@ -0,0 +1,15 @@
+
+### --- Install WDCM R packages in production (currently: stat1005)
+### --- Hey hey: 
+### --- run as sudo -u analytics-wmde Rscript 
/srv/analytics-wmde/installRlib/_installProduction_analytics-wmde.R
+
+# - set proxy:
+Sys.setenv(
+  http_proxy = "http://webproxy.eqiad.wmnet:8080;,
+  https_proxy = "http://webproxy.eqiad.wmnet:8080;)
+
+# - install WDCM related packages:
+install.packages(c("dplyr", "httr", "stringr", "XML", "readr", 
+   "data.table", "tidyr", "maptpx", "Rtsne"),
+ lib = "/srv/analytics-wmde/r-library",
+ repos = c(CRAN = "https://www.stats.bris.ac.uk/R/;))
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/391071
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I82747d8fb4d9e3f658cd464692e44820ec2c85e4
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Structure change for production 13 Nov 2017

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391000 )

Change subject: Structure change for production 13 Nov 2017
..

Structure change for production 13 Nov 2017

Change-Id: I3bc40966b407b6efe1c8b68a949fa9a863de06c1
---
D WDCM_OverviewDashboard/server.R
D WDCM_OverviewDashboard/ui.R
D WDCM_OverviewDashboard/www/Wikidata-logo-en.png
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ArchitecturalStructure.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_AstronomicalObject.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Book.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ChemicalEntities.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Event.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Gene.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_GeographicalObject.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Human.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Organization.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ScientificArticle.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Taxon.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Thoroughfare.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Wikimedia.csv
D WDCM_SemanticsDashboard/data/wdcm2_projecttopic_WorkofArt.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ArchitecturalStructure.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_AstronomicalObject.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Book.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ChemicalEntities.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Event.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Gene.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_GeographicalObject.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Human.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Organization.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ScientificArticle.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Taxon.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Thoroughfare.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Wikimedia.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_WorkofArt.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ArchitecturalStructure.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_AstronomicalObject.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Book.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ChemicalEntities.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Event.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Gene.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_GeographicalObject.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Human.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Organization.csv
D 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ScientificArticle.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Taxon.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Thoroughfare.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Wikimedia.csv
D WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_WorkofArt.csv
D WDCM_SemanticsDashboard/server.R
D WDCM_SemanticsDashboard/ui.R
D WDCM_SemanticsDashboard/www/Wikidata-logo-en.png
D WDCM_ShinyServerFrontPage/OverviewDashboard.png
D WDCM_ShinyServerFrontPage/SemanticsDashboard.png
D WDCM_ShinyServerFrontPage/UsageDashboard.png
D WDCM_ShinyServerFrontPage/WDCM_Design.png
D WDCM_ShinyServerFrontPage/Wikidata-logo-en.png
D WDCM_ShinyServerFrontPage/index.html
D WDCM_ShinyServerFrontPage/wikitech.png
D WDCM_UsageDashboard/server.R
D WDCM_UsageDashboard/ui.R
D WDCM_UsageDashboard/www/Wikidata-logo-en.png
58 files changed, 0 insertions(+), 19,399 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/00/391000/1


-- 
To view, visit https://gerrit.wikimedia.org/r/391000
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3bc40966b407b6efe1c8b68a949fa9a863de06c1
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-Usage-Dashboard[master]: Init

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391010 )

Change subject: Init
..

Init

Change-Id: Ifa894d8effc9cf713bf7b36bd23e46f8e7a7b48c
---
A server.R
A ui.R
A www/Wikidata-logo-en.png
3 files changed, 1,554 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-Usage-Dashboard 
refs/changes/10/391010/1

diff --git a/server.R b/server.R
new file mode 100644
index 000..5bdfdd6
--- /dev/null
+++ b/server.R
@@ -0,0 +1,999 @@
+### ---
+### --- WDCM Usage Dashboard, v. Beta 0.1
+### --- Script: server.R, v. Beta 0.1
+### ---
+
+### --- Setup
+rm(list = ls())
+### 
+### --- general
+library(shiny)
+library(shinydashboard)
+library(RMySQL)
+library(data.table)
+library(DT)
+library(stringr)
+library(tidyr)
+library(dplyr)
+library(reshape2)
+### --- compute
+library(parallelDist)
+### --- visualization
+library(RColorBrewer)
+library(visNetwork)
+library(networkD3)
+library(ggplot2)
+library(ggrepel)
+library(scales)
+
+### --- Server (Session) Scope
+### 
+
+### --- Credentials
+# setwd('/home/goransm/WMDE/WDCM/WDCM_RScripts/WDCM_Dashboard/aux')
+setwd('/srv/shiny-server/aux')
+
+mySQLCreds <- fread("mySQLCreds.csv", 
+header = T,
+drop = 1)
+
+### -- Connect
+con <- dbConnect(MySQL(), 
+ host = "tools.labsdb", 
+ defult.file = 
"/home/goransm/mySQL_Credentials/replica.my.cnf",
+ dbname = "u16664__wdcm_p",
+ user = mySQLCreds$user,
+ password = mySQLCreds$password)
+
+### --- list existing tables
+q <- "SHOW TABLES;"
+res <- dbSendQuery(con, q)
+st <- fetch(res, -1)
+dbClearResult(res)
+colnames(st) <- "tables"
+
+### --- SET CHARACTER SET utf8
+q <- "SET CHARACTER SET utf8;"
+res <- dbSendQuery(con, q)
+dbClearResult(res)
+
+### --- fetch wdcm2_project
+q <- "SELECT * FROM wdcm2_project;"
+res <- dbSendQuery(con, q)
+wdcmProject <- fetch(res, -1)
+dbClearResult(res)
+colnames(wdcmProject) <- c('Project', 'Usage', 'Project Type')
+
+### --- fetch wdcm2_project_category
+q <- "SELECT * FROM wdcm2_project_category;"
+res <- dbSendQuery(con, q)
+wdcmProjectCategory <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmProjectCategory) <- c('Project', 'Category', 'Usage', 'Project 
Type')
+
+### --- fetch wdcm2_project_item100
+q <- "SELECT * FROM wdcm2_project_item100;"
+res <- dbSendQuery(con, q)
+wdcmProjectItem100 <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmProjectItem100) <- c('Project', 'EntityID', 'Usage', 'Project 
Type', 'Label')
+
+### --- fetch wdcm2_project_category_item100
+q <- "SELECT * FROM wdcm2_project_category_item100;"
+res <- dbSendQuery(con, q)
+wdcmProjectCategoryItem100 <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmProjectCategoryItem100) <- c('Project', 'Category', 'EntityID', 
'Usage', 'Project Type', 'Label')
+
+### --- fetch wdcm2_category
+q <- "SELECT * FROM wdcm2_category;"
+res <- dbSendQuery(con, q)
+wdcmCategory <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmCategory) <- c('Category', 'Usage')
+
+### --- fetch wdcm2_category_item100
+q <- "SELECT * FROM wdcm2_category_item100;"
+res <- dbSendQuery(con, q)
+wdcmCategoryItem100 <- fetch(res, -1)
+dbClearResult(res) 
+colnames(wdcmCategoryItem100) <- c('EntityID', 'Usage', 'Category', 'Label')
+
+### --- Disconnect
+dbDisconnect(con)
+
+### --- Compute per `Project Type` tables
+# - wdcmProjectType
+wdcmProjectType <- wdcmProject %>% 
+  group_by(`Project Type`) %>% 
+  summarise(Usage = sum(Usage)) %>% 
+  arrange(desc(Usage))
+# - wdcmProjectTypeCategory
+wdcmProjectTypeCategory <- wdcmProjectCategory %>% 
+  group_by(`Project Type`, Category) %>% 
+  summarise(Usage = sum(Usage)) %>% 
+  arrange(desc(Usage))
+# - wdcmProjectTypeItem100
+wdcmProjectTypeItem100 <- wdcmProjectItem100 %>% 
+  select(`Project Type`, EntityID, Label, Usage) %>% 
+  group_by(`Project Type`, EntityID, Label) %>% 
+  summarise(Usage = sum(Usage)) %>% 
+  arrange(`Project Type`, desc(Usage))
+
+### --- Compute project similarity structure
+projectSimilarity <- wdcmProjectCategory %>% 
+  dplyr::select(Project, Category, Usage) %>% 
+  tidyr::spread(key = Category,
+ value = Usage,
+ fill = 0)
+projectNames <- projectSimilarity$Project
+projectSimilarity$Project <- NULL
+# - normalize:
+projectSimilarity <- t(apply(projectSimilarity, 1, function(x) {x/sum(x)}))
+# projectSimilarity[projectSimilarity > 0] <- 1
+projectSimilarity <- as.matrix(parDist(as.matrix(projectSimilarity), method = 
"kullback"))
+rownames(projectSimilarity) <- projectNames
+colnames(projectSimilarity) <- projectNames
+
+### - Determine Constants
+# - determine Projects

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Production - stat1005 structure

2017-11-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/392209 )

Change subject: Production - stat1005 structure
..

Production - stat1005 structure

Change-Id: I0bf98633c0ab722761a064e5eebee9fb1fb5511e
---
A WDCM_Engine.R
A WDCM_Ontology/WDCM_Ontology_Berlin_05032017.csv
M _installProduction_analytics-wmde.R
3 files changed, 221 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/09/392209/1

diff --git a/WDCM_Engine.R b/WDCM_Engine.R
new file mode 100644
index 000..39a6330
--- /dev/null
+++ b/WDCM_Engine.R
@@ -0,0 +1,206 @@
+### ---
+### --- WDCM Engine, v. Beta 0.1
+### --- Script: WDCM_Engine.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data 
Kolektiv 
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Engine.R unifies the previous
+### --- four WDCM Engine scripts:
+### --- WDCM_Collect_Items.R
+### --- WDCM_Sqoop_Clients.R
+### --- WDCM_Search_Clients.R
+### --- WDCM_Pre-Process.R
+### --- each section in WDCM_Engine.R provides additional explanation.
+### --- NOTE: WDCM_Engine.R is exclusively the only WDCM R script
+### --- that is run in production from stat1005 (currently)
+### --- to produce the WDCM update
+### ---
+### --- LICENSE:
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### --- 
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### --- 
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of
+### --- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+### --- GNU General Public License for more details.
+### --- 
+### --- You should have received a copy of the GNU General Public License
+### --- along with WDCM. If not, see .
+### ---
+
+### ---
+### --- Script 1: WDCM_Collect_Items.R, WDCM Search Module
+### ---
+
+### ---
+### --- WDCM Search Module, v. Beta 0.1
+### --- Script: WDCM_Collect_Items.R, v. Beta 0.1
+### --- Author: Goran S. Milovanovic, Data Analyst, WMDE
+### --- Developed under the contract between Goran Milovanovic PR Data 
Kolektiv 
+### --- and WMDE.
+### --- Contact: goran.milovanovic_...@wikimedia.de
+### ---
+### --- DESCRIPTION:
+### --- WDCM_Collect_Items.R takes a list of concepts (categories)
+### --- defined by a given WDCM Ontology (human input) and then 
+### --- contacts the SPARQL endpoint to fetch all relevant item IDs.
+### ---
+### --- INPUT: 
+### --- the WDCM_Collect_Items.R reads the WDCM Ontology file (csv)
+### --- from /home/goransm/WMDE/WDCM/WDCM_DataIN/WDCM_Ontology
+### --- on wikidataconcepts.wmflabs.org
+### ---
+### --- OUTPUT: 
+### --- Results are stored locally as .csv files on the wikidataconcepts Labs 
instance:
+### --- wikidataconcepts.wmflabs.org
+### --- in: /home/goransm/WMDE/WDCM/WDCM_DataOUT
+### --- These output .csv files migrate to production (stat1005.eqiad.wmnet, 
currently):
+### --- where they are then further processed by the WDCM Search Module 
(running: 
+### --- WDCM_Search_Clients.R)
+### ---
+
+### ---
+### --- GPL v2
+### --- This file is part of Wikidata Concepts Monitor (WDCM)
+### --- 
+### --- WDCM is free software: you can redistribute it and/or modify
+### --- it under the terms of the GNU General Public License as published by
+### --- the Free Software Foundation, either version 2 of the License, or
+### --- (at your option) any later version.
+### --- 
+### --- WDCM is distributed in the hope that it will be useful,
+### --- but WITHOUT ANY WARRANTY; without even the implied warranty of

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor nov 18 2017

2017-11-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/392211 )

Change subject: minor nov 18 2017
..

minor nov 18 2017

Change-Id: Icccac506b7e07cc6015a59ffddfb64b9990ef80e
---
D WDCM_Ontology/WDCM_Ontology_Berlin_05032017.csv
1 file changed, 0 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/11/392211/1

diff --git a/WDCM_Ontology/WDCM_Ontology_Berlin_05032017.csv 
b/WDCM_Ontology/WDCM_Ontology_Berlin_05032017.csv
deleted file mode 100644
index 5c44db0..000
--- a/WDCM_Ontology/WDCM_Ontology_Berlin_05032017.csv
+++ /dev/null
@@ -1,15 +0,0 @@
-Category,CategoryItems,WikidataDescription
-Human,Q5,human
-Wikimedia_Internal,"Q4167836, Q4167410, Q11266439","Wikimedia category, 
Wikimedia disambiguation page, wikimedia template"
-Work Of Art,Q838948,work of art
-Scientific Article,Q13442814,scientific article
-Book,Q571,book
-Geographical Object,Q618123,geographical object
-Organization,"Q783794, Q5971060, Q43229","company, club, organization"
-Architectural Structure,"Q4989906, Q41176","monument, building"
-Gene,Q7187,gene
-Chemical Entities,"Q11344, Q11173, Q79529","chemical element, chemical 
compound, chemical substance"
-Astronomical Object,Q6999,astronomical object
-Taxon,Q16521,taxon
-Event,Q1656682,event
-Thoroughfare,Q83620,thoroughfare

-- 
To view, visit https://gerrit.wikimedia.org/r/392211
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icccac506b7e07cc6015a59ffddfb64b9990ef80e
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: WDCM_Engine.R

2017-11-18 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/392212 )

Change subject: WDCM_Engine.R
..

WDCM_Engine.R

Change-Id: I3e3d7bf9abe23fcd003958f689442c966cef9aa2
---
M WDCM_Engine.R
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/12/392212/1

diff --git a/WDCM_Engine.R b/WDCM_Engine.R
index 39a6330..8a936a4 100644
--- a/WDCM_Engine.R
+++ b/WDCM_Engine.R
@@ -198,7 +198,7 @@
   
 }
 
-# - log uncompleted queries:
+# - log uncompleted queries;
 # - set log dir:
 outDir <- './WDCM_Logs'
 setwd(outDir)

-- 
To view, visit https://gerrit.wikimedia.org/r/392212
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3e3d7bf9abe23fcd003958f689442c966cef9aa2
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: install script knows where it lives

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391080 )

Change subject: install script knows where it lives
..

install script knows where it lives

Change-Id: I79774f46487619763089a163c3d0c0684e743ebd
---
M _installProduction_analytics-wmde.R
1 file changed, 15 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/80/391080/1

diff --git a/_installProduction_analytics-wmde.R 
b/_installProduction_analytics-wmde.R
index 1f54867..8a020db 100644
--- a/_installProduction_analytics-wmde.R
+++ b/_installProduction_analytics-wmde.R
@@ -3,13 +3,26 @@
 ### --- Hey hey: 
 ### --- run as sudo -u analytics-wmde Rscript 
/srv/analytics-wmde/installRlib/_installProduction_analytics-wmde.R
 
+# - setup
+### --- NOTE: {kimisc} is the only WDCM relevant R package
+### --- that needs to be installed manually
+
 # - set proxy:
 Sys.setenv(
   http_proxy = "http://webproxy.eqiad.wmnet:8080;,
   https_proxy = "http://webproxy.eqiad.wmnet:8080;)
 
+fPath <- as.character(commandArgs(trailingOnly = FALSE)[4])
+fPath <- gsub("--file=", "", fPath, fixed = T)
+fPath <- unlist(strsplit(fPath, split = "/", fixed = T))
+fPath <- paste(
+  paste(fPath[1:length(fPath) - 1], collapse = "/"),
+  "/",
+  sep = "")
+
 # - install WDCM related packages:
 install.packages(c("dplyr", "httr", "stringr", "XML", "readr", 
"data.table", "tidyr", "maptpx", "Rtsne"),
- lib = "/srv/analytics-wmde/r-library",
- repos = c(CRAN = "https://www.stats.bris.ac.uk/R/;))
\ No newline at end of file
+ lib = paste(fPath, "r-library", sep = ""),
+ repos = c(CRAN = "https://www.stats.bris.ac.uk/R/;))
+

-- 
To view, visit https://gerrit.wikimedia.org/r/391080
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I79774f46487619763089a163c3d0c0684e743ebd
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391083 )

Change subject: minor
..

minor

Change-Id: I332e7e4f7213794b1075d412adf1de28c53e4b9b
---
M _installProduction_analytics-wmde.R
1 file changed, 0 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/83/391083/1

diff --git a/_installProduction_analytics-wmde.R 
b/_installProduction_analytics-wmde.R
index 8a020db..aab4cb0 100644
--- a/_installProduction_analytics-wmde.R
+++ b/_installProduction_analytics-wmde.R
@@ -3,10 +3,6 @@
 ### --- Hey hey: 
 ### --- run as sudo -u analytics-wmde Rscript 
/srv/analytics-wmde/installRlib/_installProduction_analytics-wmde.R
 
-# - setup
-### --- NOTE: {kimisc} is the only WDCM relevant R package
-### --- that needs to be installed manually
-
 # - set proxy:
 Sys.setenv(
   http_proxy = "http://webproxy.eqiad.wmnet:8080;,

-- 
To view, visit https://gerrit.wikimedia.org/r/391083
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I332e7e4f7213794b1075d412adf1de28c53e4b9b
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Installation script mkdirs r-library

2017-11-13 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391086 )

Change subject: Installation script mkdirs r-library
..

Installation script mkdirs r-library

Change-Id: I23aec808dc0d1eee4beb9085c7099a218f4035b1
---
M _installProduction_analytics-wmde.R
1 file changed, 13 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/86/391086/1

diff --git a/_installProduction_analytics-wmde.R 
b/_installProduction_analytics-wmde.R
index aab4cb0..d973370 100644
--- a/_installProduction_analytics-wmde.R
+++ b/_installProduction_analytics-wmde.R
@@ -8,6 +8,8 @@
   http_proxy = "http://webproxy.eqiad.wmnet:8080;,
   https_proxy = "http://webproxy.eqiad.wmnet:8080;)
 
+
+# - fPath: where the scripts is run from?
 fPath <- as.character(commandArgs(trailingOnly = FALSE)[4])
 fPath <- gsub("--file=", "", fPath, fixed = T)
 fPath <- unlist(strsplit(fPath, split = "/", fixed = T))
@@ -16,6 +18,17 @@
   "/",
   sep = "")
 
+
+# - find out whether the fPath/r-library directory exists
+# - YES: delete it and mkdir, NO: mkdir only
+if (dir.exists(paths = paste(fPath, "r-library", sep = ""))) {
+  unlink(x = paste(fPath, "r-library", sep = ""), 
+ recursive = T)
+  dir.create(path = paste(fPath, "r-library", sep = ""))
+} else {
+  dir.create(path = paste(fPath, "r-library", sep = ""))
+}
+
 # - install WDCM related packages:
 install.packages(c("dplyr", "httr", "stringr", "XML", "readr", 
"data.table", "tidyr", "maptpx", "Rtsne"),

-- 
To view, visit https://gerrit.wikimedia.org/r/391086
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I23aec808dc0d1eee4beb9085c7099a218f4035b1
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor Shiny portal change

2017-10-30 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/387212 )

Change subject: minor Shiny portal change
..

minor Shiny portal change

Change-Id: I604f060633bd94a38463e40231144d5b3bbeba1d
---
R WDCM_ShinyServerFrontPage/index.html
1 file changed, 3 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/12/387212/1

diff --git a/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html 
b/WDCM_ShinyServerFrontPage/index.html
similarity index 97%
rename from WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
rename to WDCM_ShinyServerFrontPage/index.html
index 4a82be8..627138d 100644
--- a/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
+++ b/WDCM_ShinyServerFrontPage/index.html
@@ -190,7 +190,9 @@
  previous experiences in managing Shiny Dashboards on behalf of
  https://wikimediafoundation.org/wiki/User:MPopov_(WMF)" target = 
"_blank">Mikhail Popov and  the team that built our
  https://discovery.wmflabs.org/; target = "_blank">Discovery 
Dashboards
- were very helpful in the development of the WDCM Dashboards.
+ were very helpful in the development of the WDCM Dashboards. Of course, 
enlightening
+ discussions with https://meta.wikimedia.org/wiki/User:Halfak_(WMF)" target = "_blank">Aaron 
Halfaker,
+   Research Scientist, Wikimedia Foundation, and his team.
 
 3. How does it work?
 

-- 
To view, visit https://gerrit.wikimedia.org/r/387212
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I604f060633bd94a38463e40231144d5b3bbeba1d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor Shiny portal change

2017-10-30 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/387212 )

Change subject: minor Shiny portal change
..


minor Shiny portal change

Change-Id: I604f060633bd94a38463e40231144d5b3bbeba1d
---
R WDCM_ShinyServerFrontPage/index.html
1 file changed, 3 insertions(+), 1 deletion(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html 
b/WDCM_ShinyServerFrontPage/index.html
similarity index 97%
rename from WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
rename to WDCM_ShinyServerFrontPage/index.html
index 4a82be8..627138d 100644
--- a/WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
+++ b/WDCM_ShinyServerFrontPage/index.html
@@ -190,7 +190,9 @@
  previous experiences in managing Shiny Dashboards on behalf of
  https://wikimediafoundation.org/wiki/User:MPopov_(WMF)" target = 
"_blank">Mikhail Popov and  the team that built our
  https://discovery.wmflabs.org/; target = "_blank">Discovery 
Dashboards
- were very helpful in the development of the WDCM Dashboards.
+ were very helpful in the development of the WDCM Dashboards. Of course, 
enlightening
+ discussions with https://meta.wikimedia.org/wiki/User:Halfak_(WMF)" target = "_blank">Aaron 
Halfaker,
+   Research Scientist, Wikimedia Foundation, and his team.
 
 3. How does it work?
 

-- 
To view, visit https://gerrit.wikimedia.org/r/387212
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I604f060633bd94a38463e40231144d5b3bbeba1d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor deletion

2017-10-30 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/387213 )

Change subject: minor deletion
..

minor deletion

Change-Id: Ibd45c1702822ca8b7d410c522cec1d49b23c78c7
---
D WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
1 file changed, 0 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/13/387213/1

diff --git a/WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt 
b/WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
deleted file mode 100644
index 46387b0..000
--- a/WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
+++ /dev/null
Binary files differ

-- 
To view, visit https://gerrit.wikimedia.org/r/387213
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibd45c1702822ca8b7d410c522cec1d49b23c78c7
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor deletion

2017-10-30 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/387213 )

Change subject: minor deletion
..


minor deletion

Change-Id: Ibd45c1702822ca8b7d410c522cec1d49b23c78c7
---
D WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
1 file changed, 0 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt 
b/WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
deleted file mode 100644
index 46387b0..000
--- a/WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
+++ /dev/null
Binary files differ

-- 
To view, visit https://gerrit.wikimedia.org/r/387213
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibd45c1702822ca8b7d410c522cec1d49b23c78c7
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Semantics - t-SNE Maps

2017-10-24 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/386323 )

Change subject: Semantics - t-SNE Maps
..


Semantics - t-SNE Maps

Change-Id: I9d3c5ec541ceb4fd7041dfed75907b5de8398843
---
M WDCM_OverviewDashboard/ui.R
M WDCM_SemanticsDashboard/server.R
M WDCM_SemanticsDashboard/ui.R
M WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
M WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
7 files changed, 277 insertions(+), 182 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_OverviewDashboard/ui.R b/WDCM_OverviewDashboard/ui.R
index 25b0bdb..7ae997d 100644
--- a/WDCM_OverviewDashboard/ui.R
+++ b/WDCM_OverviewDashboard/ui.R
@@ -4,7 +4,7 @@
 ### ---
 
 ### --- Setup
-rm(list=ls())
+rm(list = ls())
 ### --- general
 library(shiny)
 library(shinydashboard)
@@ -238,7 +238,9 @@

N.B. The current Wikidata item usage statistic definition is 
the count of the number of pages in a particular client project
where the respective 
Wikidata item is used. Thus, the current definition ignores the usage 
aspects completely. This definition is motivated by the currently 
-   present constraints in 
Wikidata usage tracking across the client projects. With more mature Wikidata 
usage tracking systems, the definition will become a subject 
+   present constraints in 
Wikidata usage tracking across the client projects 
+   (see https://www.mediawiki.org/wiki/Wikibase/Schema/wbc_entity_usage; target = 
"_blank">Wikibase/Schema/wbc entity usage). 
+   With more mature Wikidata 
usage tracking systems, the definition will become a subject 
of change. The term 
Wikidata usage volume is reserved for total Wikidata usage (i.e. the sum 
of usage statistics) in a particular 
client project, group of 
client projects, or semantic categories. By a Wikidata semantic category 
we mean a selection of Wikidata items that is 
that is operationally 
defined by a respective SPARQL query returning a selection of items that 
intuitivelly match a human, natural semantic category. 
@@ -247,7 +249,10 @@
categories in WDCM is not 
necessarily exhaustive (i.e. they do not necessarily cover all Wikidata items), 
neither the categories are necessarily 
mutually exclusive. The 
Wikidata ontology is very complex and a product of work of many people, so 
there is an optimization price to be paid in every attempt to 
adapt or simplify its 
present structure to the needs of a statistical analytical system such as WDCM. 
The current set of WDCM semantic categories is thus not 
-   normative in any sense and 
a subject  of change in any moment, depending upon the analytical needs of the 
community.
+   normative in any sense and 
can become a subject of change in any moment, depending upon the analytical 
needs of the community.
+   The 
currently used WDCM Taxonomy of Wikidata items encompasses the following 
14 semantic categories: Geographical Object, Organization, 
Architectural Structure, 
+   Human, 
Wikimedia, Work of Art, Book, Gene, Scientific 
Article, Chemical Entities, Astronomical Object, 
Thoroughfare, Event, 
+   and Taxon.

Wikidata Usage 
Overview

@@ -258,10 +263,10 @@
of the client project 
pairwise Euclidean distances derived from the Projects x Categories contingency 
table. Given that the original higher-dimensional space 
from which the 2D map is 
derived is rather constrained by the choice of a small number of semantic 
categories, the similarity mapping is somewhat 
imprecise and should be 
taken as an attempt at an approximate big picture of the

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Semantics - t-SNE Maps

2017-10-24 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/386323 )

Change subject: Semantics - t-SNE Maps
..

Semantics - t-SNE Maps

Change-Id: I9d3c5ec541ceb4fd7041dfed75907b5de8398843
---
M WDCM_OverviewDashboard/ui.R
M WDCM_SemanticsDashboard/server.R
M WDCM_SemanticsDashboard/ui.R
M WDCM_ShinyServerFrontPage/wdcm_ShinyFront.html
M WDCM_TechDocumentation/WikidataConcepts_TechDocumentation.odt
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
7 files changed, 277 insertions(+), 182 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/23/386323/1

diff --git a/WDCM_OverviewDashboard/ui.R b/WDCM_OverviewDashboard/ui.R
index 25b0bdb..7ae997d 100644
--- a/WDCM_OverviewDashboard/ui.R
+++ b/WDCM_OverviewDashboard/ui.R
@@ -4,7 +4,7 @@
 ### ---
 
 ### --- Setup
-rm(list=ls())
+rm(list = ls())
 ### --- general
 library(shiny)
 library(shinydashboard)
@@ -238,7 +238,9 @@

N.B. The current Wikidata item usage statistic definition is 
the count of the number of pages in a particular client project
where the respective 
Wikidata item is used. Thus, the current definition ignores the usage 
aspects completely. This definition is motivated by the currently 
-   present constraints in 
Wikidata usage tracking across the client projects. With more mature Wikidata 
usage tracking systems, the definition will become a subject 
+   present constraints in 
Wikidata usage tracking across the client projects 
+   (see https://www.mediawiki.org/wiki/Wikibase/Schema/wbc_entity_usage; target = 
"_blank">Wikibase/Schema/wbc entity usage). 
+   With more mature Wikidata 
usage tracking systems, the definition will become a subject 
of change. The term 
Wikidata usage volume is reserved for total Wikidata usage (i.e. the sum 
of usage statistics) in a particular 
client project, group of 
client projects, or semantic categories. By a Wikidata semantic category 
we mean a selection of Wikidata items that is 
that is operationally 
defined by a respective SPARQL query returning a selection of items that 
intuitivelly match a human, natural semantic category. 
@@ -247,7 +249,10 @@
categories in WDCM is not 
necessarily exhaustive (i.e. they do not necessarily cover all Wikidata items), 
neither the categories are necessarily 
mutually exclusive. The 
Wikidata ontology is very complex and a product of work of many people, so 
there is an optimization price to be paid in every attempt to 
adapt or simplify its 
present structure to the needs of a statistical analytical system such as WDCM. 
The current set of WDCM semantic categories is thus not 
-   normative in any sense and 
a subject  of change in any moment, depending upon the analytical needs of the 
community.
+   normative in any sense and 
can become a subject of change in any moment, depending upon the analytical 
needs of the community.
+   The 
currently used WDCM Taxonomy of Wikidata items encompasses the following 
14 semantic categories: Geographical Object, Organization, 
Architectural Structure, 
+   Human, 
Wikimedia, Work of Art, Book, Gene, Scientific 
Article, Chemical Entities, Astronomical Object, 
Thoroughfare, Event, 
+   and Taxon.

Wikidata Usage 
Overview

@@ -258,10 +263,10 @@
of the client project 
pairwise Euclidean distances derived from the Projects x Categories contingency 
table. Given that the original higher-dimensional space 
from which the 2D map is 
derived is rather constrained by the choice of a small number of semantic 
categories, the similarity mapping is somewhat 
imprecise and should be 
taken as an attempt at an approximate big

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: WDCM_SemanticsDashboard - Skeleton 21 Oct 2017

2017-10-20 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385486 )

Change subject: WDCM_SemanticsDashboard - Skeleton 21 Oct 2017
..

WDCM_SemanticsDashboard - Skeleton 21 Oct 2017

Change-Id: Ifab45d2c35231a4d53ce6e1ccba843db7aa5978a
---
A WDCM_SemanticsDashboard/server.R
A WDCM_SemanticsDashboard/ui.R
2 files changed, 50 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/86/385486/1

diff --git a/WDCM_SemanticsDashboard/server.R b/WDCM_SemanticsDashboard/server.R
new file mode 100644
index 000..da51dec
--- /dev/null
+++ b/WDCM_SemanticsDashboard/server.R
@@ -0,0 +1,30 @@
+### ---
+### --- WDCM Semantics Dashboard, v. Beta 0.1
+### --- Script: server.R, v. Beta 0.1
+### ---
+
+### --- Setup
+rm(list = ls())
+### 
+### --- general
+library(shiny)
+library(shinydashboard)
+library(RMySQL)
+library(data.table)
+library(DT)
+library(stringr)
+library(tidyr)
+library(dplyr)
+library(reshape2)
+### --- compute
+library(parallelDist)
+### --- visualization
+library(RColorBrewer)
+library(visNetwork)
+library(networkD3)
+library(ggplot2)
+library(ggrepel)
+library(scales)
+
+### --- Server (Session) Scope
+### 
\ No newline at end of file
diff --git a/WDCM_SemanticsDashboard/ui.R b/WDCM_SemanticsDashboard/ui.R
new file mode 100644
index 000..4b80f58
--- /dev/null
+++ b/WDCM_SemanticsDashboard/ui.R
@@ -0,0 +1,20 @@
+### ---
+### --- WDCM Semantics Dashboard, v. Beta 0.1
+### --- Script: ui.R, v. Beta 0.1
+### ---
+
+### --- Setup
+rm(list = ls())
+### --- general
+library(shiny)
+library(shinydashboard)
+library(shinycssloaders)
+### --- outputs
+library(visNetwork)
+library(rbokeh)
+library(networkD3)
+library(ggvis)
+library(DT)
+
+# - options
+options(warn = -1)
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/385486
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifab45d2c35231a4d53ce6e1ccba843db7aa5978a
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: rm - minor

2017-10-21 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385807 )

Change subject: rm - minor
..


rm - minor

Change-Id: I68976257eed326eec9f36ef00b426f902b030b2f
---
D creds.csv
1 file changed, 0 insertions(+), 2 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/creds.csv b/creds.csv
deleted file mode 100644
index 7f4f901..000
--- a/creds.csv
+++ /dev/null
@@ -1,2 +0,0 @@
-username,password
-research,Kfj%5M$d134DEwxa6098!!33

-- 
To view, visit https://gerrit.wikimedia.org/r/385807
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I68976257eed326eec9f36ef00b426f902b030b2f
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: fix DT width

2017-10-21 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385808 )

Change subject: fix DT width
..

fix DT width

Change-Id: I6184b4eaae1cf89fdf2621109e70b2c13ed4701d
---
M WDCM_OverviewDashboard/server.R
M WDCM_UsageDashboard/server.R
2 files changed, 14 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/08/385808/1

diff --git a/WDCM_OverviewDashboard/server.R b/WDCM_OverviewDashboard/server.R
index fd1d5fe..21afbfe 100644
--- a/WDCM_OverviewDashboard/server.R
+++ b/WDCM_OverviewDashboard/server.R
@@ -372,10 +372,10 @@
 datatable(wdcmProjectCategory,
 options = list(
   pageLength = 20,
-  autoWidth = TRUE,
+  width = '100%',
   columnDefs = list(list(className = 'dt-center', targets = "_all"))
   ),
-rownames= FALSE
+rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -389,10 +389,10 @@
 datatable(dataSet,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
index 8cded77..be1df87 100644
--- a/WDCM_UsageDashboard/server.R
+++ b/WDCM_UsageDashboard/server.R
@@ -915,10 +915,10 @@
 datatable(wdcmProject,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -930,10 +930,10 @@
 datatable(wdcmCategory,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -945,10 +945,10 @@
 datatable(wdcmProjectCategory,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -960,10 +960,10 @@
 datatable(wdcmProjectType,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -975,10 +975,10 @@
 datatable(wdcmProjectTypeCategory,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,

-- 
To view, visit https://gerrit.wikimedia.org/r/385808
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6184b4eaae1cf89fdf2621109e70b2c13ed4701d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: fix DT width

2017-10-21 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385808 )

Change subject: fix DT width
..


fix DT width

Change-Id: I6184b4eaae1cf89fdf2621109e70b2c13ed4701d
---
M WDCM_OverviewDashboard/server.R
M WDCM_UsageDashboard/server.R
2 files changed, 14 insertions(+), 14 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_OverviewDashboard/server.R b/WDCM_OverviewDashboard/server.R
index fd1d5fe..21afbfe 100644
--- a/WDCM_OverviewDashboard/server.R
+++ b/WDCM_OverviewDashboard/server.R
@@ -372,10 +372,10 @@
 datatable(wdcmProjectCategory,
 options = list(
   pageLength = 20,
-  autoWidth = TRUE,
+  width = '100%',
   columnDefs = list(list(className = 'dt-center', targets = "_all"))
   ),
-rownames= FALSE
+rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -389,10 +389,10 @@
 datatable(dataSet,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
index 8cded77..be1df87 100644
--- a/WDCM_UsageDashboard/server.R
+++ b/WDCM_UsageDashboard/server.R
@@ -915,10 +915,10 @@
 datatable(wdcmProject,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -930,10 +930,10 @@
 datatable(wdcmCategory,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -945,10 +945,10 @@
 datatable(wdcmProjectCategory,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -960,10 +960,10 @@
 datatable(wdcmProjectType,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,
@@ -975,10 +975,10 @@
 datatable(wdcmProjectTypeCategory,
   options = list(
 pageLength = 20,
-autoWidth = TRUE,
+width = '100%',
 columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
   ),
-  rownames= FALSE
+  rownames = FALSE
 )
   }) %>% withProgress(message = 'Generating data',
   min = 0,

-- 
To view, visit https://gerrit.wikimedia.org/r/385808
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6184b4eaae1cf89fdf2621109e70b2c13ed4701d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: rm - minor

2017-10-21 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385807 )

Change subject: rm - minor
..

rm - minor

Change-Id: I68976257eed326eec9f36ef00b426f902b030b2f
---
D creds.csv
1 file changed, 0 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/07/385807/1

diff --git a/creds.csv b/creds.csv
deleted file mode 100644
index 7f4f901..000
--- a/creds.csv
+++ /dev/null
@@ -1,2 +0,0 @@
-username,password
-research,Kfj%5M$d134DEwxa6098!!33

-- 
To view, visit https://gerrit.wikimedia.org/r/385807
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I68976257eed326eec9f36ef00b426f902b030b2f
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: WDCM_SemanticsDashboard - Skeleton 21 Oct 2017

2017-10-20 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385486 )

Change subject: WDCM_SemanticsDashboard - Skeleton 21 Oct 2017
..


WDCM_SemanticsDashboard - Skeleton 21 Oct 2017

Change-Id: Ifab45d2c35231a4d53ce6e1ccba843db7aa5978a
---
A WDCM_SemanticsDashboard/server.R
A WDCM_SemanticsDashboard/ui.R
2 files changed, 50 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_SemanticsDashboard/server.R b/WDCM_SemanticsDashboard/server.R
new file mode 100644
index 000..da51dec
--- /dev/null
+++ b/WDCM_SemanticsDashboard/server.R
@@ -0,0 +1,30 @@
+### ---
+### --- WDCM Semantics Dashboard, v. Beta 0.1
+### --- Script: server.R, v. Beta 0.1
+### ---
+
+### --- Setup
+rm(list = ls())
+### 
+### --- general
+library(shiny)
+library(shinydashboard)
+library(RMySQL)
+library(data.table)
+library(DT)
+library(stringr)
+library(tidyr)
+library(dplyr)
+library(reshape2)
+### --- compute
+library(parallelDist)
+### --- visualization
+library(RColorBrewer)
+library(visNetwork)
+library(networkD3)
+library(ggplot2)
+library(ggrepel)
+library(scales)
+
+### --- Server (Session) Scope
+### 
\ No newline at end of file
diff --git a/WDCM_SemanticsDashboard/ui.R b/WDCM_SemanticsDashboard/ui.R
new file mode 100644
index 000..4b80f58
--- /dev/null
+++ b/WDCM_SemanticsDashboard/ui.R
@@ -0,0 +1,20 @@
+### ---
+### --- WDCM Semantics Dashboard, v. Beta 0.1
+### --- Script: ui.R, v. Beta 0.1
+### ---
+
+### --- Setup
+rm(list = ls())
+### --- general
+library(shiny)
+library(shinydashboard)
+library(shinycssloaders)
+### --- outputs
+library(visNetwork)
+library(rbokeh)
+library(networkD3)
+library(ggvis)
+library(DT)
+
+# - options
+options(warn = -1)
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/385486
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifab45d2c35231a4d53ce6e1ccba843db7aa5978a
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Minor Oct 21 2017

2017-10-20 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385485 )

Change subject: Minor Oct 21 2017
..


Minor Oct 21 2017

Change-Id: I9641666bc65a38268ca92a51d3065840e3bb608b
---
M WDCM_Collect_Items.R
M WDCM_Pre-Process.R
M WDCM_Process.R
3 files changed, 4 insertions(+), 5 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_Collect_Items.R b/WDCM_Collect_Items.R
index 055e468..b61c630 100644
--- a/WDCM_Collect_Items.R
+++ b/WDCM_Collect_Items.R
@@ -1,4 +1,4 @@
-### --- WDCM Script
+
 ### ---
 ### --- WDCM Search Module, v. Beta 0.1
 ### --- Script: WDCM_Collect_Items.R, v. Beta 0.1
diff --git a/WDCM_Pre-Process.R b/WDCM_Pre-Process.R
index 37aada0..58fb609 100644
--- a/WDCM_Pre-Process.R
+++ b/WDCM_Pre-Process.R
@@ -18,9 +18,8 @@
 ### --- wdcm_maintable Hive table on hdfs, database: goransm
 ### ---
 ### --- OUTPUT: 
-### --- Results are stored locally as .tsv files on production,
-### --- with each file encompassing the data for one client project.
-### --- The outputs are stored locally on stat1005.eqiad.wmnet in:
+### --- Results are stored locally as .tsv files on production -
+### --- - on stat1005.eqiad.wmnet - in:
 ### --- /home/goransm/WDCM_DataOUT/WDCM_DataOUT_ClientWDUsage
 ### --- These output .tsv files migrate to Labs:
 ### --- wikidataconcepts.wmflabs.org Cloud VPS instance
diff --git a/WDCM_Process.R b/WDCM_Process.R
index a9e9702..0e4ec54 100644
--- a/WDCM_Process.R
+++ b/WDCM_Process.R
@@ -1,7 +1,7 @@
 
 ### ---
 ### --- WDCM Process Module, v. Beta 0.1
-### --- Script: WDCM_Process_v2.R, v. Beta 0.1
+### --- Script: WDCM_Process.R, v. Beta 0.1
 ### ---
 ### --- DESCRIPTION:
 ### --- WDCM_Process_v2.R takes a list of .tsv files that present

-- 
To view, visit https://gerrit.wikimedia.org/r/385485
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9641666bc65a38268ca92a51d3065840e3bb608b
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Minor Oct 21 2017

2017-10-20 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385485 )

Change subject: Minor Oct 21 2017
..

Minor Oct 21 2017

Change-Id: I9641666bc65a38268ca92a51d3065840e3bb608b
---
M WDCM_Collect_Items.R
M WDCM_Pre-Process.R
M WDCM_Process.R
3 files changed, 4 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/85/385485/1

diff --git a/WDCM_Collect_Items.R b/WDCM_Collect_Items.R
index 055e468..b61c630 100644
--- a/WDCM_Collect_Items.R
+++ b/WDCM_Collect_Items.R
@@ -1,4 +1,4 @@
-### --- WDCM Script
+
 ### ---
 ### --- WDCM Search Module, v. Beta 0.1
 ### --- Script: WDCM_Collect_Items.R, v. Beta 0.1
diff --git a/WDCM_Pre-Process.R b/WDCM_Pre-Process.R
index 37aada0..58fb609 100644
--- a/WDCM_Pre-Process.R
+++ b/WDCM_Pre-Process.R
@@ -18,9 +18,8 @@
 ### --- wdcm_maintable Hive table on hdfs, database: goransm
 ### ---
 ### --- OUTPUT: 
-### --- Results are stored locally as .tsv files on production,
-### --- with each file encompassing the data for one client project.
-### --- The outputs are stored locally on stat1005.eqiad.wmnet in:
+### --- Results are stored locally as .tsv files on production -
+### --- - on stat1005.eqiad.wmnet - in:
 ### --- /home/goransm/WDCM_DataOUT/WDCM_DataOUT_ClientWDUsage
 ### --- These output .tsv files migrate to Labs:
 ### --- wikidataconcepts.wmflabs.org Cloud VPS instance
diff --git a/WDCM_Process.R b/WDCM_Process.R
index a9e9702..0e4ec54 100644
--- a/WDCM_Process.R
+++ b/WDCM_Process.R
@@ -1,7 +1,7 @@
 
 ### ---
 ### --- WDCM Process Module, v. Beta 0.1
-### --- Script: WDCM_Process_v2.R, v. Beta 0.1
+### --- Script: WDCM_Process.R, v. Beta 0.1
 ### ---
 ### --- DESCRIPTION:
 ### --- WDCM_Process_v2.R takes a list of .tsv files that present

-- 
To view, visit https://gerrit.wikimedia.org/r/385485
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9641666bc65a38268ca92a51d3065840e3bb608b
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: fix Labels to UTF-8

2017-10-21 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385809 )

Change subject: fix Labels to UTF-8
..


fix Labels to UTF-8

Change-Id: Ib59933f7be1658039dc3aaf21c3ab289982958f0
---
M WDCM_OverviewDashboard/server.R
M WDCM_UsageDashboard/server.R
2 files changed, 10 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/WDCM_OverviewDashboard/server.R b/WDCM_OverviewDashboard/server.R
index 21afbfe..b66201b 100644
--- a/WDCM_OverviewDashboard/server.R
+++ b/WDCM_OverviewDashboard/server.R
@@ -59,6 +59,11 @@
 dbClearResult(res)
 colnames(st) <- "tables"
 
+### --- SET CHARACTER SET utf8
+q <- "SET CHARACTER SET utf8;"
+res <- dbSendQuery(con, q)
+dbClearResult(res)
+
 ### --- fetch wdcm2_project
 q <- "SELECT * FROM wdcm2_project;"
 res <- dbSendQuery(con, q)
diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
index be1df87..db927a5 100644
--- a/WDCM_UsageDashboard/server.R
+++ b/WDCM_UsageDashboard/server.R
@@ -52,6 +52,11 @@
 dbClearResult(res)
 colnames(st) <- "tables"
 
+### --- SET CHARACTER SET utf8
+q <- "SET CHARACTER SET utf8;"
+res <- dbSendQuery(con, q)
+dbClearResult(res)
+
 ### --- fetch wdcm2_project
 q <- "SELECT * FROM wdcm2_project;"
 res <- dbSendQuery(con, q)

-- 
To view, visit https://gerrit.wikimedia.org/r/385809
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib59933f7be1658039dc3aaf21c3ab289982958f0
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: fix Labels to UTF-8

2017-10-21 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385809 )

Change subject: fix Labels to UTF-8
..

fix Labels to UTF-8

Change-Id: Ib59933f7be1658039dc3aaf21c3ab289982958f0
---
M WDCM_OverviewDashboard/server.R
M WDCM_UsageDashboard/server.R
2 files changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/09/385809/1

diff --git a/WDCM_OverviewDashboard/server.R b/WDCM_OverviewDashboard/server.R
index 21afbfe..b66201b 100644
--- a/WDCM_OverviewDashboard/server.R
+++ b/WDCM_OverviewDashboard/server.R
@@ -59,6 +59,11 @@
 dbClearResult(res)
 colnames(st) <- "tables"
 
+### --- SET CHARACTER SET utf8
+q <- "SET CHARACTER SET utf8;"
+res <- dbSendQuery(con, q)
+dbClearResult(res)
+
 ### --- fetch wdcm2_project
 q <- "SELECT * FROM wdcm2_project;"
 res <- dbSendQuery(con, q)
diff --git a/WDCM_UsageDashboard/server.R b/WDCM_UsageDashboard/server.R
index be1df87..db927a5 100644
--- a/WDCM_UsageDashboard/server.R
+++ b/WDCM_UsageDashboard/server.R
@@ -52,6 +52,11 @@
 dbClearResult(res)
 colnames(st) <- "tables"
 
+### --- SET CHARACTER SET utf8
+q <- "SET CHARACTER SET utf8;"
+res <- dbSendQuery(con, q)
+dbClearResult(res)
+
 ### --- fetch wdcm2_project
 q <- "SELECT * FROM wdcm2_project;"
 res <- dbSendQuery(con, q)

-- 
To view, visit https://gerrit.wikimedia.org/r/385809
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib59933f7be1658039dc3aaf21c3ab289982958f0
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Semantics Dashboard + File Structure

2017-10-22 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385929 )

Change subject: Semantics Dashboard + File Structure
..


Semantics Dashboard + File Structure

Change-Id: I9385416ebf6b7100687d57b2bb9640fdd7e2b9c8
---
A .directory
A .gitignore
D WDCM_Dahsboard/server.R
D WDCM_Dahsboard/ui.R
M WDCM_OverviewDashboard/server.R
M WDCM_OverviewDashboard/ui.R
M WDCM_Process.R
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ArchitecturalStructure.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_AstronomicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Book.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ChemicalEntities.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Event.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Gene.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_GeographicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Human.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Organization.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ScientificArticle.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Taxon.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Thoroughfare.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Wikimedia.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_WorkofArt.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ArchitecturalStructure.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_AstronomicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Book.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ChemicalEntities.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Event.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Gene.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_GeographicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Human.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Organization.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ScientificArticle.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Taxon.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Thoroughfare.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Wikimedia.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_WorkofArt.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ArchitecturalStructure.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_AstronomicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Book.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ChemicalEntities.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Event.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Gene.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_GeographicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Human.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Organization.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ScientificArticle.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Taxon.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Thoroughfare.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Wikimedia.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_WorkofArt.csv
M WDCM_SemanticsDashboard/server.R
M WDCM_SemanticsDashboard/ui.R
A WDCM_SemanticsDashboard/www/Wikidata-logo-en.png
A WDCM_TechDocumentation/.directory
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
55 files changed, 16,343 insertions(+), 2,460 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved




-- 
To view, visit https://gerrit.wikimedia.org/r/385929
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9385416ebf6b7100687d57b2bb9640fdd7e2b9c8
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: Semantics Dashboard + File Structure

2017-10-22 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385929 )

Change subject: Semantics Dashboard + File Structure
..

Semantics Dashboard + File Structure

Change-Id: I9385416ebf6b7100687d57b2bb9640fdd7e2b9c8
---
A .directory
A .gitignore
D WDCM_Dahsboard/server.R
D WDCM_Dahsboard/ui.R
M WDCM_OverviewDashboard/server.R
M WDCM_OverviewDashboard/ui.R
M WDCM_Process.R
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ArchitecturalStructure.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_AstronomicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Book.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ChemicalEntities.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Event.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Gene.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_GeographicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Human.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Organization.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_ScientificArticle.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Taxon.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Thoroughfare.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_Wikimedia.csv
A WDCM_SemanticsDashboard/data/wdcm2_projecttopic_WorkofArt.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ArchitecturalStructure.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_AstronomicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Book.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ChemicalEntities.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Event.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Gene.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_GeographicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Human.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Organization.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_ScientificArticle.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Taxon.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Thoroughfare.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_Wikimedia.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkEdges_project_WorkofArt.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ArchitecturalStructure.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_AstronomicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Book.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ChemicalEntities.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Event.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Gene.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_GeographicalObject.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Human.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Organization.csv
A 
WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_ScientificArticle.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Taxon.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Thoroughfare.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_Wikimedia.csv
A WDCM_SemanticsDashboard/data/wdcm2_visNetworkNodes_project_WorkofArt.csv
M WDCM_SemanticsDashboard/server.R
M WDCM_SemanticsDashboard/ui.R
A WDCM_SemanticsDashboard/www/Wikidata-logo-en.png
A WDCM_TechDocumentation/.directory
M WDCM_UsageDashboard/server.R
M WDCM_UsageDashboard/ui.R
55 files changed, 16,343 insertions(+), 2,460 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/29/385929/1


-- 
To view, visit https://gerrit.wikimedia.org/r/385929
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9385416ebf6b7100687d57b2bb9640fdd7e2b9c8
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor

2017-10-22 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/385931 )

Change subject: minor
..


minor

Change-Id: Ib64af37ae1dd1ade02763fa057a755864e89
---
D _misc/.directory
D _misc/SqoopRuntime.png
D _misc/WDCM_Navigate.html
D _misc/WDCM_OverviewDashboard_Description.html
D _misc/clienttables.tsv
D _misc/databasesTest.tsv
D _misc/hiveQLQuery.hql
D _misc/hiveQLQueryTest.hql
D _misc/projectsTracking.csv
D _misc/wdcmSearchReport_2017-08-01.csv
D _misc/wdcmSqoopReport_2017-07-21.csv
11 files changed, 0 insertions(+), 2,746 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved




-- 
To view, visit https://gerrit.wikimedia.org/r/385931
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib64af37ae1dd1ade02763fa057a755864e89
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM[master]: minor

2017-10-22 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/385931 )

Change subject: minor
..

minor

Change-Id: Ib64af37ae1dd1ade02763fa057a755864e89
---
D _misc/.directory
D _misc/SqoopRuntime.png
D _misc/WDCM_Navigate.html
D _misc/WDCM_OverviewDashboard_Description.html
D _misc/clienttables.tsv
D _misc/databasesTest.tsv
D _misc/hiveQLQuery.hql
D _misc/hiveQLQueryTest.hql
D _misc/projectsTracking.csv
D _misc/wdcmSearchReport_2017-08-01.csv
D _misc/wdcmSqoopReport_2017-07-21.csv
11 files changed, 0 insertions(+), 2,746 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM 
refs/changes/31/385931/1


-- 
To view, visit https://gerrit.wikimedia.org/r/385931
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib64af37ae1dd1ade02763fa057a755864e89
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-GeoDashboard[master]: initial

2017-12-31 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401178 )

Change subject: initial
..


initial

Change-Id: Ie9b6a717f2203ab428fecc4518cdc057ef90882f
---
A .gitignore
A server.R
A ui.R
3 files changed, 370 insertions(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..c23da9e
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+**/data
+**/www
+**/update
+
diff --git a/server.R b/server.R
new file mode 100644
index 000..58191de
--- /dev/null
+++ b/server.R
@@ -0,0 +1,143 @@
+### ---
+### --- WDCM Geo Dashboard, v. Beta 0.1
+### --- Script: server.R, v. Beta 0.1
+### ---
+
+### --- Setup
+
+### 
+### --- general
+library(shiny)
+library(shinydashboard)
+library(data.table)
+library(DT)
+library(stringr)
+### --- visualize
+library(leaflet)
+
+### --- Server (Session) Scope
+### 
+
+### --- Fetch local files
+setwd('/srv/shiny-server/WDCM_GeoDashboard/data/')
+
+### --- fetch projecttopic tables
+lF <- list.files()
+lF <- lF[grepl("^wdcm_geoitem_", lF)]
+categories <- vector(mode = "list", length = length(lF))
+for (i in 1:length(lF)) {
+  categories[[i]] <- fread(lF[i], data.table = F)
+}
+names(categories) <- str_to_title(sapply(lF, function(x) {
+  strsplit(strsplit(x, split = ".", fixed = T)[[1]][1],
+   split = "_",
+   fixed = T)[[1]][3]
+}))
+
+
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_GeoDashboard/update/')
+update <- read.csv('toLabsGeoReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
+### --- shinyServer
+shinyServer(function(input, output, session) {
+  
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
+  
+  ### --
+  ### --- TAB: tabPanel Maps
+  ### --
+  
+  ### --- SELECT: update select 'selectCategory'
+  updateSelectizeInput(session,
+   'selectCategory',
+   "Select Semantic Category:",
+   choices = names(categories),
+   selected = names(categories)[round(runif(1, 1, 
length(categories)))],
+   server = TRUE)
+  
+  ### --- LEAFLET MAP:
+  points <- eventReactive(input$selectCategory, {
+if (is.null(input$selectCategory) | (input$selectCategory == "")) {
+  return(NULL)
+} else {
+  outCat <- categories[[which(names(categories) %in% 
input$selectCategory)]]
+  outCat[, 2:dim(outCat)[2]] 
+}
+  }, ignoreNULL = FALSE)
+  
+  output$wdcmMap <- renderLeaflet({
+if (is.null(points())) {
+  return(NULL) 
+} else {
+  leaflet() %>%
+addTiles() %>%
+addMarkers(data = points(), 
+   popup = (paste('', points()$label, '',
+  'https://www.wikidata.org/wiki/', 
points()$item, '" target = "_blank">', points()$item, '',
+  'Usage: ', points()$usage, sep = "")
+),
+   clusterOptions = markerClusterOptions()
+   )
+}
+}) %>% withProgress(message = 'Generating map',
+  min = 0,
+  max = 1,
+  value = 1, {incProgress(amount = 1)})
+
+  ### --
+  ### --- TAB: tabPanel Data
+  ### --
+  
+  ### --- output$mapData
+  output$mapData <- DT::renderDataTable({
+datatable(points(),
+  options = list(
+pageLength = 20,
+width = '100%',
+columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
+  ),
+  rownames = FALSE
+)
+  }) %>% withProgress(message = 'Generating data',
+  min = 0,
+  max = 1,
+  value = 1, {incProgress(amount = 1)})
+  
+  ### --- download map data
+  # - Download: tabulations_projectsChart
+  output$mapDataCSV <- downloadHandler(
+filename = function() {
+  'WDCM_Data.csv'},
+content = function(file) {
+  write.csv(points(),
+file,
+quote = FALSE,
+

[MediaWiki-commits] [Gerrit] analytics...WDCM-GeoDashboard[master]: initial

2017-12-31 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401178 )

Change subject: initial
..

initial

Change-Id: Ie9b6a717f2203ab428fecc4518cdc057ef90882f
---
A .gitignore
A server.R
A ui.R
3 files changed, 370 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-GeoDashboard 
refs/changes/78/401178/1

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..c23da9e
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+**/data
+**/www
+**/update
+
diff --git a/server.R b/server.R
new file mode 100644
index 000..58191de
--- /dev/null
+++ b/server.R
@@ -0,0 +1,143 @@
+### ---
+### --- WDCM Geo Dashboard, v. Beta 0.1
+### --- Script: server.R, v. Beta 0.1
+### ---
+
+### --- Setup
+
+### 
+### --- general
+library(shiny)
+library(shinydashboard)
+library(data.table)
+library(DT)
+library(stringr)
+### --- visualize
+library(leaflet)
+
+### --- Server (Session) Scope
+### 
+
+### --- Fetch local files
+setwd('/srv/shiny-server/WDCM_GeoDashboard/data/')
+
+### --- fetch projecttopic tables
+lF <- list.files()
+lF <- lF[grepl("^wdcm_geoitem_", lF)]
+categories <- vector(mode = "list", length = length(lF))
+for (i in 1:length(lF)) {
+  categories[[i]] <- fread(lF[i], data.table = F)
+}
+names(categories) <- str_to_title(sapply(lF, function(x) {
+  strsplit(strsplit(x, split = ".", fixed = T)[[1]][1],
+   split = "_",
+   fixed = T)[[1]][3]
+}))
+
+
+### --- Fetch update info
+setwd('/srv/shiny-server/WDCM_GeoDashboard/update/')
+update <- read.csv('toLabsGeoReport.csv', 
+   header = T,
+   check.names = F,
+   stringsAsFactors = F,
+   row.names = 1)
+
+### --- shinyServer
+shinyServer(function(input, output, session) {
+  
+  ### --- output: updateInfo
+  output$updateInfo <- renderText({
+date <- update$timeStamp[dim(update)[1]]
+date <- strsplit(as.character(date), split = " ", fixed = T)[[1]][1]
+date <- strsplit(date, split = "-", fixed = T)
+date[[1]][2] <- month.name[as.numeric(date[[1]][2])]
+date <- paste(unlist(date), collapse = " ")
+return(paste("Last update: ", date, "", sep = 
""))
+  })
+  
+  ### --
+  ### --- TAB: tabPanel Maps
+  ### --
+  
+  ### --- SELECT: update select 'selectCategory'
+  updateSelectizeInput(session,
+   'selectCategory',
+   "Select Semantic Category:",
+   choices = names(categories),
+   selected = names(categories)[round(runif(1, 1, 
length(categories)))],
+   server = TRUE)
+  
+  ### --- LEAFLET MAP:
+  points <- eventReactive(input$selectCategory, {
+if (is.null(input$selectCategory) | (input$selectCategory == "")) {
+  return(NULL)
+} else {
+  outCat <- categories[[which(names(categories) %in% 
input$selectCategory)]]
+  outCat[, 2:dim(outCat)[2]] 
+}
+  }, ignoreNULL = FALSE)
+  
+  output$wdcmMap <- renderLeaflet({
+if (is.null(points())) {
+  return(NULL) 
+} else {
+  leaflet() %>%
+addTiles() %>%
+addMarkers(data = points(), 
+   popup = (paste('', points()$label, '',
+  'https://www.wikidata.org/wiki/', 
points()$item, '" target = "_blank">', points()$item, '',
+  'Usage: ', points()$usage, sep = "")
+),
+   clusterOptions = markerClusterOptions()
+   )
+}
+}) %>% withProgress(message = 'Generating map',
+  min = 0,
+  max = 1,
+  value = 1, {incProgress(amount = 1)})
+
+  ### --
+  ### --- TAB: tabPanel Data
+  ### --
+  
+  ### --- output$mapData
+  output$mapData <- DT::renderDataTable({
+datatable(points(),
+  options = list(
+pageLength = 20,
+width = '100%',
+columnDefs = list(list(className = 'dt-center', targets = 
"_all"))
+  ),
+  rownames = FALSE
+)
+  }) %>% withProgress(message = 'Generating data',
+  min = 0,
+  max = 1,
+  value = 1, {incProgress(amount = 1)})
+  
+  ### --- download map data
+  # - Download: tabulations_projectsChart
+  output$mapDataCSV <- downloadHandler(
+filename = function() {
+  'WDCM_Data.csv'},
+content = function(file) {
+  write.csv(points(),
+file,
+

[MediaWiki-commits] [Gerrit] analytics...WDCM-ShinyServerFrontPage[master]: minor Jan 1 2018

2017-12-31 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401196 )

Change subject: minor Jan 1 2018
..


minor Jan 1 2018

Change-Id: Ib62f02d297d4dfe361033d5d4c58689ed0507689
---
A GeoDashboard.png
M Wikidata-logo-en.png
M index.html
M wikitech.png
4 files changed, 12 insertions(+), 3 deletions(-)

Approvals:
  GoranSMilovanovic: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/GeoDashboard.png b/GeoDashboard.png
new file mode 100644
index 000..eda2535
--- /dev/null
+++ b/GeoDashboard.png
Binary files differ
diff --git a/Wikidata-logo-en.png b/Wikidata-logo-en.png
index 5e52bba..eee7383 100644
--- a/Wikidata-logo-en.png
+++ b/Wikidata-logo-en.png
Binary files differ
diff --git a/index.html b/index.html
index 627138d..a20e0d9 100644
--- a/index.html
+++ b/index.html
@@ -266,13 +266,22 @@
 
 
 
+http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM 
Geo
+ http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>
+ 
+
+The Geo Dashboard provides interactive Leaflet maps of Wikidata items 
and their usage statistics.
+
+
+
+
 https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM on 
Wikitech
- https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>
+ https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>
  
 
 
-https://www.wikidata.org/wiki/Q42376073;>WDCM on 
Wikidata
- https://www.wikidata.org/wiki/Q42376073; target = 
"blank">
+https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor;>WDCM on 
Wikidata
+ https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor; target = 
"blank">
 
   
 
diff --git a/wikitech.png b/wikitech.png
index 2cee20d..8d2be7e 100644
--- a/wikitech.png
+++ b/wikitech.png
Binary files differ

-- 
To view, visit https://gerrit.wikimedia.org/r/401196
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib62f02d297d4dfe361033d5d4c58689ed0507689
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-ShinyServerFrontPage
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-ShinyServerFrontPage[master]: minor Jan 1 2018

2017-12-31 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401196 )

Change subject: minor Jan 1 2018
..

minor Jan 1 2018

Change-Id: Ib62f02d297d4dfe361033d5d4c58689ed0507689
---
A GeoDashboard.png
M Wikidata-logo-en.png
M index.html
M wikitech.png
4 files changed, 12 insertions(+), 3 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-ShinyServerFrontPage 
refs/changes/96/401196/1

diff --git a/GeoDashboard.png b/GeoDashboard.png
new file mode 100644
index 000..eda2535
--- /dev/null
+++ b/GeoDashboard.png
Binary files differ
diff --git a/Wikidata-logo-en.png b/Wikidata-logo-en.png
index 5e52bba..eee7383 100644
--- a/Wikidata-logo-en.png
+++ b/Wikidata-logo-en.png
Binary files differ
diff --git a/index.html b/index.html
index 627138d..a20e0d9 100644
--- a/index.html
+++ b/index.html
@@ -266,13 +266,22 @@
 
 
 
+http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>WDCM 
Geo
+ http://wdcm.wmflabs.org/WDCM_GeoDashboard/;>
+ 
+
+The Geo Dashboard provides interactive Leaflet maps of Wikidata items 
and their usage statistics.
+
+
+
+
 https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>WDCM on 
Wikitech
- https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>
+ https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor;>
  
 
 
-https://www.wikidata.org/wiki/Q42376073;>WDCM on 
Wikidata
- https://www.wikidata.org/wiki/Q42376073; target = 
"blank">
+https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor;>WDCM on 
Wikidata
+ https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor; target = 
"blank">
 
   
 
diff --git a/wikitech.png b/wikitech.png
index 2cee20d..8d2be7e 100644
--- a/wikitech.png
+++ b/wikitech.png
Binary files differ

-- 
To view, visit https://gerrit.wikimedia.org/r/401196
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib62f02d297d4dfe361033d5d4c58689ed0507689
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-ShinyServerFrontPage
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-GeoDashboard[master]: server.R quick fix

2018-01-01 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401406 )

Change subject: server.R quick fix
..

server.R quick fix

Change-Id: I202d4bfd326602502f95545cda209ddb1a7db146
---
M server.R
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/wmde/WDCM-GeoDashboard 
refs/changes/06/401406/1

diff --git a/server.R b/server.R
index 58191de..c477050 100644
--- a/server.R
+++ b/server.R
@@ -24,6 +24,7 @@
 ### --- fetch projecttopic tables
 lF <- list.files()
 lF <- lF[grepl("^wdcm_geoitem_", lF)]
+lF <- lF[grepl(".csv", lF)]
 categories <- vector(mode = "list", length = length(lF))
 for (i in 1:length(lF)) {
   categories[[i]] <- fread(lF[i], data.table = F)

-- 
To view, visit https://gerrit.wikimedia.org/r/401406
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I202d4bfd326602502f95545cda209ddb1a7db146
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-GeoDashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics...WDCM-GeoDashboard[master]: server.R quick fix

2018-01-01 Thread GoranSMilovanovic (Code Review) 
GoranSMilovanovic has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401406 )

Change subject: server.R quick fix
..


server.R quick fix

Change-Id: I202d4bfd326602502f95545cda209ddb1a7db146
---
M server.R
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  GoranSMilovanovic: Verified; Looks good to me, approved



diff --git a/server.R b/server.R
index 58191de..c477050 100644
--- a/server.R
+++ b/server.R
@@ -24,6 +24,7 @@
 ### --- fetch projecttopic tables
 lF <- list.files()
 lF <- lF[grepl("^wdcm_geoitem_", lF)]
+lF <- lF[grepl(".csv", lF)]
 categories <- vector(mode = "list", length = length(lF))
 for (i in 1:length(lF)) {
   categories[[i]] <- fread(lF[i], data.table = F)

-- 
To view, visit https://gerrit.wikimedia.org/r/401406
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I202d4bfd326602502f95545cda209ddb1a7db146
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wmde/WDCM-GeoDashboard
Gerrit-Branch: master
Gerrit-Owner: GoranSMilovanovic 
Gerrit-Reviewer: GoranSMilovanovic 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits