Repository: incubator-metron Updated Branches: refs/heads/master b2e46f574 -> c793308cb
METRON-752 Fix documentation typos and MD issues closes apache/incubator-metron#470 Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/c793308c Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/c793308c Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/c793308c Branch: refs/heads/master Commit: c793308cbd44e4cdbc223b141ed89978d8a52acb Parents: b2e46f5 Author: JonZeolla <zeo...@gmail.com> Authored: Thu Mar 9 13:50:41 2017 -0500 Committer: cstella <ceste...@gmail.com> Committed: Thu Mar 9 13:50:41 2017 -0500 ---------------------------------------------------------------------- Upgrading.md | 40 +++---- metron-analytics/metron-profiler/README.md | 48 ++++---- metron-interface/metron-rest/README.md | 110 +++++++++---------- .../metron-data-management/README.md | 66 +++++------ site-book/bin/fix-md-dialect.py | 16 +-- 5 files changed, 140 insertions(+), 140 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/Upgrading.md ---------------------------------------------------------------------- diff --git a/Upgrading.md b/Upgrading.md index 2a2ea50..92e958b 100644 --- a/Upgrading.md +++ b/Upgrading.md @@ -2,26 +2,6 @@ This document constitutes a per-version listing of changes of configuration which are non-backwards compatible. -## 0.2.0BETA to 0.3.0 -### [METRON-447: Monit fails to reload when upgrading from 0.2.0BETA to master](https://issues.apache.org/jira/browse/METRON-447) - -#### Description - -`/etc/monit.d/enrichment-elasticsearch.monit` was renamed to -`/etc/monit.d/indexing-elasticsearch.monit`, however the old file isn't -removed via ansible, which causes the below error during an upgrade: -`Starting monit: /etc/monit.d/enrichment-elasticsearch.monit:18: Service -name conflict, enrichment already defined -'/usr/local/monit/status_enrichment_topology.sh'` - -### [METRON-448:Upgrading via Ansible deployment does not add topology.classpath ](https://issues.apache.org/jira/browse/METRON-448) - -#### Description -When using Ansible to deploy the latest Metron bits to an existing installation, storm-site is not being updated with the new 0.2.1BETA parameter `topology.classpath`. Topologies are unable to find the client configs as a result. - -#### Workaround -Set the `topology.classpath` property for storm in Ambari to `/etc/hbase/conf:/etc/hadoop/conf` - ## 0.3.0 to 0.3.1 ### [METRON-664: Make the index configuration per-writer with enabled/disabled](https://issues.apache.org/jira/browse/METRON-664) @@ -152,3 +132,23 @@ into a separate function. The consequence is that existing calls to Existing calls to `PROFILE_GET` will need to change from `PROFILE_GET('profile', 'entity', duration, 'durationUnits')` to `PROFILE_GET('profile', 'entity', PROFILE_FIXED(duration, 'durationUnits'))` +## 0.2.0BETA to 0.3.0 +### [METRON-447: Monit fails to reload when upgrading from 0.2.0BETA to master](https://issues.apache.org/jira/browse/METRON-447) + +#### Description + +`/etc/monit.d/enrichment-elasticsearch.monit` was renamed to +`/etc/monit.d/indexing-elasticsearch.monit`, however the old file isn't +removed via ansible, which causes the below error during an upgrade: +`Starting monit: /etc/monit.d/enrichment-elasticsearch.monit:18: Service +name conflict, enrichment already defined +'/usr/local/monit/status_enrichment_topology.sh'` + +### [METRON-448:Upgrading via Ansible deployment does not add topology.classpath ](https://issues.apache.org/jira/browse/METRON-448) + +#### Description +When using Ansible to deploy the latest Metron bits to an existing installation, storm-site is not being updated with the new 0.2.1BETA parameter `topology.classpath`. Topologies are unable to find the client configs as a result. + +#### Workaround +Set the `topology.classpath` property for storm in Ambari to `/etc/hbase/conf:/etc/hadoop/conf` + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/metron-analytics/metron-profiler/README.md ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-profiler/README.md b/metron-analytics/metron-profiler/README.md index b4fc104..bb9f530 100644 --- a/metron-analytics/metron-profiler/README.md +++ b/metron-analytics/metron-profiler/README.md @@ -69,7 +69,6 @@ More information on configuring and using the client can be found [here](../metr It is assumed that the `PROFILE_GET` client is correctly configured before using it. ``` $ bin/stellar -z node1:2181 - [Stellar]>>> PROFILE_GET( "test", "10.0.0.1", PROFILE_FIXED(30, "MINUTES")) [451, 448] ``` @@ -80,15 +79,15 @@ The Profiler specification requires a JSON-formatted set of elements, many of wh The specification for the Profiler topology is stored in Zookeeper at `/metron/topology/profiler`. These properties also exist in the local filesystem at `$METRON_HOME/config/zookeeper/profiler.json`. The values can be changed on disk and then uploaded to Zookeeper using `$METRON_HOME/bin/zk_load_configs.sh`. -| Name | | Description -|--- |--- |--- -| [profile](#profile) | Required | Unique name identifying the profile. -| [foreach](#foreach) | Required | A separate profile is maintained "for each" of these. -| [onlyif](#onlyif) | Optional | Boolean expression that determines if a message should be applied to the profile. +| Name | | Description +|--- |--- |--- +| [profile](#profile) | Required | Unique name identifying the profile. +| [foreach](#foreach) | Required | A separate profile is maintained "for each" of these. +| [onlyif](#onlyif) | Optional | Boolean expression that determines if a message should be applied to the profile. | [groupBy](#groupby) | Optional | One or more Stellar expressions used to group the profile measurements when persisted. -| [init](#init) | Optional | One or more expressions executed at the start of a window period. -| [update](#update) | Required | One or more expressions executed when a message is applied to the profile. -| [result](#result) | Required | Stellar expressions that are executed when the window period expires. +| [init](#init) | Optional | One or more expressions executed at the start of a window period. +| [update](#update) | Required | One or more expressions executed when a message is applied to the profile. +| [result](#result) | Required | Stellar expressions that are executed when the window period expires. | [expires](#expires) | Optional | Profile data is purged after this period of time, specified in milliseconds. ### `profile` @@ -174,7 +173,6 @@ A required Stellar expression that results in a value that is persisted in the p An alternative, simplified form is also acceptable. ``` "result": "2 + 2" - ``` **triage** @@ -204,21 +202,21 @@ The Profiler runs as an independent Storm topology. The configuration for the P The values can be changed on disk and then the Profiler topology must be restarted. -| Setting | Description | -|--- |--- | -| profiler.workers | The number of worker processes to create for the topology. | -| profiler.executors | The number of executors to spawn per component. | -| profiler.input.topic | The name of the Kafka topic from which to consume data. | -| profiler.output.topic | The name of the Kafka topic to which profile data is written. Only used with profiles that use the [`triage` result field](#result). | -| profiler.period.duration | The duration of each profile period. This value should be defined along with `profiler.period.duration.units`. | -| profiler.period.duration.units | The units used to specify the `profiler.period.duration`. | -| profiler.ttl | If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with `profiler.ttl.units`. | -| profiler.ttl.units | The units used to specify the `profiler.ttl`. | -| profiler.hbase.salt.divisor | A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster. | -| profiler.hbase.table | The name of the HBase table that profiles are written to. | -| profiler.hbase.column.family | The column family used to store profiles. | -| profiler.hbase.batch | The number of puts that are written in a single batch. | -| profiler.hbase.flush.interval.seconds | The maximum number of seconds between batch writes to HBase. | +| Setting | Description +|--- |--- +| profiler.workers | The number of worker processes to create for the topology. +| profiler.executors | The number of executors to spawn per component. +| profiler.input.topic | The name of the Kafka topic from which to consume data. +| profiler.output.topic | The name of the Kafka topic to which profile data is written. Only used with profiles that use the [`triage` result field](#result). +| profiler.period.duration | The duration of each profile period. This value should be defined along with `profiler.period.duration.units`. +| profiler.period.duration.units | The units used to specify the `profiler.period.duration`. +| profiler.ttl | If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with `profiler.ttl.units`. +| profiler.ttl.units | The units used to specify the `profiler.ttl`. +| profiler.hbase.salt.divisor | A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster. +| profiler.hbase.table | The name of the HBase table that profiles are written to. +| profiler.hbase.column.family | The column family used to store profiles. +| profiler.hbase.batch | The number of puts that are written in a single batch. +| profiler.hbase.flush.interval.seconds | The maximum number of seconds between batch writes to HBase. After altering the configuration, start the Profiler. http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/metron-interface/metron-rest/README.md ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md index cf41bf3..8de1982 100644 --- a/metron-interface/metron-rest/README.md +++ b/metron-interface/metron-rest/README.md @@ -10,24 +10,24 @@ This module provides a RESTful API for interacting with Metron. ## Installation 1. Package the application with Maven: -``` -mvn clean package -``` + ``` + mvn clean package + ``` 1. Untar the archive in the target directory. The directory structure will look like: -``` -bin - start_metron_rest.sh -lib - metron-rest-$METRON_VERSION.jar -``` + ``` + bin + start_metron_rest.sh + lib + metron-rest-$METRON_VERSION.jar + ``` -1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml). Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc) in properties containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details). +1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml). Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc.) in properties containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details). 1. Start the application with this command: -``` -./bin/start_metron_rest.sh /path/to/application.yml -``` + ``` + ./bin/start_metron_rest.sh /path/to/application.yml + ``` ## Usage @@ -45,36 +45,36 @@ For [production use](http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/refere 1. Create a MySQL user for the Metron REST application (http://dev.mysql.com/doc/refman/5.7/en/adding-users.html). 1. Connect to MySQL and create a Metron REST database: -``` -CREATE DATABASE IF NOT EXISTS metronrest -``` + ``` + CREATE DATABASE IF NOT EXISTS metronrest + ``` 1. Add users: -``` -use metronrest; -insert into users (username, password, enabled) values ('your_username','your_password',1); -insert into authorities (username, authority) values ('your_username', 'ROLE_USER'); -``` + ``` + use metronrest; + insert into users (username, password, enabled) values ('your_username','your_password',1); + insert into authorities (username, authority) values ('your_username', 'ROLE_USER'); + ``` 1. Replace the H2 connection information in the application.yml file with MySQL connection information: -``` -spring: - datasource: - driverClassName: com.mysql.jdbc.Driver - url: jdbc:mysql://mysql_host:3306/metronrest - username: metron_rest_user - password: metron_rest_password - platform: mysql -``` + ``` + spring: + datasource: + driverClassName: com.mysql.jdbc.Driver + url: jdbc:mysql://mysql_host:3306/metronrest + username: metron_rest_user + password: metron_rest_password + platform: mysql + ``` 1. Add a dependency for the MySQL JDBC connector in the metron-rest pom.xml: -``` -<dependency> - <groupId>mysql</groupId> - <artifactId>mysql-connector-java</artifactId> - <version>${mysql.client.version}</version> -</dependency> -``` + ``` + <dependency> + <groupId>mysql</groupId> + <artifactId>mysql-connector-java</artifactId> + <version>${mysql.client.version}</version> + </dependency> + ``` 1. Follow the steps in the [Installation](#installation) section @@ -95,25 +95,25 @@ Request and Response objects are JSON formatted. The JSON schemas are available | [ `GET /api/v1/hdfs/list`](#get-apiv1hdfslist)| | [ `GET /api/v1/kafka/topic`](#get-apiv1kafkatopic)| | [ `POST /api/v1/kafka/topic`](#post-apiv1kafkatopic)| -| [ `GET /api/v1/kafka/topic/{name}`](#get-apiv1kafkatopic{name})| -| [ `DELETE /api/v1/kafka/topic/{name}`](#delete-apiv1kafkatopic{name})| -| [ `GET /api/v1/kafka/topic/{name}/sample`](#get-apiv1kafkatopic{name}sample)| +| [ `GET /api/v1/kafka/topic/{name}`](#get-apiv1kafkatopicname)| +| [ `DELETE /api/v1/kafka/topic/{name}`](#delete-apiv1kafkatopicname)| +| [ `GET /api/v1/kafka/topic/{name}/sample`](#get-apiv1kafkatopicnamesample)| | [ `GET /api/v1/sensor/enrichment/config`](#get-apiv1sensorenrichmentconfig)| | [ `GET /api/v1/sensor/enrichment/config/list/available`](#get-apiv1sensorenrichmentconfiglistavailable)| -| [ `DELETE /api/v1/sensor/enrichment/config/{name}`](#delete-apiv1sensorenrichmentconfig{name})| -| [ `POST /api/v1/sensor/enrichment/config/{name}`](#post-apiv1sensorenrichmentconfig{name})| -| [ `GET /api/v1/sensor/enrichment/config/{name}`](#get-apiv1sensorenrichmentconfig{name})| +| [ `DELETE /api/v1/sensor/enrichment/config/{name}`](#delete-apiv1sensorenrichmentconfigname)| +| [ `POST /api/v1/sensor/enrichment/config/{name}`](#post-apiv1sensorenrichmentconfigname)| +| [ `GET /api/v1/sensor/enrichment/config/{name}`](#get-apiv1sensorenrichmentconfigname)| | [ `GET /api/v1/sensor/indexing/config`](#get-apiv1sensorindexingconfig)| -| [ `DELETE /api/v1/sensor/indexing/config/{name}`](#delete-apiv1sensorindexingconfig{name})| -| [ `POST /api/v1/sensor/indexing/config/{name}`](#post-apiv1sensorindexingconfig{name})| -| [ `GET /api/v1/sensor/indexing/config/{name}`](#get-apiv1sensorindexingconfig{name})| +| [ `DELETE /api/v1/sensor/indexing/config/{name}`](#delete-apiv1sensorindexingconfigname)| +| [ `POST /api/v1/sensor/indexing/config/{name}`](#post-apiv1sensorindexingconfigname)| +| [ `GET /api/v1/sensor/indexing/config/{name}`](#get-apiv1sensorindexingconfigname)| | [ `POST /api/v1/sensor/parser/config`](#post-apiv1sensorparserconfig)| | [ `GET /api/v1/sensor/parser/config`](#get-apiv1sensorparserconfig)| | [ `GET /api/v1/sensor/parser/config/list/available`](#get-apiv1sensorparserconfiglistavailable)| | [ `POST /api/v1/sensor/parser/config/parseMessage`](#post-apiv1sensorparserconfigparsemessage)| | [ `GET /api/v1/sensor/parser/config/reload/available`](#get-apiv1sensorparserconfigreloadavailable)| -| [ `DELETE /api/v1/sensor/parser/config/{name}`](#delete-apiv1sensorparserconfig{name})| -| [ `GET /api/v1/sensor/parser/config/{name}`](#get-apiv1sensorparserconfig{name})| +| [ `DELETE /api/v1/sensor/parser/config/{name}`](#delete-apiv1sensorparserconfigname)| +| [ `GET /api/v1/sensor/parser/config/{name}`](#get-apiv1sensorparserconfigname)| | [ `POST /api/v1/stellar/apply/transformations`](#post-apiv1stellarapplytransformations)| | [ `GET /api/v1/stellar/list`](#get-apiv1stellarlist)| | [ `GET /api/v1/stellar/list/functions`](#get-apiv1stellarlistfunctions)| @@ -131,11 +131,11 @@ Request and Response objects are JSON formatted. The JSON schemas are available | [ `GET /api/v1/storm/indexing/deactivate`](#get-apiv1stormindexingdeactivate)| | [ `GET /api/v1/storm/indexing/start`](#get-apiv1stormindexingstart)| | [ `GET /api/v1/storm/indexing/stop`](#get-apiv1stormindexingstop)| -| [ `GET /api/v1/storm/parser/activate/{name}`](#get-apiv1stormparseractivate{name})| -| [ `GET /api/v1/storm/parser/deactivate/{name}`](#get-apiv1stormparserdeactivate{name})| -| [ `GET /api/v1/storm/parser/start/{name}`](#get-apiv1stormparserstart{name})| -| [ `GET /api/v1/storm/parser/stop/{name}`](#get-apiv1stormparserstop{name})| -| [ `GET /api/v1/storm/{name}`](#get-apiv1storm{name})| +| [ `GET /api/v1/storm/parser/activate/{name}`](#get-apiv1stormparseractivatename)| +| [ `GET /api/v1/storm/parser/deactivate/{name}`](#get-apiv1stormparserdeactivatename)| +| [ `GET /api/v1/storm/parser/start/{name}`](#get-apiv1stormparserstartname)| +| [ `GET /api/v1/storm/parser/stop/{name}`](#get-apiv1stormparserstopname)| +| [ `GET /api/v1/storm/{name}`](#get-apiv1stormname)| | [ `GET /api/v1/user`](#get-apiv1user)| ### `GET /api/v1/global/config` @@ -171,7 +171,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available * 200 - JSON results ### `POST /api/v1/hdfs` - * Description: Writes contents to an HDFS file. Warning: this will overwite the contents of a file if it already exists. + * Description: Writes contents to an HDFS file. Warning: this will overwrite the contents of a file if it already exists. * Input: * path - Path to HDFS file * contents - File contents @@ -222,7 +222,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available * 404 - Kafka topic is missing ### `DELETE /api/v1/kafka/topic/{name}` - * Description: Delets a Kafka topic + * Description: Deletes a Kafka topic * Input: * name - Kafka topic name * Returns: http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/metron-platform/metron-data-management/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-data-management/README.md b/metron-platform/metron-data-management/README.md index 1bb7994..014f3ef 100644 --- a/metron-platform/metron-data-management/README.md +++ b/metron-platform/metron-data-management/README.md @@ -159,12 +159,12 @@ As an example, we will be providing a CSV list of top domains as an enrichment a There are 2 property maps that work with full Stellar expressions, and 2 properties that will work with Stellar predicates. -| Property | Description | -|---------------------|-------------| -| value_transform | Transform fields defined in the "columns" mapping with Stellar transformations. New keys introduced in the transform will be added to the key metadata. | -| value_filter | Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose domain property is empty after removing the TLD will be omitted. | -| indicator_transform | Transform the indicator column independent of the value transformations. You can refer to the original indicator value by using "indicator" as the variable name, as shown in the example above. In addition, if you prefer to piggyback your transformations, you can refer to the variable "domain", which will allow your indicator transforms to inherit transformations done to this value during the value transformations. | -| indicator_filter | Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose indicator value is empty after removing the TLD will be omitted. | +| Property | Description +|---------------------|--- +| value_transform | Transform fields defined in the "columns" mapping with Stellar transformations. New keys introduced in the transform will be added to the key metadata. +| value_filter | Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose domain property is empty after removing the TLD will be omitted. +| indicator_transform | Transform the indicator column independent of the value transformations. You can refer to the original indicator value by using "indicator" as the variable name, as shown in the example above. In addition, if you prefer to piggyback your transformations, you can refer to the variable "domain", which will allow your indicator transforms to inherit transformations done to this value during the value transformations. +| indicator_filter | Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose indicator value is empty after removing the TLD will be omitted. top-list.csv ``` @@ -175,10 +175,10 @@ top-list.csv Running a file import with the above data and extractor configuration would result in the following 2 extracted data records: -| Indicator | Type | Value | -|-----------|------|-------| +| Indicator | Type | Value | +|-----------|-------------|---------------------------------------| | google | top_domains | { "rank" : "1", "domain" : "google" } | -| yahoo | top_domains | { "rank" : "2", "domain" : "yahoo" } | +| yahoo | top_domains | { "rank" : "2", "domain" : "yahoo" } | Similar to the parser framework, providing a Zookeeper quorum via the zk_quorum property will enable Stellar to access properties that reside in the global config. Expanding on our example above, if the global config looks as follows: @@ -196,15 +196,14 @@ And we expand our value_tranform: "a-new-prop" : "global_property" }, ... - ``` The resulting value data would look like the following: -| Indicator | Type | Value | -|-----------|------|-------| +| Indicator | Type | Value | +|-----------|-------------|--------------------------------------------------------------------| | google | top_domains | { "rank" : "1", "domain" : "google", "a-new-prop" : "metron-ftw" } | -| yahoo | top_domains | { "rank" : "2", "domain" : "yahoo", "a-new-prop" : "metron-ftw" } | +| yahoo | top_domains | { "rank" : "2", "domain" : "yahoo", "a-new-prop" : "metron-ftw" } | ## Enrichment Config @@ -303,19 +302,20 @@ each document to be considered as input to the Extractor. The parameters for the utility are as follows: -| Short Code | Long Code | Is Required? | Description | | -|------------|---------------------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---| -| -h | | No | Generate the help screen/set of options | | -| -q | --quiet | No | Do not update progress -| -e | --extractor_config | Yes | JSON Document describing the extractor for this input data source | | -| -m | --import_mode | No | The Import mode to use: LOCAL, MR. Default: LOCAL | | -| -t | --hbase_table | Yes | The HBase table to import into | | -| -c | --hbase_cf | Yes | The HBase table column family to import into | | -| -i | --input | Yes | The input data location on local disk. If this is a file, then that file will be loaded. If this is a directory, then the files will be loaded recursively under that directory. | | -| -l | --log4j | No | The log4j properties file to load | | -| -n | --enrichment_config | No | The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. | | -| -p | --threads | No | The number of threads to use when extracting data. The default is the number of cores. | | -| -b | --batchSize | No | The batch size to use for HBase puts | | +| Short Code | Long Code | Is Required? | Description | +|------------|---------------------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| -h | | No | Generate the help screen/set of options | +| -q | --quiet | No | Do not update progress | +| -e | --extractor_config | Yes | JSON Document describing the extractor for this input data source | +| -m | --import_mode | No | The Import mode to use: LOCAL, MR. Default: LOCAL | +| -t | --hbase_table | Yes | The HBase table to import into | +| -c | --hbase_cf | Yes | The HBase table column family to import into | +| -i | --input | Yes | The input data location on local disk. If this is a file, then that file will be loaded. If this is a directory, then the files will be loaded recursively under that directory. | +| -l | --log4j | No | The log4j properties file to load | +| -n | --enrichment_config | No | The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. | +| -p | --threads | No | The number of threads to use when extracting data. The default is the number of cores. | +| -b | --batchSize | No | The batch size to use for HBase puts | + ### GeoLite2 Loader The shell script `$METRON_HOME/bin/geo_enrichment_load.sh` will retrieve MaxMind GeoLite2 data and load data into HDFS, and update the configuration. @@ -324,10 +324,10 @@ THIS SCRIPT WILL NOT UPDATE AMBARI'S GLOBAL.JSON, JUST THE ZK CONFIGS. CHANGES The parameters for the utility are as follows: -| Short Code | Long Code | Is Required? | Description | -|------------|---------------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| -h | | No | Generate the help screen/set of options | -| -g | --geo_url | No | GeoIP URL - defaults to http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz -| -r | --remote_dir | No | HDFS directory to land formatted GeoIP file - defaults to /apps/metron/geo/\<epoch millis\>/ -| -t | --tmp_dir | No | Directory for landing the temporary GeoIP data - defaults to /tmp -| -z | --zk_quorum | Yes | Zookeeper Quorum URL (zk1:port,zk2:port,...) +| Short Code | Long Code | Is Required? | Description | +|------------|---------------------|--------------|--------------------------------------------------------------------------------------------------| +| -h | | No | Generate the help screen/set of options | +| -g | --geo_url | No | GeoIP URL - defaults to http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz | +| -r | --remote_dir | No | HDFS directory to land formatted GeoIP file - defaults to /apps/metron/geo/\<epoch millis\>/ | +| -t | --tmp_dir | No | Directory for landing the temporary GeoIP data - defaults to /tmp | +| -z | --zk_quorum | Yes | Zookeeper Quorum URL (zk1:port,zk2:port,...) | http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/site-book/bin/fix-md-dialect.py ---------------------------------------------------------------------- diff --git a/site-book/bin/fix-md-dialect.py b/site-book/bin/fix-md-dialect.py index d594ca2..5e6db3e 100755 --- a/site-book/bin/fix-md-dialect.py +++ b/site-book/bin/fix-md-dialect.py @@ -59,7 +59,7 @@ import inspect import re # These are the characters excluded by Markdown from use in auto-generated anchor text for Headings. -EXCLUDED_CHARS_REGEX = r'[()[\]`"' + r"'" + r']' # May add chars in future as needed +EXCLUDED_CHARS_REGEX = r'[^\w\-]' # all non-alphanumerics except "-" and "_". Whitespace are previously converted. def report_error(s) : print >>sys.stderr, "ERROR: " + s @@ -238,14 +238,16 @@ def rewrite_relative_links() : sharp = href.find("#") if (sharp >= 0) : named_anchor = href[sharp+1 : ] - scratch = labeltext.lower() - scratch = re.sub(r' ', "-", scratch) - scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch) + trace('named_anchor = "' + named_anchor + '"') + trace('labeltext = "' + labeltext + '"') + scratch = labeltext.lower() # Github-MD forces all anchors to lowercase + scratch = re.sub(r'[\s]', "-", scratch) # convert whitespace to "-" + scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch) # strip non-alphanumerics if (scratch == named_anchor) : trace("Found a rewritable case") - scratch = labeltext - scratch = re.sub(r' ', "_", scratch) - scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch) + scratch = labeltext # Doxia-markdown doesn't change case + scratch = re.sub(r'[\s]', "_", scratch) # convert whitespace to "_" + scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch) # strip non-alphanumerics href = re.sub("#" + named_anchor, "#" + scratch, href) trace("After anchor rewrite, href is: " + href)
