[
https://issues.apache.org/jira/browse/METRON-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961219#comment-15961219
]
ASF GitHub Bot commented on METRON-831:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
I spun this up in full-dev and did the following to test:
* Download the Alexa top 1m data set
```
wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
unzip top-1m.csv.zip
```
* Stage import file
```
head -n 10000 top-1m.csv > top-10k.csv
head -n 10 top-1m.csv > top-10.csv
hadoop fs -put top-10k.csv /tmp
```
* Create an extractor.json for the CSV data by editing `extractor.json` and
pasting in these contents:
```
{
"config" : {
"zk_quorum" : "node1:2181",
"columns" : {
"rank" : 0,
"domain" : 1
},
"value_transform" : {
"domain" : "DOMAIN_REMOVE_TLD(domain)",
"port" : "es.port"
},
"value_filter" : "LENGTH(domain) > 0",
"indicator_column" : "domain",
"indicator_transform" : {
"indicator" : "DOMAIN_REMOVE_TLD(indicator)"
},
"indicator_filter" : "LENGTH(indicator) > 0",
"type" : "top_domains",
"separator" : ","
},
"extractor" : "CSV"
}
```
* Import enriched data
`echo "truncate 'enrichment'" | hbase shell &&
$METRON_HOME/bin/flatfile_loader.sh -i ./top-10k.csv -t enrichment -c t -e
./extractor.json -p 5 -b 128 && echo "count 'enrichment'" | hbase shell`
* Open up the stellar shell via `$METRON_HOME/bin/stellar -z node1` and
execute the following:
```
MAP(['google', 'pdf2doc', 'yahoo'], indicator -> MAP_GET('domain',
ENRICHMENT_GET('top_domains', indicator, 'enrichment', 't')) )
```
You should see `[google, pdf2doc, yahoo]` returned.
> Add lambda expressions and rudimentary functional programming primitives to
> Stellar
> -----------------------------------------------------------------------------------
>
> Key: METRON-831
> URL: https://issues.apache.org/jira/browse/METRON-831
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
>
> To enable use-cases where the user may want to do simple iteration, we can
> expose simple functional primitives such as:
> * MAP
> * REDUCE
> * FILTER
> To fully support this, we will need lambda expressions as arguments to these
> functions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
