[jira] [Commented] (METRON-678) Multithread the flat file loader

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843749#comment-15843749
 ] 

ASF GitHub Bot commented on METRON-678:
---

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/428


> Multithread the flat file loader
> 
>
> Key: METRON-678
> URL: https://issues.apache.org/jira/browse/METRON-678
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Currently the flat file loader is single threaded in its writing to HBase.  
> We could make this a lot faster by multithreading the HBase puts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-678) Multithread the flat file loader

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843683#comment-15843683
 ] 

ASF GitHub Bot commented on METRON-678:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/428
  
Just as a comment: on my vagrant box, importing the alexa 1m took 4 minutes 
with 6 threads and 14 minutes with 1 thread.


> Multithread the flat file loader
> 
>
> Key: METRON-678
> URL: https://issues.apache.org/jira/browse/METRON-678
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Currently the flat file loader is single threaded in its writing to HBase.  
> We could make this a lot faster by multithreading the HBase puts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-678) Multithread the flat file loader

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843663#comment-15843663
 ] 

ASF GitHub Bot commented on METRON-678:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/428
  
Note, the batch size is just the split size (number of lines processed in 
each split) for the spliterator.  The default of 128 is probably pretty good 
for almost all cases, I think.


> Multithread the flat file loader
> 
>
> Key: METRON-678
> URL: https://issues.apache.org/jira/browse/METRON-678
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Currently the flat file loader is single threaded in its writing to HBase.  
> We could make this a lot faster by multithreading the HBase puts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-678) Multithread the flat file loader

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843657#comment-15843657
 ] 

ASF GitHub Bot commented on METRON-678:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/428
  
Testing Plan

* Download the alexa 1m dataset:
```
wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
unzip top-1m.csv.zip
```
* Create a 100k and single entry selection:
```
head -n 10 top-1m.csv > top-100k.csv
head -n 1 top-1m.csv > top-1.csv
```
* Create an extractor.json for the CSV data by editing `extractor.json` and 
pasting in these contents:
```
{
  "config" : {
"columns" : {
   "domain" : 1,
   "rank" : 0
}
,"indicator_column" : "domain"
,"type" : "alexa"
,"separator" : ","
 },
  "extractor" : "CSV"
}
```
* Verify 100k import with 5 threads:
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase using 5 threads
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-100k.csv -t enrichment -c 
t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 100k
echo "count 'enrichment'" | hbase shell
```
* Verify 100k import with 5 threads and a batch of 1000:
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase using 5 threads
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-100k.csv -t enrichment -c 
t -e ./extractor.json -p 5 -b 1000
# count data written and verify it's 100k
echo "count 'enrichment'" | hbase shell
```
* Verify 100k import with 1 threads:
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase using 5 threads
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-100k.csv -t enrichment -c 
t -e ./extractor.json -p 1 -b 128
# count data written and verify it's 100k
echo "count 'enrichment'" | hbase shell
```
* Verify 1 entry import with 5 threads:
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase using 5 threads
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-1.csv -t enrichment -c t 
-e ./extractor.json -p 5 -b 128
# count data written and verify it's 1
echo "count 'enrichment'" | hbase shell
```
* Verify 1 entry import with 1 threads:
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase using 5 threads
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-1.csv -t enrichment -c t 
-e ./extractor.json -p 1 -b 128
# count data written and verify it's 1
echo "count 'enrichment'" | hbase shell
```


> Multithread the flat file loader
> 
>
> Key: METRON-678
> URL: https://issues.apache.org/jira/browse/METRON-678
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Currently the flat file loader is single threaded in its writing to HBase.  
> We could make this a lot faster by multithreading the HBase puts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-678) Multithread the flat file loader

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843645#comment-15843645
 ] 

ASF GitHub Bot commented on METRON-678:
---

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/428

METRON-678: Multithread the flat file loader

Currently the flat file loader is single threaded in its writing to HBase. 
We could make this a lot faster by multithreading the HBase puts.

Executing this on single node vagrant with a batch size of 128 and a number 
of threads varying between 1 and 6 for a 2 column CSV enrichment, a reasonable 
speedup was achieved:

1. 91.019 seconds
2. 76.07 seconds
3. 39.974 seconds
4. 35.039 seconds
5. 30.531 seconds
6. 30.559 seconds


![chart](https://cloud.githubusercontent.com/assets/540359/22391274/507f08a6-e4be-11e6-9f9a-19604791cf37.png)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron parallel_extractor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/428.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #428


commit 47d814ef95d67738d20ce5dc530ba7b05d418a96
Author: cstella 
Date:   2017-01-27T23:15:44Z

Multithreading the SimpleEnrichmentFlatFileLoader

commit 918d4ce4aea5d7dfde992f32bf049c70f35dd182
Author: cstella 
Date:   2017-01-27T23:23:19Z

doc changes.




> Multithread the flat file loader
> 
>
> Key: METRON-678
> URL: https://issues.apache.org/jira/browse/METRON-678
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Currently the flat file loader is single threaded in its writing to HBase.  
> We could make this a lot faster by multithreading the HBase puts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (METRON-678) Multithread the flat file loader

2017-01-27 Thread Casey Stella (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Casey Stella updated METRON-678:

Description: Currently the flat file loader is single threaded in its 
writing to HBase.  We could make this a lot faster by multithreading the HBase 
puts.

> Multithread the flat file loader
> 
>
> Key: METRON-678
> URL: https://issues.apache.org/jira/browse/METRON-678
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Currently the flat file loader is single threaded in its writing to HBase.  
> We could make this a lot faster by multithreading the HBase puts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (METRON-678) Multithread the flat file loader

2017-01-27 Thread Casey Stella (JIRA) 
Casey Stella created METRON-678:
---

 Summary: Multithread the flat file loader
 Key: METRON-678
 URL: https://issues.apache.org/jira/browse/METRON-678
 Project: Metron
  Issue Type: Improvement
Reporter: Casey Stella
Assignee: Casey Stella






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (METRON-677) FILL_LEFT and FILL_RIGHT not working

2017-01-27 Thread sapinkumar amin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sapinkumar amin updated METRON-677:
---
Description: 
FILL_LEFT not working

{code}
[Stellar]>>> FILL_LEFT('123','X', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.dsl.functions.StringFunctions$FillLeft.apply(StringFunctions.java:246)
at 
org.apache.metron.common.dsl.BaseStellarFunction.apply(BaseStellarFunction.java:30)
at 
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
at 
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at 
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92)
... 8 more
{code}


---

FILL_RIGHT not working
{code}
[Stellar]>>> FILL_RIGHT('123','X', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError:

[jira] [Updated] (METRON-677) FILL_LEFT and FILL_RIGHT not working

2017-01-27 Thread sapinkumar amin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sapinkumar amin updated METRON-677:
---
Description: 
FILL_LEFT not working

{code}
[Stellar]>>> FILL_LEFT('123','', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.dsl.functions.StringFunctions$FillLeft.apply(StringFunctions.java:246)
at 
org.apache.metron.common.dsl.BaseStellarFunction.apply(BaseStellarFunction.java:30)
at 
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
at 
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at 
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92)
... 8 more

{code}


---

FILL_RIGHT not working
{code}
[Stellar]>>> FILL_RIGHT('123','', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError:

[jira] [Updated] (METRON-677) FILL_LEFT and FILL_RIGHT not working

2017-01-27 Thread sapinkumar amin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sapinkumar amin updated METRON-677:
---
Description: 
FILL_LEFT not working

{code}
[Stellar]>>> FILL_LEFT('123','', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.dsl.functions.StringFunctions$FillLeft.apply(StringFunctions.java:246)
at 
org.apache.metron.common.dsl.BaseStellarFunction.apply(BaseStellarFunction.java:30)
at 
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
at 
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at 
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92)
... 8 more

{/code}


---

FILL_RIGHT not working
{code}
[Stellar]>>> FILL_RIGHT('123','', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError:

[jira] [Created] (METRON-677) FILL_LEFT and FILL_RIGHT not working

2017-01-27 Thread sapinkumar amin (JIRA) 
sapinkumar amin created METRON-677:
--

 Summary: FILL_LEFT and FILL_RIGHT not working
 Key: METRON-677
 URL: https://issues.apache.org/jira/browse/METRON-677
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.3.0
 Environment: centos 6,7 . 
java version "1.8.0_60"
Reporter: sapinkumar amin
 Fix For: 0.3.0


FILL_LEFT not working


[Stellar]>>> FILL_LEFT('123','', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.dsl.functions.StringFunctions$FillLeft.apply(StringFunctions.java:246)
at 
org.apache.metron.common.dsl.BaseStellarFunction.apply(BaseStellarFunction.java:30)
at 
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
at 
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at 
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
at 
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
at 
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92)
... 8 more




---

FILL_RIGHT not working

[Stellar]>>> FILL_RIGHT('123','', 10)
[!] Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
org.apache.metron.common.dsl.ParseException: Unable to execute: 
org.apache.metron.common.dsl.functions.StringFunctions.access$000(Lorg/apache/metron/common/dsl/functions/StringFunctions$FillDirection;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at 
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:93)
at 
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:253)
at 
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:290)
at 
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:193)
at 
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:329)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at

[jira] [Commented] (METRON-503) Metron REST API

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843564#comment-15843564
 ] 

ASF GitHub Bot commented on METRON-503:
---

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/316
  
Conflict is resolved


> Metron REST API
> ---
>
> Key: METRON-503
> URL: https://issues.apache.org/jira/browse/METRON-503
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
> Attachments: Metron REST API.docx
>
>
> As discussed on the dev list ([DISCUSS] Metron REST API Requirements), this 
> Jira includes adding a REST API to Metron.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-676) Create Zeppelin Notebook for YAF Telemetry

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843189#comment-15843189
 ] 

ASF GitHub Bot commented on METRON-676:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/427
  
Here are the links to screen captures of what the dashboard looks-like.
* [Report, read-only 
mode](https://issues.apache.org/jira/secure/attachment/12849721/Metron%20-%20YAF%20Telemetry%20-%20Report%20Mode.png)
* [Edit 
mode](https://issues.apache.org/jira/secure/attachment/12849578/Metron%20-%20YAF%20Telemetry.png)


> Create Zeppelin Notebook for YAF Telemetry
> --
>
> Key: METRON-676
> URL: https://issues.apache.org/jira/browse/METRON-676
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.3.1
>
> Attachments: Metron - YAF Telemetry.png, Metron - YAF Telemetry - 
> Report Mode.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (METRON-676) Create Zeppelin Notebook for YAF Telemetry

2017-01-27 Thread Nick Allen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-676:
--
Attachment: Metron - YAF Telemetry - Report Mode.png

What the notebook looks-like in report mode, which is much cleaner, read-only 
view of the same notebook.

> Create Zeppelin Notebook for YAF Telemetry
> --
>
> Key: METRON-676
> URL: https://issues.apache.org/jira/browse/METRON-676
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.3.1
>
> Attachments: Metron - YAF Telemetry.png, Metron - YAF Telemetry - 
> Report Mode.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-503) Metron REST API

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843128#comment-15843128
 ] 

ASF GitHub Bot commented on METRON-503:
---

Github user jjmeyer0 commented on the issue:

https://github.com/apache/incubator-metron/pull/316
  
@merrimanr looks like you will need to resolve some conflicts.

Also, I looked at this some more. It looks like there aren't many unit 
tests for the service layer. It seems like it's indirectly tested through the 
controllers, but I think there is still some benefit to having unit tests 
against the services themselves. Before I made this comment I wanted to work 
through an example, and I created a test for `KafkaServiceImpl`. If you want to 
see that I'd be more than glad to share. Otherwise, I was just thinking we 
could inject all dependencies as mocks using PowerMock or Mockito. Then 
continue to test normally. This takes things like spring and any external 
system out of the picture and focuses just on the implementation of the 
services.


> Metron REST API
> ---
>
> Key: METRON-503
> URL: https://issues.apache.org/jira/browse/METRON-503
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
> Attachments: Metron REST API.docx
>
>
> As discussed on the dev list ([DISCUSS] Metron REST API Requirements), this 
> Jira includes adding a REST API to Metron.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-627) Add HyperLogLogPlus implementation to Stellar

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843087#comment-15843087
 ] 

ASF GitHub Bot commented on METRON-627:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/397


> Add HyperLogLogPlus implementation to Stellar
> -
>
> Key: METRON-627
> URL: https://issues.apache.org/jira/browse/METRON-627
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
> Fix For: 0.3.1
>
>
> Calculating set cardinality can be a useful tool for a security analyst. For 
> instance, a large volume of non-unique src ip addresses hitting your network 
> may be an indication that you are currently under attack. There have been 
> many advancements in distinct value (DV) estimation over the years. We have 
> seen implementations evolve from K-Minimum-Values (KMV), to LogLog, to 
> HyperLogLog, and now to Google's much-improved HyperLogLogPlu algorithm. The 
> key improvements in this latest manifestation of the algorithm are:
> moves to a 64-bit hash
> handles sparse sets
> is more accurate with small cardinality
> This Jira tracks the effort to add a HyperLogLogPlus implementation to Metron.
> References:
> https://research.neustar.biz/2013/01/24/hyperloglog-googles-take-on-engineering-hll/
> http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (METRON-627) Add HyperLogLogPlus implementation to Stellar

2017-01-27 Thread Casey Stella (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Casey Stella updated METRON-627:

Fix Version/s: 0.3.1

> Add HyperLogLogPlus implementation to Stellar
> -
>
> Key: METRON-627
> URL: https://issues.apache.org/jira/browse/METRON-627
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
> Fix For: 0.3.1
>
>
> Calculating set cardinality can be a useful tool for a security analyst. For 
> instance, a large volume of non-unique src ip addresses hitting your network 
> may be an indication that you are currently under attack. There have been 
> many advancements in distinct value (DV) estimation over the years. We have 
> seen implementations evolve from K-Minimum-Values (KMV), to LogLog, to 
> HyperLogLog, and now to Google's much-improved HyperLogLogPlu algorithm. The 
> key improvements in this latest manifestation of the algorithm are:
> moves to a 64-bit hash
> handles sparse sets
> is more accurate with small cardinality
> This Jira tracks the effort to add a HyperLogLogPlus implementation to Metron.
> References:
> https://research.neustar.biz/2013/01/24/hyperloglog-googles-take-on-engineering-hll/
> http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-676) Create Zeppelin Notebook for YAF Telemetry

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842900#comment-15842900
 ] 

ASF GitHub Bot commented on METRON-676:
---

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/427
  
@nickwallen #423 is in (and breaking the README here)


> Create Zeppelin Notebook for YAF Telemetry
> --
>
> Key: METRON-676
> URL: https://issues.apache.org/jira/browse/METRON-676
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.3.1
>
> Attachments: Metron - YAF Telemetry.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-270) Add Zeppelin to the platform

2017-01-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842898#comment-15842898
 ] 

ASF GitHub Bot commented on METRON-270:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/423


> Add Zeppelin to the platform
> 
>
> Key: METRON-270
> URL: https://issues.apache.org/jira/browse/METRON-270
> Project: Metron
>  Issue Type: New Feature
>Reporter: James Sirota
>Assignee: Justin Leet
>  Labels: METRON_ML
> Fix For: 0.3.1
>
>
> I propose adding Zeppelin to the platform to aid in interactive dashboarding 
> and data visualizations 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)