[
https://issues.apache.org/jira/browse/METRON-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854144#comment-15854144
]
ASF GitHub Bot commented on METRON-682:
---------------------------------------
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/432
METRON-682: Unify and Improve the Flat File Loader
Currently the flat file loader is deficient in a couple ways:
* It only supports importing local data despite there being a separate,
poorly named, application which supports importing enrichment via MapReduce
called threat_intel_loader.sh
* It does not support local imports from HDFS
* It does not support local imports from URLs
* It does not support importing zipped archives locally
* You cannot import more than one file at once
This JIRA will:
* Unify the MapReduce and local imports into one program and allow the user
to specify the import mode with a CLI flag
* Support local imports from HDFS and URLs
* Support local imports from zipped files
* Support importing more than one file at once
Testing plan in comments
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron unified_loader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #432
----
commit 47d814ef95d67738d20ce5dc530ba7b05d418a96
Author: cstella <[email protected]>
Date: 2017-01-27T23:15:44Z
Multithreading the SimpleEnrichmentFlatFileLoader
commit 918d4ce4aea5d7dfde992f32bf049c70f35dd182
Author: cstella <[email protected]>
Date: 2017-01-27T23:23:19Z
doc changes.
commit c6ca3a86881eb77bc9598a61e3c0cf8280ccb03f
Author: cstella <[email protected]>
Date: 2017-01-27T23:39:56Z
Updating docs.
commit 8c9a79cdfa38ea2fbd161095d5e346147558ec5f
Author: cstella <[email protected]>
Date: 2017-01-28T03:36:31Z
Investigating integration tests.
commit 315bd181aa634290ab987441d81c28addb7952e2
Author: cstella <[email protected]>
Date: 2017-01-28T04:09:28Z
Update integration test to be a proper integration test.
commit 004c6f41b6c1cc3ecea70513e1a468501bd32e3c
Author: cstella <[email protected]>
Date: 2017-01-28T04:49:37Z
Adding spliterator unit test for completeness
commit f8dd48ef920c948e1fc5ff736e386f641e551b2b
Author: cstella <[email protected]>
Date: 2017-01-28T05:01:42Z
Updating test to use a proper file
commit 9b04f9723d442c8f4fb7a8bcaa1d733fc1305dc4
Author: cstella <[email protected]>
Date: 2017-01-28T05:17:12Z
Updating docs and renaming a few things.
commit eb5b82cc35bd767a169f548ea8144dd9ae165f84
Author: cstella <[email protected]>
Date: 2017-01-28T05:23:25Z
Update one more test case.
commit 310c98bd946b2fdb320193cce85d368f016bf8c3
Author: cstella <[email protected]>
Date: 2017-01-30T20:36:23Z
Merge branch 'master' into unified_loader
commit 3f6e3ba4f30e41c94ff25027f1fd7c839ea6c9bf
Author: cstella <[email protected]>
Date: 2017-01-31T15:39:03Z
Updating simple enrichment flat file loader to be complete.
commit 79cfdb4fba5e82e9e170bfc77c7133e6646f9787
Author: cstella <[email protected]>
Date: 2017-01-31T22:12:05Z
Removing old threatintel_bulk_load.sh script and integrating into the
flatfile load script
commit bf7756b52e66907ca23a576ba9be9ab40b33f77d
Author: cstella <[email protected]>
Date: 2017-01-31T22:22:17Z
Forgot licenses.
commit a104f464e6b882121c7ab44079a5570d282c8457
Author: cstella <[email protected]>
Date: 2017-02-01T00:28:46Z
updating script.
commit b121e13d892834865847ddd806cbf10da63fa44e
Author: cstella <[email protected]>
Date: 2017-02-01T00:34:28Z
Merge branch 'master' into unified_loader
commit b5a9e5a9243576b27d59e959dfab3e99d34eb761
Author: cstella <[email protected]>
Date: 2017-02-01T00:57:02Z
Added gzip and zip to regular files
commit 323267ddfb52ab1aa7488e02643a8158044797e2
Author: cstella <[email protected]>
Date: 2017-02-01T15:04:53Z
Fixed stupid zip issue.
commit bc26b5b3992b91097bb4fc4b214d4b6bacaddfbb
Author: cstella <[email protected]>
Date: 2017-02-01T16:27:58Z
Updating readme and making progress bar optional and better.
commit 6cdf35d94f72be7da524fd5f854876f131ddb9f9
Author: cstella <[email protected]>
Date: 2017-02-01T17:39:59Z
updating tests to include gzip and zip
commit 799811ccd587d9fcd2135ea3970d5bbf1121a29b
Author: cstella <[email protected]>
Date: 2017-02-03T22:21:34Z
Reacted to @mmiklavcic
----
> Unify and Improve the Flat File Loader
> --------------------------------------
>
> Key: METRON-682
> URL: https://issues.apache.org/jira/browse/METRON-682
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
>
> Currently the flat file loader is deficient in a couple ways:
> * It only supports importing local data despite there being a separate,
> poorly named, application which supports importing enrichment via MapReduce
> called threat_intel_loader.sh
> * It does not support local imports from HDFS
> * It does not support local imports from URLs
> * It does not support importing zipped archives locally
> * You cannot import more than one file at once
> This JIRA will:
> * Unify the MapReduce and local imports into one program and allow the user
> to specify the import mode with a CLI flag
> * Support local imports from HDFS and URLs
> * Support local imports from zipped files
> * Support importing more than one file at once
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
