I think the feature branch is a good idea, but what is in the feature
branch or feature branches will have to shake out.
I agree in concept with what you have in the jira, but I have two points.
1. We will need a break down of introducing Spark to the stack
- required version due to HDP support
- do we want to update HDP support before this?
- Spark tuning/defaults
- Spark configuration support / UI etc
- more….
2.
When I read this, it seems like a Lambda architecture approach. Should
we, as part of this start exploring the possibility to replacing storm with
spark streaming such that we do not have to maintain separate streaming vs.
batch codebases?
3. This mechanism would be used in the future for telemetry ‘replay’.
That would mean that ( IMHO )
- we should understand that case as well for this
- build this capability out such that it is generic enough that a
second use will not warrant a re-write or huge refactor
I think this breaks down to a few sets of functionality:
-
Base support for deployment, management or spark
-
Metron services for triggering, and monitoring of Apache Spark ( on
demand and constant ), maybe rest stuff like the caps
-
UI / Stellar base support
-
Build out of Batch Profiler service on top of that
-
Build out of replay service on top of that ( plus all the replay stuff
that needs to also be done - like are you replacing data or having two
sets…. trial runs etc )
-
????
-
profit
On July 27, 2018 at 11:29:51, Nick Allen (n...@nickallen.org) wrote:
Hi Everyone -
A while back I opened up a discuss thread around the general idea of a
Batch Profiler [1]. I'd like to start making progress on a first draft of
that functionality.
I created METRON-1699 [2] which outlines the general approach and ideas.
If you're interested, review that JIRA and let me know if you have
feedback. I will be adding sub-tasks to that JIRA as I make progress and
can separate it into logical bits for review.
I would like this effort to use a feature branch as it will take a number
of PRs to get a first cut on the functionality. Pending no disagreement, I
will create the feature branch based on METRON-1699.
[1]
https://lists.apache.org/thread.html/d28d18cc9358f5d9c276c7c304ff4ee601041fb47bfc97acb6825083@%3Cdev...
<
https://lists.apache.org/thread.html/d28d18cc9358f5d9c276c7c304ff4ee601041fb47bfc97acb6825083@%3Cdev.metron.apache.org%3E>
[2] https://issues.apache.org/jira/browse/METRON-1699
