To me Metron is big and broad in the scope of technology required to get it
running. If things were more modular that would go a long way to reducing the
learning curve or at least putting it into smaller bites (and it might
encourage more people to get involved).
If the UI were an add-on module in another project, it would have made it
easier for me and it could also encourage my hypothetical buddy who is a web
developer expert to get involved since he could focus on the web-ui module
instead of trying to tackle all the other pieces that are probably not part of
his bailiwick.
Stellar is very intriguing, maybe that is not unique to Metron? The
architecture of Metron with respect to parsing, enriching, etc., makes a lot of
sense to anyone I talk with. These two aspects of Metron seem like standout
examples that make for a powerful platform to develop on.
Thanks for continuing this discussion,
Tom.
On 2020-04-08 15:32:46-07:00 Casey Stella wrote:
As far as I know there is no minimum bar of development activity to keep a
project open. I think we would all be grateful for any investment that you or
your organization would want to make.
It also occurs to me that your observation is absolutely spot on: we have a LOT
of moving parts.
I see some deficiencies here:
* We depend on a lot of the various hadoop ecosystem projects and they have
to work together very precisely:
* This makes for a system that is hard to install.
* This also makes for a system which is hard to tune/manage
* We have a large surface area of coverage
* We have an installer, backend system and front-end UI, which stretches
our developers a bit thin, especially since there isn't even interest in those
systems
Perhaps a reconsideration of the scope and technologies that we use would be
merited? If we were to decide to, for instance:
* Consolidate scope: focus on a viable backend/API rather than a UI
* Consolidate technology: reposition ourselves on top of Spark as a
consolidated streaming/batch system
* Make SQL our external interface: write out to parquet + the Hive
metastore and let users pin up presto tables or hive tables as they see fit
This might reduce some of our surface area and make it more viable to get
started?
Anyway, just some thoughts.
Casey
On Wed, Apr 8, 2020 at 6:20 PM Yerex, Tom
<tom.ye...@ubc.ca<mailto:tom.ye...@ubc.ca>> wrote:
Hi Casey,
I'm new here and new to contributing to an open source project. Thus far my
contribution has been questions, however the steep learning curve has had me
working to understand all the moving parts for the last 18 months and I see
that as a big investment by my organization.
What is a level that would be viable?
If my organization were to contribute I don't know that it would be soon enough
or at the volume that is recognized as viable, which is why I ask the question.
On 2020-04-08 15:05:51-07:00 Casey Stella wrote:
Hi all,
When composing the board report today, I realized that we have effectively
had no development in the last quarter on this project. Please be aware
that I say this without a shred of blame or judgement (especially so
considering I have not contributed in a long time). That being said, I
would like to pose the question to the community:
Do we feel that this project is viable? If so, how are we going to spur
new contributions? If not, then should we begin the process to fold the
project?
Best,
Casey
