[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1019 Thanks for the adjustments @merrimanr. Looks good, +1. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1019 @merrimanr Can you kick Travis? ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1019 I'm good with this right now with the follow-on changes and tests, assuming @mmiklavc is good. I do have the same concerns Mike had, but I think being watchful going forward is a good compromise, but that involves a bit of vigilance here. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1019 How about starting with 1 of those, fixed, and maybe making it synchronous in the first pass so you can integration test? I'd consider many of those additional api endpoints as features to be built out in this feature branch. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1019 Sure. In that case we need to break up https://issues.apache.org/jira/browse/METRON-1559 into smaller tasks which we need to do that anyways. Which endpoints would you include in a basic PcapService? Maybe just these 2: - POST /api/v1/pcap/fixed - POST /api/v1/pcap/query We could manually test in full dev by inspecting results in HDFS. I still think a basic PcapService with a couple endpoints is going to require a large PR. Just want to make sure you're ok with that. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1019 I guess I'm saying combine this with a basic PcapService that has an api somewhat flushed out - not necessarily final, but close enough to start filling in implementation details when new features/PR's with those features are added. It's effectively what you have already with just a bit more of the service implemented/defined. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1019 This was my attempt to split it up into a more manageable review. I think this will end up being a large PR but that's fine with me as long as everyone else is ok with it. Happy to keep going. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1019 @merrimanr Thanks for adding the discuss thread links. It sounds like this doesn't do much aside from a pcap service. Why not just handle them together? That way we're not dealing with TODO's and extra lists, and together they would form a concise unit of work. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1019 I added a link to the discuss thread to the description. There is also a link in the top-level Jira: https://issues.apache.org/jira/browse/METRON-1554. The latest commit removes the Service classes so that only a sample controller remains. I added a comment to that controller stating it should be replaced. There was a Jira created for service development and API design: https://issues.apache.org/jira/browse/METRON-1559. Hopefully that makes it clearer that API design is out of scope for this PR. I can also add a comment to that Jira once this PR gets accepted. I feel like spinning up and testing this in full dev should be part of the review. Most of the changes here are Ambari and Maven related. To test this properly there will need to be some way to submit a MR job (any job) from REST, hence the throwaway sample controller. If you want to create your own sample controller and test that way I can remove the one in this PR. Is there an easier option I'm not thinking of? Maintaining a needs addressed list sounds like a good idea. I'm not clear on how this would work in practice though. Where would we keep it? Would Jira tasks serve this purpose? This seems like it could be applicable to all tasks in this feature branch (all feature branches really) so I don't think this is the appropriate place to discuss it. Can we move it back to a discuss thread so you can clarify and expand on your idea? I think it needs some more thought if we're to adopt it as a process. ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1019 Gotcha, GitHub mobile seems to collapse that information. ð ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1019 This pr is against the feature branch @JonZeolla so, it is not in play ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1019 Based on the temporary/example state of some of this PR would it make sense to explicitly keep this one out of the upcoming release, regardless of the review cycle? ---
