Interest in extending to Cascading?

Cascading is a higher level API for Hadoop-mapreduce, Tez and Flink. The Pirk roadmap mentions support for a number of other frameworks (Flink and Storm being two), this would take care of Flink and add Tez support as well. If there's interest I'll add a JIRA and link other issues accordingly.

Re: Next short term goal?

Regarding (ii) I think docker-compose might help with distributed testing for development and Terraform scripts for benchmark work in AWS/GCE. I might be doing this soon - if interested I can share both. On Sep 14, 2016 8:06 AM, "Tim Ellison" wrote: > I'm reluctant to

Re: Next short term goal?

I'm actually looking at (I) now. I'm starting small with pirk-63. A WIP PR should be in late tonight. This will keep the driver from needing all framework dependencies, but we'll need to discuss passing framework specific options as well. I could definitely see the use case for separate

Re: Next short term goal?

want to contribute to Pirk (who don't > work > > on these systems often) to be able to get up and running quickly to test > > their improvements and additions to the codebase. > > > > On Wed, Sep 14, 2016 at 8:46 AM, Darin Johnson <dbjohnson1...@gmail.com> > > wrote: >

Re: Next short term goal?

ime consuming to get through the first time. > > The goal is for smart people who want to contribute to Pirk (who don't work > on these systems often) to be able to get up and running quickly to test > their improvements and additions to the codebase. > > On Wed, Sep 14, 2016 at 8:46 AM, Dar

Making ResponderDriver more general

Hey guys, I was looking into creating my own responder as a general exercise but as the jar was getting pretty big I thought it might be useful to first create a modular build as someone using hadoop not would want to push around storm dependencies and vice versa. As I was scoping this I noticed

Re: Distributed tests take a long time, and that's ok.

Being pretty familiar with that section of the code base right now. I think a few things could be done. Regardless I'm refactoring it for submodules do it'll be easier to iterate these. The main place I'd start is currently each platform test creates a new input, which means the querier logic is

Re: [GitHub] incubator-pirk pull request #106: [PIRK-73] - Change All Properties in Respo...

Being pretty familiar with that section of the code base right now. I think a few things could be done. Regardless I'm refactoring it for submodules do it'll be easier to iterate these. The main place I'd start is currently each platform test creates a new input, which means the querier logic is

Re: [GitHub] incubator-pirk pull request #106: [PIRK-73] - Change All Properties in Respo...

Sorry wrong thread disregard. On Oct 6, 2016 6:37 PM, "Darin Johnson" <dbjohnson1...@gmail.com> wrote: > Being pretty familiar with that section of the code base right now. I > think a few things could be done. Regardless I'm refactoring it for > submodules do i

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Hey guys, Thanks for looking at the PR, I apologize if it offended anyone's eyes:). I'm glad it generated some discussion about the configuration. I didn't really like where things were heading with the config. However, didn't want to create to much scope creep. I think any hierarchical

Re: Google docs

21/09/16 13:13, Suneel Marthi wrote: > > On Wed, Sep 21, 2016 at 1:10 PM, Tim Ellison <t.p.elli...@gmail.com> > wrote: > > > >> On 21/09/16 11:36, Suneel Marthi wrote: > >>> On Wed, Sep 21, 2016 at 10:43 AM, Tim Ellison <t.p.elli...@gmail.com> > &

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

PM, Suneel Marthi <suneel.mar...@gmail.com> wrote: > A shared Google doc would be more convenient than a bunch of Jiras. Its > easier to comment and add notes that way. > > > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson <dbjohnson1...@gmail.com> > wrote: > >

Re: Pirk Submodule Refactor

will be least intrusive and easiest to maintain. Darin On Sep 15, 2016 6:17 PM, "Ellison Anne Williams" <eawilli...@apache.org> wrote: > On Thu, Sep 15, 2016 at 9:25 AM, Tim Ellison <t.p.elli...@gmail.com> > wrote: > > > On 15/09/16 09:21, Darin Johnson wrote: > &

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

ll gain some ability to generate multiple jars which is nice. > > > > > > > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <t.p.elli...@gmail.com> > > wrote: > > > > > On 19/09/16 15:46, Darin Johnson wrote: > > > > Hey guys, > > &g

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

a command line option for the launcher for now as we don't have submodules we can decide between the two choices after we break out submodules and improve the config. On Sep 19, 2016 12:19 PM, "Tim Ellison" <t.p.elli...@gmail.com> wrote: > On 19/09/16 15:46, Darin Johnson

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Sure will do tonight. On Sep 19, 2016 5:07 PM, "Suneel Marthi" <suneel.mar...@gmail.com> wrote: > A shared Google doc would be more convenient than a bunch of Jiras. Its > easier to comment and add notes that way. > > > On Mon, Sep 19, 2016 at 10:38 PM, Darin J

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

t; > > > > > > On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi < > suneel.mar...@gmail.com > > > > > > wrote: > > > > > > > A shared Google doc would be more convenient than a bunch of Jiras. > Its > > > > easier to com

Re: Next Release

I think it's very possible I can have things done by the end of next week. There may be some minor issues that may move part of it into next week. I'm +1 (non binding) for 0.2.0. Darin On Sep 24, 2016 2:58 PM, "Ellison Anne Williams" wrote: > Hi Guys, > > In the spirit

Re: Some Obversations while refactoring into submodules

and hasn't had a good refactoring yet. ;) > > What you suggested sounds reasonable - look forward to seeing the code. > > On Thu, Sep 29, 2016 at 4:48 AM, Tim Ellison <t.p.elli...@gmail.com> > wrote: > > > On 29/09/16 01:50, Darin Johnson wrote: > > > Hey guy

Re: Some Obversations while refactoring into submodules

ibutedTestSuite is ripe for refactor - it grew >> pretty organically and hasn't had a good refactoring yet. ;) >> >> What you suggested sounds reasonable - look forward to seeing the code. >> >> On Thu, Sep 29, 2016 at 4:48 AM, Tim Ellison <t.p.elli...@gmail.com>

Re: [Discuss] Submodules (Phase 2) observations

propriately. > > On Thu, Oct 20, 2016 at 12:22 AM, Darin Johnson <dbjohnson1...@gmail.com> > wrote: > > > I threw up a WIP for phase 1 of the submodule refactor. This involved > > pulling out the pieces for the storm, spark, and mapreduce responsers. > > M

[Discuss] Submodules (Phase 2) observations

I threw up a WIP for phase 1 of the submodule refactor. This involved pulling out the pieces for the storm, spark, and mapreduce responsers. Mostly ran into some difficulties of hard coded conditionals for each framework in DistributedTestSuite and/or BaseTests. The next logical module to pull

Re: [Discuss] Submodules (Phase 2) observations

leaving the Responder refactor (and test > refactor) to a subsequent PR makes more sense. > > Absent a complete test refactor, what are your thoughts on the best path > forward? > > Thanks- > > Ellison Anne > > On Thu, Oct 27, 2016 at 5:33 PM, Darin Johnson <dbjohnson1

PR 119

Hey guys, I updated PR 119 per Ellison Anne's requests, it's here: https://github.com/apache/incubator-pirk/pull/119. I'd like to iterate on it some more and work on configuration issues. It's definitely not perfect, but I think if we're going to do submodules this is a good place to start.