impala support json format table

2017-04-17 Thread yu feng 
Hi impala community:
  I am Newly join to Impala, and In our scenario, we have lots of tables
shared by hive and impala, their storage format are various, such as
parquet/textfile/textfile with json/orc ..., In our plan, we need support
textfile with json at least;, I want to know what is the attitude of impala
community for supporting json format. If this match the roadmap, maybe I
can make some contribution.

By the way, Where can I find the newest roadmap of impala?

Look forward to your reply.

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Tim Armstrong 
Makes sense to me

On Mon, Apr 17, 2017 at 3:11 PM, Jim Apple  wrote:

> I'm OK with us calling this "experimental" (once the patches are in and we
> think it works) and letting patches continue to land with our current
> testing regime.
>
> Later, if we add daily test reports with contributor-managed hardware or
> even pre-commit testing on jenkins.impala.io in a VM, maybe we could
> upgrade to other names than "experimental", like "beta" or "best-effort",
> if we still want to have caveats.
>
> How does that sound to everyone else?
>
> On Mon, Apr 17, 2017 at 2:54 PM, Tim Armstrong 
> wrote:
>
> > Something like that makes sense - I think it should be an experimental or
> > unsupported feature until if/when we have a critical mass of committers
> to
> > maintain it.
> >
> > On Mon, Apr 17, 2017 at 2:46 PM, Jim Apple  wrote:
> >
> > > That makes sense to me. It would be good for us to provide support
> > without
> > > completely focusing all development effort on a HW platform with fewer
> > > users.
> > >
> > > If ppc64le support lands in 2.10, and between 2.10 and 2.11 the ppc64le
> > > tests start breaking for reasons nobody with HW access can debug,
> should
> > we
> > > say in 2.11 release notes that ppc64le is no longer supported?
> > >
> > > Or perhaps, even in the first release where we think it works, we
> should
> > > spell it out as a platform with only "best-effort" support, that way we
> > > don't have to retract support?
> > >
> > > On Mon, Apr 17, 2017 at 2:41 PM, Marcel Kornacker  >
> > > wrote:
> > >
> > > > My main concern is that we don't unduly burden the development
> process.
> > > As
> > > > such, it makes a lot of sense to keep the PPC tests out of the
> regular
> > > > tests and have the party that's interested in the PPC tests create
> the
> > > > infrastructure and run those tests.
> > > >
> > > > On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple 
> > wrote:
> > > >
> > > > > One concern I have is sustainability. If only one Impala
> contributor
> > > can
> > > > > work with ppc64le, and that contributor is not as seasoned as some
> of
> > > the
> > > > > other committers, what happens if ppc64le breaks and the one person
> > > with
> > > > VM
> > > > > access can't fix it?
> > > > >
> > > > > Part of my concern is just how flaky the current tests are, too. It
> > > takes
> > > > > some time to be able to distinguish broken tests that are flaky and
> > > > broken
> > > > > tests that are the result of a specific commit.
> > > > >
> > > > > My hope was that with a ppc64le VM (maybe through Qemu?) that runs
> on
> > > > > x86-64 Linux, the other contributors could help fix things that
> broke
> > > on
> > > > > that platform.
> > > > >
> > > > > On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker <
> > > mar...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson <
> > he...@cloudera.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong <
> > > > tarmstr...@cloudera.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I feel like we shouldn't make PPC part of pre-commit at least
> > > > > > initially -
> > > > > > > > it's an unreasonable barrier if contributors/committers to
> > debug
> > > > > issues
> > > > > > > on
> > > > > > > > a platform they don't have easy access to. Having the testing
> > > infra
> > > > > is
> > > > > > > > still important because we don't want to have code in there
> > that
> > > > will
> > > > > > > > gradually bit-rot without us noticing.
> > > > > > > >
> > > > > > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus <
> > s...@cloudera.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Would it make sense to _not_ run PPC tests as part of
> > > presubmit?
> > > > > > > Instead
> > > > > > > > > Valencia could set up nightly tests using in-house
> > > > infrastructure.
> > > > > > And
> > > > > > > > > share the test results, e.g., by sending them to a new
> email
> > > list
> > > > > > > > > te...@impala.incubator.apache.org (that we'd need to
> create)
> > > so
> > > > > > > everyone
> > > > > > > > > can see when there are failures or if coverage stops for
> > > whatever
> > > > > > > reason.
> > > > > > > > > GCC has been doing something like this for long,
> > > > > > > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > > > > > > > >
> > > > > > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple <
> > > jbap...@cloudera.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Locally, I work on native-toolchain using a VM
> configured
> > > > with
> > > > > > > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we
> > provide
> > > > > you a
> > > > > > > VM
> > > > > > > > > with
> > > > > > > > > > > this config, will

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Jim Apple 
I'm OK with us calling this "experimental" (once the patches are in and we
think it works) and letting patches continue to land with our current
testing regime.

Later, if we add daily test reports with contributor-managed hardware or
even pre-commit testing on jenkins.impala.io in a VM, maybe we could
upgrade to other names than "experimental", like "beta" or "best-effort",
if we still want to have caveats.

How does that sound to everyone else?

On Mon, Apr 17, 2017 at 2:54 PM, Tim Armstrong 
wrote:

> Something like that makes sense - I think it should be an experimental or
> unsupported feature until if/when we have a critical mass of committers to
> maintain it.
>
> On Mon, Apr 17, 2017 at 2:46 PM, Jim Apple  wrote:
>
> > That makes sense to me. It would be good for us to provide support
> without
> > completely focusing all development effort on a HW platform with fewer
> > users.
> >
> > If ppc64le support lands in 2.10, and between 2.10 and 2.11 the ppc64le
> > tests start breaking for reasons nobody with HW access can debug, should
> we
> > say in 2.11 release notes that ppc64le is no longer supported?
> >
> > Or perhaps, even in the first release where we think it works, we should
> > spell it out as a platform with only "best-effort" support, that way we
> > don't have to retract support?
> >
> > On Mon, Apr 17, 2017 at 2:41 PM, Marcel Kornacker 
> > wrote:
> >
> > > My main concern is that we don't unduly burden the development process.
> > As
> > > such, it makes a lot of sense to keep the PPC tests out of the regular
> > > tests and have the party that's interested in the PPC tests create the
> > > infrastructure and run those tests.
> > >
> > > On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple 
> wrote:
> > >
> > > > One concern I have is sustainability. If only one Impala contributor
> > can
> > > > work with ppc64le, and that contributor is not as seasoned as some of
> > the
> > > > other committers, what happens if ppc64le breaks and the one person
> > with
> > > VM
> > > > access can't fix it?
> > > >
> > > > Part of my concern is just how flaky the current tests are, too. It
> > takes
> > > > some time to be able to distinguish broken tests that are flaky and
> > > broken
> > > > tests that are the result of a specific commit.
> > > >
> > > > My hope was that with a ppc64le VM (maybe through Qemu?) that runs on
> > > > x86-64 Linux, the other contributors could help fix things that broke
> > on
> > > > that platform.
> > > >
> > > > On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker <
> > mar...@cloudera.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson <
> he...@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong <
> > > tarmstr...@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I feel like we shouldn't make PPC part of pre-commit at least
> > > > > initially -
> > > > > > > it's an unreasonable barrier if contributors/committers to
> debug
> > > > issues
> > > > > > on
> > > > > > > a platform they don't have easy access to. Having the testing
> > infra
> > > > is
> > > > > > > still important because we don't want to have code in there
> that
> > > will
> > > > > > > gradually bit-rot without us noticing.
> > > > > > >
> > > > > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus <
> s...@cloudera.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Would it make sense to _not_ run PPC tests as part of
> > presubmit?
> > > > > > Instead
> > > > > > > > Valencia could set up nightly tests using in-house
> > > infrastructure.
> > > > > And
> > > > > > > > share the test results, e.g., by sending them to a new email
> > list
> > > > > > > > te...@impala.incubator.apache.org (that we'd need to create)
> > so
> > > > > > everyone
> > > > > > > > can see when there are failures or if coverage stops for
> > whatever
> > > > > > reason.
> > > > > > > > GCC has been doing something like this for long,
> > > > > > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > > > > > > >
> > > > > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple <
> > jbap...@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Locally, I work on native-toolchain using a VM configured
> > > with
> > > > > > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we
> provide
> > > > you a
> > > > > > VM
> > > > > > > > with
> > > > > > > > > > this config, will it be sufficient ?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > What hypervisor/emulator will it use?
> > > > > > > > >
> > > > > > > > > What are the requirements of the host OS and host hardware?
> > > > > > > > >
> > > > > > > > > Why is the config you have it set to so important that you
> > > > mention
> > > > > it
> > > > > > > in
> > > > > > > > > your email -

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Marcel Kornacker 
Agreed.

On Mon, Apr 17, 2017 at 2:54 PM, Tim Armstrong 
wrote:

> Something like that makes sense - I think it should be an experimental or
> unsupported feature until if/when we have a critical mass of committers to
> maintain it.
>
> On Mon, Apr 17, 2017 at 2:46 PM, Jim Apple  wrote:
>
> > That makes sense to me. It would be good for us to provide support
> without
> > completely focusing all development effort on a HW platform with fewer
> > users.
> >
> > If ppc64le support lands in 2.10, and between 2.10 and 2.11 the ppc64le
> > tests start breaking for reasons nobody with HW access can debug, should
> we
> > say in 2.11 release notes that ppc64le is no longer supported?
> >
> > Or perhaps, even in the first release where we think it works, we should
> > spell it out as a platform with only "best-effort" support, that way we
> > don't have to retract support?
> >
> > On Mon, Apr 17, 2017 at 2:41 PM, Marcel Kornacker 
> > wrote:
> >
> > > My main concern is that we don't unduly burden the development process.
> > As
> > > such, it makes a lot of sense to keep the PPC tests out of the regular
> > > tests and have the party that's interested in the PPC tests create the
> > > infrastructure and run those tests.
> > >
> > > On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple 
> wrote:
> > >
> > > > One concern I have is sustainability. If only one Impala contributor
> > can
> > > > work with ppc64le, and that contributor is not as seasoned as some of
> > the
> > > > other committers, what happens if ppc64le breaks and the one person
> > with
> > > VM
> > > > access can't fix it?
> > > >
> > > > Part of my concern is just how flaky the current tests are, too. It
> > takes
> > > > some time to be able to distinguish broken tests that are flaky and
> > > broken
> > > > tests that are the result of a specific commit.
> > > >
> > > > My hope was that with a ppc64le VM (maybe through Qemu?) that runs on
> > > > x86-64 Linux, the other contributors could help fix things that broke
> > on
> > > > that platform.
> > > >
> > > > On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker <
> > mar...@cloudera.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson <
> he...@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong <
> > > tarmstr...@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I feel like we shouldn't make PPC part of pre-commit at least
> > > > > initially -
> > > > > > > it's an unreasonable barrier if contributors/committers to
> debug
> > > > issues
> > > > > > on
> > > > > > > a platform they don't have easy access to. Having the testing
> > infra
> > > > is
> > > > > > > still important because we don't want to have code in there
> that
> > > will
> > > > > > > gradually bit-rot without us noticing.
> > > > > > >
> > > > > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus <
> s...@cloudera.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Would it make sense to _not_ run PPC tests as part of
> > presubmit?
> > > > > > Instead
> > > > > > > > Valencia could set up nightly tests using in-house
> > > infrastructure.
> > > > > And
> > > > > > > > share the test results, e.g., by sending them to a new email
> > list
> > > > > > > > te...@impala.incubator.apache.org (that we'd need to create)
> > so
> > > > > > everyone
> > > > > > > > can see when there are failures or if coverage stops for
> > whatever
> > > > > > reason.
> > > > > > > > GCC has been doing something like this for long,
> > > > > > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > > > > > > >
> > > > > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple <
> > jbap...@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Locally, I work on native-toolchain using a VM configured
> > > with
> > > > > > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we
> provide
> > > > you a
> > > > > > VM
> > > > > > > > with
> > > > > > > > > > this config, will it be sufficient ?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > What hypervisor/emulator will it use?
> > > > > > > > >
> > > > > > > > > What are the requirements of the host OS and host hardware?
> > > > > > > > >
> > > > > > > > > Why is the config you have it set to so important that you
> > > > mention
> > > > > it
> > > > > > > in
> > > > > > > > > your email - will the config be locked down into that
> config
> > or
> > > > can
> > > > > > it
> > > > > > > be
> > > > > > > > > reconfigured later?
> > > > > > > > >
> > > > > > > > > How is the VM controlled from the host OS? Keep in mind
> that
> > a
> > > > GUI
> > > > > > > cannot
> > > > > > > > > be the only option for automated tests.
> > > > > > > > >
> > > > > > > > > FWIW, Impala's test suite probably cannot fully complete
> > > without
> >

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Tim Armstrong 
Something like that makes sense - I think it should be an experimental or
unsupported feature until if/when we have a critical mass of committers to
maintain it.

On Mon, Apr 17, 2017 at 2:46 PM, Jim Apple  wrote:

> That makes sense to me. It would be good for us to provide support without
> completely focusing all development effort on a HW platform with fewer
> users.
>
> If ppc64le support lands in 2.10, and between 2.10 and 2.11 the ppc64le
> tests start breaking for reasons nobody with HW access can debug, should we
> say in 2.11 release notes that ppc64le is no longer supported?
>
> Or perhaps, even in the first release where we think it works, we should
> spell it out as a platform with only "best-effort" support, that way we
> don't have to retract support?
>
> On Mon, Apr 17, 2017 at 2:41 PM, Marcel Kornacker 
> wrote:
>
> > My main concern is that we don't unduly burden the development process.
> As
> > such, it makes a lot of sense to keep the PPC tests out of the regular
> > tests and have the party that's interested in the PPC tests create the
> > infrastructure and run those tests.
> >
> > On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple  wrote:
> >
> > > One concern I have is sustainability. If only one Impala contributor
> can
> > > work with ppc64le, and that contributor is not as seasoned as some of
> the
> > > other committers, what happens if ppc64le breaks and the one person
> with
> > VM
> > > access can't fix it?
> > >
> > > Part of my concern is just how flaky the current tests are, too. It
> takes
> > > some time to be able to distinguish broken tests that are flaky and
> > broken
> > > tests that are the result of a specific commit.
> > >
> > > My hope was that with a ppc64le VM (maybe through Qemu?) that runs on
> > > x86-64 Linux, the other contributors could help fix things that broke
> on
> > > that platform.
> > >
> > > On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker <
> mar...@cloudera.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson  >
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong <
> > tarmstr...@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I feel like we shouldn't make PPC part of pre-commit at least
> > > > initially -
> > > > > > it's an unreasonable barrier if contributors/committers to debug
> > > issues
> > > > > on
> > > > > > a platform they don't have easy access to. Having the testing
> infra
> > > is
> > > > > > still important because we don't want to have code in there that
> > will
> > > > > > gradually bit-rot without us noticing.
> > > > > >
> > > > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus 
> > > > wrote:
> > > > > >
> > > > > > > Would it make sense to _not_ run PPC tests as part of
> presubmit?
> > > > > Instead
> > > > > > > Valencia could set up nightly tests using in-house
> > infrastructure.
> > > > And
> > > > > > > share the test results, e.g., by sending them to a new email
> list
> > > > > > > te...@impala.incubator.apache.org (that we'd need to create)
> so
> > > > > everyone
> > > > > > > can see when there are failures or if coverage stops for
> whatever
> > > > > reason.
> > > > > > > GCC has been doing something like this for long,
> > > > > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > > > > > >
> > > > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple <
> jbap...@cloudera.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > >
> > > > > > > > > Locally, I work on native-toolchain using a VM configured
> > with
> > > > > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide
> > > you a
> > > > > VM
> > > > > > > with
> > > > > > > > > this config, will it be sufficient ?
> > > > > > > > >
> > > > > > > >
> > > > > > > > What hypervisor/emulator will it use?
> > > > > > > >
> > > > > > > > What are the requirements of the host OS and host hardware?
> > > > > > > >
> > > > > > > > Why is the config you have it set to so important that you
> > > mention
> > > > it
> > > > > > in
> > > > > > > > your email - will the config be locked down into that config
> or
> > > can
> > > > > it
> > > > > > be
> > > > > > > > reconfigured later?
> > > > > > > >
> > > > > > > > How is the VM controlled from the host OS? Keep in mind that
> a
> > > GUI
> > > > > > cannot
> > > > > > > > be the only option for automated tests.
> > > > > > > >
> > > > > > > > FWIW, Impala's test suite probably cannot fully complete
> > without
> > > at
> > > > > > least
> > > > > > > > 8, and I suspect 16, GB of RAM, and we might need more disk
> > > space,
> > > > > too,
> > > > > > > but
> > > > > > > > these should be reconfigurable with most
> hypervisors/emulators.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > --
> > > > > Henry Robinson
> > > > > Software Engineer
> > > > > Cloudera
> > > > > 415-994-6679
> > >

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Jim Apple 
That makes sense to me. It would be good for us to provide support without
completely focusing all development effort on a HW platform with fewer
users.

If ppc64le support lands in 2.10, and between 2.10 and 2.11 the ppc64le
tests start breaking for reasons nobody with HW access can debug, should we
say in 2.11 release notes that ppc64le is no longer supported?

Or perhaps, even in the first release where we think it works, we should
spell it out as a platform with only "best-effort" support, that way we
don't have to retract support?

On Mon, Apr 17, 2017 at 2:41 PM, Marcel Kornacker 
wrote:

> My main concern is that we don't unduly burden the development process. As
> such, it makes a lot of sense to keep the PPC tests out of the regular
> tests and have the party that's interested in the PPC tests create the
> infrastructure and run those tests.
>
> On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple  wrote:
>
> > One concern I have is sustainability. If only one Impala contributor can
> > work with ppc64le, and that contributor is not as seasoned as some of the
> > other committers, what happens if ppc64le breaks and the one person with
> VM
> > access can't fix it?
> >
> > Part of my concern is just how flaky the current tests are, too. It takes
> > some time to be able to distinguish broken tests that are flaky and
> broken
> > tests that are the result of a specific commit.
> >
> > My hope was that with a ppc64le VM (maybe through Qemu?) that runs on
> > x86-64 Linux, the other contributors could help fix things that broke on
> > that platform.
> >
> > On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker 
> > wrote:
> >
> > > +1
> > >
> > > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong <
> tarmstr...@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > I feel like we shouldn't make PPC part of pre-commit at least
> > > initially -
> > > > > it's an unreasonable barrier if contributors/committers to debug
> > issues
> > > > on
> > > > > a platform they don't have easy access to. Having the testing infra
> > is
> > > > > still important because we don't want to have code in there that
> will
> > > > > gradually bit-rot without us noticing.
> > > > >
> > > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus 
> > > wrote:
> > > > >
> > > > > > Would it make sense to _not_ run PPC tests as part of presubmit?
> > > > Instead
> > > > > > Valencia could set up nightly tests using in-house
> infrastructure.
> > > And
> > > > > > share the test results, e.g., by sending them to a new email list
> > > > > > te...@impala.incubator.apache.org (that we'd need to create) so
> > > > everyone
> > > > > > can see when there are failures or if coverage stops for whatever
> > > > reason.
> > > > > > GCC has been doing something like this for long,
> > > > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > > > > >
> > > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple  >
> > > > wrote:
> > > > > >
> > > > > > > >
> > > > > > > > Locally, I work on native-toolchain using a VM configured
> with
> > > > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide
> > you a
> > > > VM
> > > > > > with
> > > > > > > > this config, will it be sufficient ?
> > > > > > > >
> > > > > > >
> > > > > > > What hypervisor/emulator will it use?
> > > > > > >
> > > > > > > What are the requirements of the host OS and host hardware?
> > > > > > >
> > > > > > > Why is the config you have it set to so important that you
> > mention
> > > it
> > > > > in
> > > > > > > your email - will the config be locked down into that config or
> > can
> > > > it
> > > > > be
> > > > > > > reconfigured later?
> > > > > > >
> > > > > > > How is the VM controlled from the host OS? Keep in mind that a
> > GUI
> > > > > cannot
> > > > > > > be the only option for automated tests.
> > > > > > >
> > > > > > > FWIW, Impala's test suite probably cannot fully complete
> without
> > at
> > > > > least
> > > > > > > 8, and I suspect 16, GB of RAM, and we might need more disk
> > space,
> > > > too,
> > > > > > but
> > > > > > > these should be reconfigurable with most hypervisors/emulators.
> > > > > > >
> > > > > >
> > > > >
> > > > --
> > > > Henry Robinson
> > > > Software Engineer
> > > > Cloudera
> > > > 415-994-6679
> > > >
> > >
> >
>

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Marcel Kornacker 
My main concern is that we don't unduly burden the development process. As
such, it makes a lot of sense to keep the PPC tests out of the regular
tests and have the party that's interested in the PPC tests create the
infrastructure and run those tests.

On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple  wrote:

> One concern I have is sustainability. If only one Impala contributor can
> work with ppc64le, and that contributor is not as seasoned as some of the
> other committers, what happens if ppc64le breaks and the one person with VM
> access can't fix it?
>
> Part of my concern is just how flaky the current tests are, too. It takes
> some time to be able to distinguish broken tests that are flaky and broken
> tests that are the result of a specific commit.
>
> My hope was that with a ppc64le VM (maybe through Qemu?) that runs on
> x86-64 Linux, the other contributors could help fix things that broke on
> that platform.
>
> On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker 
> wrote:
>
> > +1
> >
> > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson 
> > wrote:
> >
> > > +1
> > >
> > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong  >
> > > wrote:
> > >
> > > > I feel like we shouldn't make PPC part of pre-commit at least
> > initially -
> > > > it's an unreasonable barrier if contributors/committers to debug
> issues
> > > on
> > > > a platform they don't have easy access to. Having the testing infra
> is
> > > > still important because we don't want to have code in there that will
> > > > gradually bit-rot without us noticing.
> > > >
> > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus 
> > wrote:
> > > >
> > > > > Would it make sense to _not_ run PPC tests as part of presubmit?
> > > Instead
> > > > > Valencia could set up nightly tests using in-house infrastructure.
> > And
> > > > > share the test results, e.g., by sending them to a new email list
> > > > > te...@impala.incubator.apache.org (that we'd need to create) so
> > > everyone
> > > > > can see when there are failures or if coverage stops for whatever
> > > reason.
> > > > > GCC has been doing something like this for long,
> > > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > > > >
> > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple 
> > > wrote:
> > > > >
> > > > > > >
> > > > > > > Locally, I work on native-toolchain using a VM configured with
> > > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide
> you a
> > > VM
> > > > > with
> > > > > > > this config, will it be sufficient ?
> > > > > > >
> > > > > >
> > > > > > What hypervisor/emulator will it use?
> > > > > >
> > > > > > What are the requirements of the host OS and host hardware?
> > > > > >
> > > > > > Why is the config you have it set to so important that you
> mention
> > it
> > > > in
> > > > > > your email - will the config be locked down into that config or
> can
> > > it
> > > > be
> > > > > > reconfigured later?
> > > > > >
> > > > > > How is the VM controlled from the host OS? Keep in mind that a
> GUI
> > > > cannot
> > > > > > be the only option for automated tests.
> > > > > >
> > > > > > FWIW, Impala's test suite probably cannot fully complete without
> at
> > > > least
> > > > > > 8, and I suspect 16, GB of RAM, and we might need more disk
> space,
> > > too,
> > > > > but
> > > > > > these should be reconfigurable with most hypervisors/emulators.
> > > > > >
> > > > >
> > > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
>

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Jim Apple 
One concern I have is sustainability. If only one Impala contributor can
work with ppc64le, and that contributor is not as seasoned as some of the
other committers, what happens if ppc64le breaks and the one person with VM
access can't fix it?

Part of my concern is just how flaky the current tests are, too. It takes
some time to be able to distinguish broken tests that are flaky and broken
tests that are the result of a specific commit.

My hope was that with a ppc64le VM (maybe through Qemu?) that runs on
x86-64 Linux, the other contributors could help fix things that broke on
that platform.

On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker 
wrote:

> +1
>
> On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson 
> wrote:
>
> > +1
> >
> > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong 
> > wrote:
> >
> > > I feel like we shouldn't make PPC part of pre-commit at least
> initially -
> > > it's an unreasonable barrier if contributors/committers to debug issues
> > on
> > > a platform they don't have easy access to. Having the testing infra is
> > > still important because we don't want to have code in there that will
> > > gradually bit-rot without us noticing.
> > >
> > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus 
> wrote:
> > >
> > > > Would it make sense to _not_ run PPC tests as part of presubmit?
> > Instead
> > > > Valencia could set up nightly tests using in-house infrastructure.
> And
> > > > share the test results, e.g., by sending them to a new email list
> > > > te...@impala.incubator.apache.org (that we'd need to create) so
> > everyone
> > > > can see when there are failures or if coverage stops for whatever
> > reason.
> > > > GCC has been doing something like this for long,
> > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > > >
> > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple 
> > wrote:
> > > >
> > > > > >
> > > > > > Locally, I work on native-toolchain using a VM configured with
> > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide you a
> > VM
> > > > with
> > > > > > this config, will it be sufficient ?
> > > > > >
> > > > >
> > > > > What hypervisor/emulator will it use?
> > > > >
> > > > > What are the requirements of the host OS and host hardware?
> > > > >
> > > > > Why is the config you have it set to so important that you mention
> it
> > > in
> > > > > your email - will the config be locked down into that config or can
> > it
> > > be
> > > > > reconfigured later?
> > > > >
> > > > > How is the VM controlled from the host OS? Keep in mind that a GUI
> > > cannot
> > > > > be the only option for automated tests.
> > > > >
> > > > > FWIW, Impala's test suite probably cannot fully complete without at
> > > least
> > > > > 8, and I suspect 16, GB of RAM, and we might need more disk space,
> > too,
> > > > but
> > > > > these should be reconfigurable with most hypervisors/emulators.
> > > > >
> > > >
> > >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Marcel Kornacker 
+1

On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson  wrote:

> +1
>
> On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong 
> wrote:
>
> > I feel like we shouldn't make PPC part of pre-commit at least initially -
> > it's an unreasonable barrier if contributors/committers to debug issues
> on
> > a platform they don't have easy access to. Having the testing infra is
> > still important because we don't want to have code in there that will
> > gradually bit-rot without us noticing.
> >
> > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus  wrote:
> >
> > > Would it make sense to _not_ run PPC tests as part of presubmit?
> Instead
> > > Valencia could set up nightly tests using in-house infrastructure.  And
> > > share the test results, e.g., by sending them to a new email list
> > > te...@impala.incubator.apache.org (that we'd need to create) so
> everyone
> > > can see when there are failures or if coverage stops for whatever
> reason.
> > > GCC has been doing something like this for long,
> > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> > >
> > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple 
> wrote:
> > >
> > > > >
> > > > > Locally, I work on native-toolchain using a VM configured with
> > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide you a
> VM
> > > with
> > > > > this config, will it be sufficient ?
> > > > >
> > > >
> > > > What hypervisor/emulator will it use?
> > > >
> > > > What are the requirements of the host OS and host hardware?
> > > >
> > > > Why is the config you have it set to so important that you mention it
> > in
> > > > your email - will the config be locked down into that config or can
> it
> > be
> > > > reconfigured later?
> > > >
> > > > How is the VM controlled from the host OS? Keep in mind that a GUI
> > cannot
> > > > be the only option for automated tests.
> > > >
> > > > FWIW, Impala's test suite probably cannot fully complete without at
> > least
> > > > 8, and I suspect 16, GB of RAM, and we might need more disk space,
> too,
> > > but
> > > > these should be reconfigurable with most hypervisors/emulators.
> > > >
> > >
> >
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Henry Robinson 
+1

On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong 
wrote:

> I feel like we shouldn't make PPC part of pre-commit at least initially -
> it's an unreasonable barrier if contributors/committers to debug issues on
> a platform they don't have easy access to. Having the testing infra is
> still important because we don't want to have code in there that will
> gradually bit-rot without us noticing.
>
> On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus  wrote:
>
> > Would it make sense to _not_ run PPC tests as part of presubmit?  Instead
> > Valencia could set up nightly tests using in-house infrastructure.  And
> > share the test results, e.g., by sending them to a new email list
> > te...@impala.incubator.apache.org (that we'd need to create) so everyone
> > can see when there are failures or if coverage stops for whatever reason.
> > GCC has been doing something like this for long,
> > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
> >
> > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple  wrote:
> >
> > > >
> > > > Locally, I work on native-toolchain using a VM configured with
> > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide you a VM
> > with
> > > > this config, will it be sufficient ?
> > > >
> > >
> > > What hypervisor/emulator will it use?
> > >
> > > What are the requirements of the host OS and host hardware?
> > >
> > > Why is the config you have it set to so important that you mention it
> in
> > > your email - will the config be locked down into that config or can it
> be
> > > reconfigured later?
> > >
> > > How is the VM controlled from the host OS? Keep in mind that a GUI
> cannot
> > > be the only option for automated tests.
> > >
> > > FWIW, Impala's test suite probably cannot fully complete without at
> least
> > > 8, and I suspect 16, GB of RAM, and we might need more disk space, too,
> > but
> > > these should be reconfigurable with most hypervisors/emulators.
> > >
> >
>
-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Tim Armstrong 
I feel like we shouldn't make PPC part of pre-commit at least initially -
it's an unreasonable barrier if contributors/committers to debug issues on
a platform they don't have easy access to. Having the testing infra is
still important because we don't want to have code in there that will
gradually bit-rot without us noticing.

On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus  wrote:

> Would it make sense to _not_ run PPC tests as part of presubmit?  Instead
> Valencia could set up nightly tests using in-house infrastructure.  And
> share the test results, e.g., by sending them to a new email list
> te...@impala.incubator.apache.org (that we'd need to create) so everyone
> can see when there are failures or if coverage stops for whatever reason.
> GCC has been doing something like this for long,
> https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
>
> On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple  wrote:
>
> > >
> > > Locally, I work on native-toolchain using a VM configured with
> > > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide you a VM
> with
> > > this config, will it be sufficient ?
> > >
> >
> > What hypervisor/emulator will it use?
> >
> > What are the requirements of the host OS and host hardware?
> >
> > Why is the config you have it set to so important that you mention it in
> > your email - will the config be locked down into that config or can it be
> > reconfigured later?
> >
> > How is the VM controlled from the host OS? Keep in mind that a GUI cannot
> > be the only option for automated tests.
> >
> > FWIW, Impala's test suite probably cannot fully complete without at least
> > 8, and I suspect 16, GB of RAM, and we might need more disk space, too,
> but
> > these should be reconfigurable with most hypervisors/emulators.
> >
>

Re: Upstreaming ppc64le patches for native-toolchain

2017-04-17 Thread Silvius Rus 
Would it make sense to _not_ run PPC tests as part of presubmit?  Instead
Valencia could set up nightly tests using in-house infrastructure.  And
share the test results, e.g., by sending them to a new email list
te...@impala.incubator.apache.org (that we'd need to create) so everyone
can see when there are failures or if coverage stops for whatever reason.
GCC has been doing something like this for long,
https://gcc.gnu.org/ml/gcc-testresults/2017-04/.

On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple  wrote:

> >
> > Locally, I work on native-toolchain using a VM configured with
> > Ubuntu16.04ppc64le, 4GB RAM and 50GB of HDD. If  we provide you a VM with
> > this config, will it be sufficient ?
> >
>
> What hypervisor/emulator will it use?
>
> What are the requirements of the host OS and host hardware?
>
> Why is the config you have it set to so important that you mention it in
> your email - will the config be locked down into that config or can it be
> reconfigured later?
>
> How is the VM controlled from the host OS? Keep in mind that a GUI cannot
> be the only option for automated tests.
>
> FWIW, Impala's test suite probably cannot fully complete without at least
> 8, and I suspect 16, GB of RAM, and we might need more disk space, too, but
> these should be reconfigurable with most hypervisors/emulators.
>