Re: Jenkins scripts in repo?

That's just the standard Jenkins plugin to read Junit XML. I think it comes with a standard Jenkins install on Ubuntu 16. https://plugins.jenkins.io/junit Most of our tests already emit Junit XML, so that's taken care of for you already. To use the plugin, you just tell it a series of paths to f

Re: Jenkins scripts in repo?

If it's helpful, I'd be happy to send you, off-list, the XML configuration file (which is Jenkins' serialized format) and/or screenshots of any of our jobs. It's a little bit of work because I have to make sure they're devoid of passwords and such. Thanks! -- Philip On Wed, Jul 11, 2018 at 8:16

Removing IMPALA_MINICLUSTER_PROFILE=2 from master

Hi all, In the master branch, we support both Hadoop 2 and Hadoop 3 via IMPALA_MINICLUSTER_PROFILE environment variable. When Impala transitioned from Hadoop 2 to Hadoop 3, we introduced IMPALA_MINICLUSTER_PROFILE as a way to easily switch between Hadoop 2 and Hadoop 3. It made sense at that time.

Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

+1 for removing it. I'm all for reduced complexity if not a lot of users are benefitting from it. On Wed, Jul 11, 2018 at 9:29 AM, Fredy Wijaya wrote: > Hi all, > > In the master branch, we support both Hadoop 2 and Hadoop 3 via > IMPALA_MINICLUSTER_PROFILE environment variable. When Impala tra

Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

I'm +1 on removing it. On Wed, Jul 11, 2018 at 9:39 AM Sailesh Mukil wrote: > +1 for removing it. > > I'm all for reduced complexity if not a lot of users are benefitting from > it. > > On Wed, Jul 11, 2018 at 9:29 AM, Fredy Wijaya > > wrote: > > > Hi all, > > > > In the master branch, we suppo

Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

+1 on removing it. On Wed, Jul 11, 2018 at 10:49 AM, Philip Zeyliger < phi...@cloudera.com.invalid> wrote: > I'm +1 on removing it. > > On Wed, Jul 11, 2018 at 9:39 AM Sailesh Mukil > > wrote: > > > +1 for removing it. > > > > I'm all for reduced complexity if not a lot of users are benefitting

Re:Re: Jenkins scripts in repo?

Great thanks to you all! That's enough for our current need. Thank Philip for willing to be our consulter! I might contact you off-list if we have further questions. Thanks, Quanlong At 2018-07-11 23:19:12, "Philip Zeyliger" wrote: >If it's helpful, I'd be happy to send you, off-list, the X

Inconsistent handling of schema in Avro tables

Hey folks, I'm trying to understand the current behavior of tables that contain partitions of mixed format, specifically when one or more partitions is stored as Avro. Impala seems to be doing a number of things which I find surprising, and I'm not sure if they are intentional or should be conside

Re:Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

So there're no more official supports for Hadoop 2, even for bug-fix? I know the master branch may have only support Hadoop 3 with full test coverage now. But I don't identify with you to abandon users in Hadoop 2. What if they encounter a critical bug in impala-2.12 and can not upgrade their he

Re: Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

Hi Quanlong, The branch 2.x is not dead, it's just not actively maintained because we stopped the automatic cherry-picking from master to 2.x. However, we still welcome patches into 2.x branch if the community is willing to submit some patches to it. I hope that clarifies it. On Wed, Jul 11, 2018

Re: Inconsistent handling of schema in Avro tables

The behaviour of Avro schemas in all these cases has always been rather mysterious to me. Before you wrote this email I would have assumed that Impala's behaviour would be like Hive's behaviour. I agree with the principle that the creation of a partition without changes to table metadata shouldn't

How to report potential security issues

As a reminder, if you have found a potential security vulnerability, please report your findings to either priv...@impala.apache.org or follow https://www.apache.org/security/#reporting-a-vulnerability.

Re: Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

The tricky thing about backporting all bug-fixes is that 41 of the 91 resolved issues in the past four weeks are bugs: https://issues.apache.org/jira/issues/?jql=project%20%3D%20IMPALA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Fixed%20AND%20resolved%20%3E%3D%20-4w%20ORDER%20BY%20iss

Re: Inconsistent handling of schema in Avro tables

I know that Hive can deal with schema being different per partition, but I really hesitate to understand why someone would want to do this. If someone asked me to support a mixed avro/parquet table I would suggest they create a view. If they kept insisting I would reply "Well it is your funeral."

Re: Inconsistent handling of schema in Avro tables

I added this functionality where adding an Avro partition in a mixed partition table resets the table level schema. While I don't exactly remember why we chose this path, I do recall that we debated quite a bit abou

Re: Inconsistent handling of schema in Avro tables

Your commit message there makes sense, Bharath -- we should set 'avroSchema' in the descriptor in case any referenced partition is avro, because the scanner needs that info. However, we don't need to also override the table-level schema. So, I think we can preserve the fix that you made while also

Re: Inconsistent handling of schema in Avro tables

Agreed. On Wed, Jul 11, 2018 at 8:55 PM Todd Lipcon wrote: > Your commit message there makes sense, Bharath -- we should set > 'avroSchema' in the descriptor in case any referenced partition is avro, > because the scanner needs that info. However, we don't need to also > override the table-level

Re: Inconsistent handling of schema in Avro tables

Turns out it's even a bit more messy. The presence of one or more avro partitions can change the types of existing columns, even if there is no explicit avro schema specified for the table: https://gist.github.com/5018d6ff50f846c72762319eb7cf5ca8 Not quite sure how to handle this one in a world wh