Re: Add total number of started threads to /threadz

[Lars Volker]

Thank you for the feedback. I opened IMPALA-5643 to track this and pushed a
change here: https://gerrit.cloudera.org/#/c/7390/

On Mon, Jul 10, 2017 at 10:25 AM, Henry Robinson  wrote:

> This seems valuable to me.
>
> On 28 June 2017 at 21:26, Lars Volker  wrote:
>
> > Hi All,
> >
> > While investigating IMPALA-5598 I added a counter with the total number
> of
> > threads to /threadz. See below for what it looks like (I hope the ASF
> > mailer won't eat the format). Does this look helpful? If someone thinks
> it
> > does, I'll create a JIRA and push the change.
> >
> > Thanks, Lars
> >
> >
> > Thread GroupsAll threads
> >  >DataStreamSender
> > : (running: 0, total created: 2500)
> >  > group?group=DataStreamSender>common
> > : (running: 2, total created: 2)
> > 
> > coordinator-fragment-rpc
> > : (running: 12, total created: 12)
> >  > fragment-rpc>disk-io-mgr
> > : (running: 34, total created: 34)
> > 
> > fragment-mgr
> > : (running: 0, total created: 2550)
> >  > >hdfs-scan-node
> > : (running: 0, total created: 2500)
> >  group?group=hdfs-scan-node
> > >hdfs-worker-pool
> > : (running: 16, total created: 16)
> >  > group?group=hdfs-worker-pool>impala-server
> > : (running: 8, total created: 8)
> >  group?group=impala-server>
> > plan-fragment-executor
> > : (running: 0, total created: 2550)
> >  group?group=plan-fragment-
> > executor>query-exec-state
> > : (running: 0, total created: 50)
> >  > group?group=query-exec-state>rpc-pool
> > : (running: 8, total created: 8)
> > 
> > scheduling
> > : (running: 1, total created: 1)
> >  > >setup-server
> > : (running: 2, total created: 2)
> >  >
> > statestore-subscriber
> > : (running: 1, total created: 1)
> >  > subscriber>thrift-server
> > : (running: 248, total created: 248)
> >  group?group=thrift-server>
> >
>

Re: Jenkins maintenance in 30 minutes

[Lars Volker]

Hi All,

Jenkins maintenance has been completed and the service should be working
again. However, some plugins issued a warning that parts of their
configuration format have changed and thus jobs may need to be
reconfigured. I started a canary job here that is currently still running:
http://jenkins.impala.io:8080/job/parallel-all-tests/1035/console

If you encounter any issues during job creation, please let me know.

Cheers, Lars

On Mon, Jul 10, 2017 at 11:32 AM, Lars Volker  wrote:

> Hi All,
>
> In about 30 minutes, jenkins.impala.io will become unavailable for
> maintenance. I expect it to take no more than 30 minutes and will send an
> additional email to this list once the service is back online. In case your
> jobs get terminated, please restart them.
>
> Thank you, Lars
>

Jenkins maintenance in 30 minutes

[Lars Volker]

Hi All,

In about 30 minutes, jenkins.impala.io will become unavailable for
maintenance. I expect it to take no more than 30 minutes and will send an
additional email to this list once the service is back online. In case your
jobs get terminated, please restart them.

Thank you, Lars

CVE-2017-5640 Apache Impala (incubating) Information Disclosure

[Sailesh Mukil]

CVE-2017-5640 Apache Impala (incubating) Information Disclosure

Severity: High

Versions Affected:
Apache Impala (incubating) 2.7.0 to 2.8.0

Description:
It was noticed that a malicious process impersonating an Impala daemon
could cause Impala daemons to skip authentication checks when Kerberos
is enabled (but TLS is not). If the malicious server responds with
‘COMPLETE’ before the SASL handshake has completed, the client will
consider the handshake as completed even though no exchange of
credentials has happened.

Mitigation:
Users of the affected versions should apply the following mitigation:
Upgrade to Apache Impala (incubating) 2.9.0

Credit:
This issue was identified by the Cloudera Security team.

References:
https://issues.apache.org/jira/browse/IMPALA-5005

Re: Can't start minicluster

[Jim Apple]

Using that command and then running a build without -noclean WFM. Thanks!

On Sun, Jul 9, 2017 at 9:10 PM, Tim Armstrong 
wrote:

> Maybe the thrift be/generated-sources are out of sync with the source code?
>
> We had some kind of metastore scheme upgrade that caused the other one.
> Dimitris' instructions to fix them were:
>
> > To fix this without doing a full data reload, you can use the following
> command:
> > ${IMPALA_TOOLCHAIN}/cdh_components/hive-1.1.0-cdh5.13.0-
> SNAPSHOT/bin/schematool
> -upgradeSchema -dbType {type}
> > where type is one of 'postgres' or 'mysql', depending on your setup.
>
> On Sun, Jul 9, 2017 at 3:52 PM, Jim Apple  wrote:
>
> > I am getting the following message in FATAL when I try to start a
> > minicluster
> >
> > Check failed: _TImpalaQueryOptions_VALUES_TO_NAMES.size() ==
> > TImpalaQueryOptions::DEFAULT_JOIN_DISTRIBUTION_MODE + 1 (57 vs. 56)
> >
> > Any ideas what is going on? I was actually trying to buildall.sh
> > -format_metastore -format_sentry_policy_db because I was seeing messages
> > like the following (in hive.log) when I tried to start the minicluster:
> >
> >  org.postgresql.util.PSQLException: ERROR: column A0.SCHEMA_VERSION_V2
> > does
> > not exist
> >
>

Re: Add total number of started threads to /threadz

[Henry Robinson]

This seems valuable to me.

On 28 June 2017 at 21:26, Lars Volker  wrote:

> Hi All,
>
> While investigating IMPALA-5598 I added a counter with the total number of
> threads to /threadz. See below for what it looks like (I hope the ASF
> mailer won't eat the format). Does this look helpful? If someone thinks it
> does, I'll create a JIRA and push the change.
>
> Thanks, Lars
>
>
> Thread GroupsAll threads
> DataStreamSender
> : (running: 0, total created: 2500)
>  group?group=DataStreamSender>common
> : (running: 2, total created: 2)
> 
> coordinator-fragment-rpc
> : (running: 12, total created: 12)
>  fragment-rpc>disk-io-mgr
> : (running: 34, total created: 34)
> 
> fragment-mgr
> : (running: 0, total created: 2550)
>  >hdfs-scan-node
> : (running: 0, total created: 2500)
>  >hdfs-worker-pool
> : (running: 16, total created: 16)
>  group?group=hdfs-worker-pool>impala-server
> : (running: 8, total created: 8)
> 
> plan-fragment-executor
> : (running: 0, total created: 2550)
>  executor>query-exec-state
> : (running: 0, total created: 50)
>  group?group=query-exec-state>rpc-pool
> : (running: 8, total created: 8)
> 
> scheduling
> : (running: 1, total created: 1)
>  >setup-server
> : (running: 2, total created: 2)
> 
> statestore-subscriber
> : (running: 1, total created: 1)
>  subscriber>thrift-server
> : (running: 248, total created: 248)
> 
>

[SECURITY] CVE-2017-5652 Apache Impala (incubating) Information Disclosure

[Sailesh Mukil]

CVE-2017-5652 Apache Impala (incubating) Information Disclosure


Severity: High


Versions Affected:

Apache Impala (incubating) 2.7.0 to 2.8.0


Description:

During a routine security analysis, it was found that one of the ports sent
data in plaintext even when the cluster was configured to use TLS. The port
in question was used by the StatestoreSubscriber class which did not use
the appropriate secure Thrift transport when TLS was turned on. It was
therefore possible for an adversary, with access to the network, to
eavesdrop on the packets going to and coming from that port and view the
data in plaintext.


Mitigation:

Users of the affected versions should apply the following mitigation:

 - Upgrade to Apache Impala (incubating) 2.9.0


Credit:
This issue was identified and reported responsibly by the Cloudera security
team.


References:
[1] https://issues.apache.org/jira/browse/IMPALA-5253

Re: Impala Build issue

[Jim Apple]

For #1, I recommend one of the following:

1. Get access to Oracle JDK7 however you would normally do so (legally, of
course)

2. OR use OpenJDK7

8. OR use JDK8, either Oracle or OpenJDK

On Sun, Jul 9, 2017 at 1:31 PM, Suresh Pujari 
wrote:

> Hi Sir,
>
>I am receiving the below issues.
>
> 1. the JDK path
> "*http://download.oracle.com/otn-pub/java/jdk/7u75-b13/jdk-
> 7u75-linux-x64.tar.g
>  7u75-linux-x64.tar.g>z*"
> in the install.sh is incorrect because of which errors are coming.
>
> sudo ./install
>
> Compiled Resource:
> --
> # Declared in /impala-setup/cookbooks/java/recipes/oracle.rb:53:in
> `from_file'
>
> java_ark("jdk") do
>   action [:install]
>   supports {:report=>true, :exception=>true}
>   retries 0
>   retry_delay 2
>   default_guard_interpreter :default
>   declared_type :java_ark
>   cookbook_name :java
>   recipe_name "oracle"
>   url "
> http://download.oracle.com/otn-pub/java/jdk/7u75-b13/jdk-
> 7u75-linux-x64.tar.gz
> "
>   default true
>   checksum "6f1f81030a34f7a9c987f8b68a24d139"
>   app_home "/usr/lib/jvm/java-7-oracle-amd64"
>   bin_cmds ["appletviewer", "apt", "ControlPanel", "extcheck", "idlj",
> "jar", "jarsigner", "java", "javac", "javadoc", "javafxpackager", "javah",
> "javap", "javaws", "jcmd", "jconsole", "jcontrol", "jdb", "jhat", "jinfo",
> "jmap", "jps", "jrunscript", "jsadebugd", "jstack", "jstat", "jstatd",
> "jvisualvm", "keytool", "native2ascii", "orbd", "pack200", "policytool",
> "rmic", "rmid", "rmiregistry", "schemagen", "serialver", "servertool",
> "tnameserv", "unpack200", "wsgen", "wsimport", "xjc"]
>   alternatives_priority 1062
>   connect_timeout 600
>   owner "root"
> end
>
>
> Running handlers:
> [2017-07-10T00:28:41+04:00] ERROR: Running exception handlers
> Running handlers complete
> [2017-07-10T00:28:41+04:00] ERROR: Exception handlers complete
> [2017-07-10T00:28:41+04:00] FATAL: Stacktrace dumped to
> /mnt/DATA/impala-setup/chef-stacktrace.out
> Chef Client failed. 4 resources updated in 7.840529619 seconds
> [2017-07-10T00:28:41+04:00] ERROR: java_ark[jdk] (java::oracle line 53) had
> an error: SystemExit: exit
> [2017-07-10T00:28:41+04:00] FATAL: Chef::Exceptions::ChildConvergeError:
> Chef run process exited unsuccessfully (exit code 1)
>
> 2. ./bin/bootstrap_build.sh
>
> Linking CXX shared library libgutil.so
> [ 10%] Built target gutil
> make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> make: *** [impalad] Error 2
> Error in /impala-2.9.0/bin/make_impala.sh at line 179: ${MAKE_CMD}
> ${MAKE_ARGS} ${MAKE_TARGETS}
>
> Please help.
>
> Regards
> Suresh
>

Re: IMPALA-4326 - split() function

[Edward Capriolo]

That standard 2016 spec did not predate hive's implementation of lateral
view

On Sunday, July 9, 2017, Greg Rahn  wrote:

> (also commented on IMPALA-4326)
>
> For this functionality, I'd prefer to follow what Postgres does and use its
> well-named functions like string_to_array().
> This becomes powerful when using the unnest() table function, which is
> defined and is part of the ANSI/ISO SQL:2016 spec (vs the non-standard
> lateral view explode Hive syntax).
>
> with t as (
>   select
> 42 as id,
> '1,2,3,4,5,6'::text as string_array
> )
> select
>   t.id,
>   u.l
> from t, unnest(string_to_array(t.string_array,',')) as u(l);
>
> id | l
> +---
> 42 | 1
> 42 | 2
> 42 | 3
> 42 | 4
> 42 | 5
> 42 | 6
>
>
> On Mon, Jun 19, 2017 at 7:40 AM, Alexander Behm  >
> wrote:
>
> > Yes and no. Extending the UDF framework might be hard, but I think
> > implementing a built-in split() is feasible. We already have a built-in
> > Expr that returns an array type to implement unnest.
> >
> > On Mon, Jun 19, 2017 at 6:22 AM, Vincent Tran  > wrote:
> >
> > > This request appears to be blocked by the current UDF framework's
> > > limitation.
> > > As far as I can tell, functions can still only return simple scalar
> > types,
> > > right?
> > >
> >
>


-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Impala Build issue

[Suresh Pujari]

Hi Sir,

   I am receiving the below issues.

1. the JDK path
"*http://download.oracle.com/otn-pub/java/jdk/7u75-b13/jdk-7u75-linux-x64.tar.g
z*"
in the install.sh is incorrect because of which errors are coming.

sudo ./install

Compiled Resource:
--
# Declared in /impala-setup/cookbooks/java/recipes/oracle.rb:53:in
`from_file'

java_ark("jdk") do
  action [:install]
  supports {:report=>true, :exception=>true}
  retries 0
  retry_delay 2
  default_guard_interpreter :default
  declared_type :java_ark
  cookbook_name :java
  recipe_name "oracle"
  url "
http://download.oracle.com/otn-pub/java/jdk/7u75-b13/jdk-7u75-linux-x64.tar.gz
"
  default true
  checksum "6f1f81030a34f7a9c987f8b68a24d139"
  app_home "/usr/lib/jvm/java-7-oracle-amd64"
  bin_cmds ["appletviewer", "apt", "ControlPanel", "extcheck", "idlj",
"jar", "jarsigner", "java", "javac", "javadoc", "javafxpackager", "javah",
"javap", "javaws", "jcmd", "jconsole", "jcontrol", "jdb", "jhat", "jinfo",
"jmap", "jps", "jrunscript", "jsadebugd", "jstack", "jstat", "jstatd",
"jvisualvm", "keytool", "native2ascii", "orbd", "pack200", "policytool",
"rmic", "rmid", "rmiregistry", "schemagen", "serialver", "servertool",
"tnameserv", "unpack200", "wsgen", "wsimport", "xjc"]
  alternatives_priority 1062
  connect_timeout 600
  owner "root"
end


Running handlers:
[2017-07-10T00:28:41+04:00] ERROR: Running exception handlers
Running handlers complete
[2017-07-10T00:28:41+04:00] ERROR: Exception handlers complete
[2017-07-10T00:28:41+04:00] FATAL: Stacktrace dumped to
/mnt/DATA/impala-setup/chef-stacktrace.out
Chef Client failed. 4 resources updated in 7.840529619 seconds
[2017-07-10T00:28:41+04:00] ERROR: java_ark[jdk] (java::oracle line 53) had
an error: SystemExit: exit
[2017-07-10T00:28:41+04:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

2. ./bin/bootstrap_build.sh

Linking CXX shared library libgutil.so
[ 10%] Built target gutil
make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
make: *** [impalad] Error 2
Error in /impala-2.9.0/bin/make_impala.sh at line 179: ${MAKE_CMD}
${MAKE_ARGS} ${MAKE_TARGETS}

Please help.

Regards
Suresh