[jira] [Commented] (IMPALA-8973) Update Kudu version to fix openssl1.1.1 compatibility issue
[ https://issues.apache.org/jira/browse/IMPALA-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943326#comment-16943326 ] ASF subversion and git services commented on IMPALA-8973: - Commit 49202ffc51f4c4198b9b03c5c508a237c40c71e3 in impala's branch refs/heads/master from Kurt Deschler [ https://gitbox.apache.org/repos/asf?p=impala.git;h=49202ff ] IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue Automated tests were failing while creating Kudu tables with openssl1.1.1 installed on the machine This issue was previously reported under KUDU-2871 and resolved by disabling TLS 1.3 in Kudu source. This change bumps the Kudu version in impala-config.sh to pull in the fix for KUDU-2871. Testing: Create kudu table via automated tests passes Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Reviewed-on: http://gerrit.cloudera.org:8080/14304 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins > Update Kudu version to fix openssl1.1.1 compatibility issue > --- > > Key: IMPALA-8973 > URL: https://issues.apache.org/jira/browse/IMPALA-8973 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: Kurt Deschler >Assignee: Kurt Deschler >Priority: Major > Labels: kudu > Fix For: Impala 3.4.0 > > > openssl1.1.1/TLS1.3 exposed an issue with Kudu connectivity that was resolved > https://issues.apache.org/jira/browse/KUDU-2871. > This issue was observed causing test failures in kudu tests: > MESSAGE: ImpalaRuntimeException: Error creating Kudu table > 'impala::tpch_kudu.lineitem' > CAUSED BY: NonRecoverableException: not enough live tablet servers to create > a table with the requested replication factor 3; 0 tablet servers are alive > 16:37:21.564919 7150 [heartbeater.cc:566]|http://heartbeater.cc:566]/] > Failed to heartbeat to 127.0.0.1:7051 (0 consecutive failures): Network > error: Failed to ping master at 127.0.0.1:7051: Client connection negotiation > failed: client connection to 127.0.0.1:7051: connect: Connection refused > (error 111) > logs/data_loading/impalad.kurt-cldr.kdeschle.log.INFO.XXX > "not enough live tablet servers" > > impala-config.sh needs to be updated to pull in a newer version of Kudu that > has this fix. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8960) test_drop_if_exists fails on S3 due to incomplete URI
[
https://issues.apache.org/jira/browse/IMPALA-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943327#comment-16943327
]
ASF subversion and git services commented on IMPALA-8960:
-
Commit 88cc930a94c600a87c3fc034733a298a37df9bfc in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=88cc930 ]
IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3
hdfs:// URIs do not work on Impala's S3 tests. The fix is to drop
the hdfs:// part of the URI. Our tests run against the default
filesystem, so dropping the hdfs:// works on both HDFS test setups
and S3 test setups.
Testing:
- Ran tests on S3
- Ran test_owner_privileges.py on HDFS locally
Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332
Reviewed-on: http://gerrit.cloudera.org:8080/14334
Reviewed-by: Michael Ho
Tested-by: Impala Public Jenkins
> test_drop_if_exists fails on S3 due to incomplete URI
> -
>
> Key: IMPALA-8960
> URL: https://issues.apache.org/jira/browse/IMPALA-8960
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Joe McDonnell
>Priority: Critical
>
> Error Message
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Incomplete
> HDFS URI, no host: hdfs:///test-warehouse/libTestUdfs.so CAUSED BY:
> IOException: Incomplete HDFS URI, no host:
> hdfs:///test-warehouse/libTestUdfs.so{noformat}
> Stacktrace
> {noformat}
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Incomplete
> HDFS URI, no host: hdfs:///test-warehouse/libTestUdfs.so CAUSED BY:
> IOException: Incomplete HDFS URI, no host:
> hdfs:///test-warehouse/libTestUdfs.soStacktrace
> authorization/test_owner_privileges.py:137: in test_drop_if_exists
> self._setup_drop_if_exist_test(unique_database, test_db)
> authorization/test_owner_privileges.py:172: in _setup_drop_if_exist_test
> self.execute_query("grant all on uri
> 'hdfs:///test-warehouse/libTestUdfs.so' to"
> common/impala_test_suite.py:751: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:782: in execute_query
> return self.__execute_query(self.client, query, query_options)
> common/impala_test_suite.py:853: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:362: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:356: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:519: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Incomplete HDFS URI, no host:
> hdfs:///test-warehouse/libTestUdfs.so
> E CAUSED BY: IOException: Incomplete HDFS URI, no host:
> hdfs:///test-warehouse/libTestUdfs.so{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8973) Update Kudu version to fix openssl1.1.1 compatibility issue
[ https://issues.apache.org/jira/browse/IMPALA-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Volker resolved IMPALA-8973. - Resolution: Fixed > Update Kudu version to fix openssl1.1.1 compatibility issue > --- > > Key: IMPALA-8973 > URL: https://issues.apache.org/jira/browse/IMPALA-8973 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: Kurt Deschler >Assignee: Kurt Deschler >Priority: Major > Labels: kudu > Fix For: Impala 3.4.0 > > > openssl1.1.1/TLS1.3 exposed an issue with Kudu connectivity that was resolved > https://issues.apache.org/jira/browse/KUDU-2871. > This issue was observed causing test failures in kudu tests: > MESSAGE: ImpalaRuntimeException: Error creating Kudu table > 'impala::tpch_kudu.lineitem' > CAUSED BY: NonRecoverableException: not enough live tablet servers to create > a table with the requested replication factor 3; 0 tablet servers are alive > 16:37:21.564919 7150 [heartbeater.cc:566]|http://heartbeater.cc:566]/] > Failed to heartbeat to 127.0.0.1:7051 (0 consecutive failures): Network > error: Failed to ping master at 127.0.0.1:7051: Client connection negotiation > failed: client connection to 127.0.0.1:7051: connect: Connection refused > (error 111) > logs/data_loading/impalad.kurt-cldr.kdeschle.log.INFO.XXX > "not enough live tablet servers" > > impala-config.sh needs to be updated to pull in a newer version of Kudu that > has this fix. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9002) Add flag to only check SELECT priviledge in GET_TABLES
Quanlong Huang created IMPALA-9002:
--
Summary: Add flag to only check SELECT priviledge in GET_TABLES
Key: IMPALA-9002
URL: https://issues.apache.org/jira/browse/IMPALA-9002
Project: IMPALA
Issue Type: Improvement
Components: Security
Reporter: Quanlong Huang
In Frontend.doGetTableNames(), if authorization is enabled, we only return
tables that current user has ANY priviledge on them:
{code:java}
private List doGetTableNames(String dbName, PatternMatcher matcher,
User user) throws ImpalaException {
FeCatalog catalog = getCatalog();
List tblNames = catalog.getTableNames(dbName, matcher);
if (authzFactory_.getAuthorizationConfig().isEnabled()) {
Iterator iter = tblNames.iterator();
while (iter.hasNext()) {
..
PrivilegeRequest privilegeRequest = new PrivilegeRequestBuilder(
authzFactory_.getAuthorizableFactory())
.any().onAnyColumn(dbName, tblName, tableOwner).build(); <--
require ANY priviledge here
if (!authzChecker_.get().hasAccess(user, privilegeRequest)) {
iter.remove();
}
}
}
return tblNames;
} {code}
In Sentry integration, checking ANY priviledge will check all possible
priviledges, i.e. ALL, OWNER, ALTER, DROP, CREATE, INSERT, SELECT, REFRESH,
until one is permitted. In the worst case that current use don't have any
priviledge on a table, we need to perform 8 checks on this table.
{code:java}
public enum Privilege {
...
static {
...
ANY.implied_ = EnumSet.of(ALL, OWNER, ALTER, DROP, CREATE, INSERT, SELECT,
REFRESH); {code}
GET_TABLES performance is poor when there're thosands of tables. It's
reasonable to only return tables that current user has SELECT priviledge on
them. Checking only the SELECT priviledge can boost the perfomance to be 8
times better. In my experiment on impala-2.12-cdh5.16.2 with 40k tables,
GET_TABLES takes 16s originally when current user only have priviledges on 6
tables. With this change, time reduces to 2s.
We can add a flag to only check on SELECT priviledge for table visuability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8998) Admission control accounting for mt_dop
[ https://issues.apache.org/jira/browse/IMPALA-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8998 started by Tim Armstrong. - > Admission control accounting for mt_dop > --- > > Key: IMPALA-8998 > URL: https://issues.apache.org/jira/browse/IMPALA-8998 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > We should account for the degree of parallelism that the query runs with on a > backend to avoid overadmitting too many parallel queries. > We could probably simply count the effective degree of parallelism (max # > instances of a fragment on that backend) toward the number of slots in > admission control (although slots are not enabled for the default group yet - > see IMPALA-8757). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8997) Allow graceful fallback to mt_dop=0
[ https://issues.apache.org/jira/browse/IMPALA-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8997 started by Tim Armstrong. - > Allow graceful fallback to mt_dop=0 > --- > > Key: IMPALA-8997 > URL: https://issues.apache.org/jira/browse/IMPALA-8997 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > We should provide a way to specify mt_dop and have Impala automatically fall > back to non-mt_dop for queries where it isn't supported. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9000) Remove all the TODO-MT comments
Tim Armstrong created IMPALA-9000: - Summary: Remove all the TODO-MT comments Key: IMPALA-9000 URL: https://issues.apache.org/jira/browse/IMPALA-9000 Project: IMPALA Issue Type: Sub-task Reporter: Tim Armstrong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9000) Fix all the TODO-MT comments
[ https://issues.apache.org/jira/browse/IMPALA-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-9000: -- Summary: Fix all the TODO-MT comments (was: Remove all the TODO-MT comments) > Fix all the TODO-MT comments > > > Key: IMPALA-9000 > URL: https://issues.apache.org/jira/browse/IMPALA-9000 > Project: IMPALA > Issue Type: Sub-task >Reporter: Tim Armstrong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8946) Prometheus histograms do not follow conventions
[
https://issues.apache.org/jira/browse/IMPALA-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943153#comment-16943153
]
ASF subversion and git services commented on IMPALA-8946:
-
Commit f08d21335c5a81d31a3de7dc838ac4487080ac42 in impala's branch
refs/heads/master from Guillem Nieto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f08d213 ]
IMPALA-8946: Fix histogram rendering to Prometheus
Prometheus has two formats that allows to calculate quantiles (on client
side): Histograms and Summaries. The former requires to pick a set of
buckets, while the latter only requires to specifiy which quantiles
(percentiles) the application wants to provide. The difference between
summaries and histograms on Prometheus can be found here:
https://prometheus.io/docs/practices/histograms/
Previous to this changes, Impala histograms were rendered as Prometheus
histograms, but it was calculating percentiles instead of bucketing the
recorded values, which is not what Prometheus clients expects.
>From now on, Impala histograms will be rendered as Prometheus summaries
instead, which expects quantiles.
To be compliant with Prometheus summaries, a `_sum` serie needs to
be added, containing the sum of all the registered values.
The current `HdrHistogram` implementation on be::utils does not have
the `TotalSum` method (and the required properties to track it). I
copied the implementation from be::kudu::utils and added a couple of
unit tests which covers the sum method.
Also adapted the Prometheus tests to verify that sum serie is properly
rendered.
Change-Id: I8849586964140c040ca4ea14b777c2d98d126b59
Reviewed-on: http://gerrit.cloudera.org:8080/14280
Tested-by: Impala Public Jenkins
Reviewed-by: Tim Armstrong
> Prometheus histograms do not follow conventions
> ---
>
> Key: IMPALA-8946
> URL: https://issues.apache.org/jira/browse/IMPALA-8946
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Guillem
>Assignee: Guillem
>Priority: Minor
> Labels: observability
> Fix For: Impala 3.4.0
>
> Attachments:
> 0001-IMPALA-8946-Fix-histogram-rendering-to-Prometheus.patch,
> 0001-IMPALA-8946-Fix-histogram.patch
>
>
> We've been using Prometheus metrics and we've found that some standard
> Prometheus parser can not properly interpret histograms from Impala.
> For example, Python official client
> ([https://github.com/prometheus/client_python)] can not properly read them.
> I've been digging a little bit why it can't read them and I've found that
> Impala does not adhere to textual histogram conventions.
> The following link describes the conventions for rendering histograms on
> Prometheus textual format:
> [https://prometheus.io/docs/instrumenting/exposition_formats/#histograms-and-summaries]
> This is an example of a rendered histogram on Impala 3.3 on Prometheus
> endpoint:
> {code:java}
> # HELP impala_thrift_server_backend_svc_thread_wait_time Amount of time
> clients of Impala Backend Server spent waiting for service threads
> # TYPE impala_thrift_server_backend_svc_thread_wait_time histogram
> impala_thrift_server_backend_svc_thread_wait_time{le="0.2"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.5"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.7"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.9"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.95"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.999"} 0
> impala_thrift_server_backend_svc_thread_wait_time_max 0
> impala_thrift_server_backend_svc_thread_wait_time_min 0
> impala_thrift_server_backend_svc_thread_wait_time_count 49
> {code}
> The linked histogram conventions say that
> {quote}Each bucket count of a histogram named x is given as a separate sample
> line with the name x_bucket and a label \{le="y"} (where y is the upper bound
> of the bucket).
> {quote}
> And also
> {quote}A histogram must have a bucket with \{le="+Inf"}. Its value must be
> identical to the value of x_count.
> {quote}
> The previous example should be formatted as:
> {code:java}
> # HELP impala_thrift_server_backend_svc_thread_wait_time Amount of time
> clients of Impala Backend Server spent waiting for service threads
> # TYPE impala_thrift_server_backend_svc_thread_wait_time histogram
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.2"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.5"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.7"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.9"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.95"} 0
>
