[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. Patch Set 5: > Patch Set 3: Code-Review+2 > > LGTM. Welcome back to the community, Fredy! We really miss you. > > BTW, it'd be nice if you could also update the wiki about this: > https://cwiki.apache.org/confluence/display/IMPALA/Impala+Development+Environment+inside+Docker Thanks Quanlong for the LGTM :) Done. Updated the wiki. -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Aug 2023 16:00:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. [tools] Add Dev Container support for Impala development. Currently only VS Code is supported since IntelliJ/CLion support for Dev Container is still beta at the time of this writing. To use it, simply open Impala source code. $ git clone https://github.com/apache/impala.git $ cd impala $ code . The bootstrap_development.sh will be automatically executed post Docker container creation and all necessary extensions for IDE-like experience will be automatically installed. For C++, it will use clangd that uses compilation database instead of the Microsoft C++ extension since it works better with Clang related tools. Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 --- A .devcontainer/Dockerfile A .devcontainer/devcontainer.json M bin/rat_exclude_files.txt 3 files changed, 33 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/20380/3 -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. [tools] Add Dev Container support for Impala development. Currently only VS Code is supported since IntelliJ/CLion support for Dev Container is still beta at the time of this writing. To use it, simply open Impala source code. $ git clone https://github.com/apache/impala.git $ cd impala $ code . The bootstrap_development.sh will be automatically executed post Docker container creation and all necessary extensions for IDE-like experience will be automatically installed. For C++, it will use clangd that uses compilation database instead of the Microsoft C++ extension since it works better with Clang related tools. Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 --- A .devcontainer/Dockerfile A .devcontainer/devcontainer.json 2 files changed, 32 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/20380/2 -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20380 Change subject: [tools] Add Dev Container support for Impala development. .. [tools] Add Dev Container support for Impala development. Currently only VS Code is supported since IntelliJ/CLion support for Dev Container is still beta at the time of this writing. To use it, simply open Impala source code. $ git clone https://github.com/apache/impala.git $ cd impala $ code . The bootstrap_development.sh will be automatically executed post Docker container creation and all necesary extensions for IDE-like experience will be automatically installed. For C++, it uses clangd that uses compilation database instead of the Microsoft C++ extension since it works better with Clang related tools. Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 --- A .devcontainer/Dockerfile A .devcontainer/devcontainer.json 2 files changed, 32 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/20380/1 -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 5: > Patch Set 5: > > Hi Fredy, > Just checking in if you had some time to review the current version. LGTM. Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 15 Sep 2020 16:30:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 15 Sep 2020 16:30:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 01 Sep 2020 14:28:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/16389/2/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/16389/2/shell/impala_shell.py@1280 PS2, Line 1280: if self.DML_REGEX.match(query_type.lower()): looks like there were failed tests in the dry-run nit: this code can be simplified like below. is_dml = self.DML_REGEX.match(query_type.lower()) return self._execute_stmt(query, is_dml=is_dml, print_web_link=True) -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 31 Aug 2020 22:06:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/14106 )
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
Patch Set 11: Code-Review+2
(3 comments)
LGTM.
http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/main/java/org/apache/impala/authorization/Authorizable.java
File fe/src/main/java/org/apache/impala/authorization/Authorizable.java:
http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@65
PS11, Line 65: int ownerHash = getOwnerUser() == null ? 0 :
getOwnerUser().hashCode();
: int nameHash = getName().hashCode();
: return nameHash * 31 + ownerHash;
nit: You can use Objects.hash(nameHash, ownerHash)
http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File
fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3068
PS11, Line 3068:
nit: I think Java 8 can infer this without explicitly specifying the type
parameters.
http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3069
PS11, Line 3069: .put(authorize("select count(*) from
functional.alltypes"),
: selectError("functional.alltypes"))
: .put(authorize("select id from functional.alltypes"),
: selectError("functional.alltypes"))
: .put(authorize("select id from
functional.alltypes_view"),
: selectError("functional.alltypes_view"))
: .put(authorize("show create table
functional.alltypes"),
: accessError("functional.alltypes"))
: .put(authorize("describe functional.alltypes"),
: accessError("functional.alltypes"))
: .put(authorize("show create table
functional.alltypes_view"),
: accessError("functional.alltypes_view"))
: .put(authorize("describe functional.alltypes_view"),
: accessError("functional.alltypes_view"))
: .put(authorize("describe
functional.allcomplextypes.int_struct_col"),
: accessError("functional.allcomplextypes"))
: .put(authorize("refresh functional.alltypes"),
: refreshError("functional.alltypes"))
: .put(authorize("invalidate metadata
functional.alltypes"),
: refreshError("functional.alltypes"))
: .put(authorize("compute stats functional.alltypes"),
: alterError("functional.alltypes"))
: .put(authorize("drop stats functional.alltypes"),
: alterError("functional.alltypes"))
: .put(authorize("create table functional.test_tbl(a
int)"),
: createError("functional"))
: .put(authorize("create table functional.test_tbl like
functional.alltypes"),
: accessError("functional.alltypes"))
: .put(authorize("create table functional.test_tbl as
select 1"),
: createError("functional"))
: .put(authorize("create view functional.test_view as
select 1"),
: createError("functional"))
: .put(authorize("alter table functional.alltypes add
column c1 int"),
: alterError("functional"))
: .put(authorize("drop table functional.alltypes"),
: dropError("functional"))
: .put(authorize("drop view functional.alltypes_view"),
: dropError("functional"))
: .put(authorize("alter view functional.alltypes_view as
select 1"),
: alterError("functional.alltypes_view"))
: .put(authorize("alter database functional set owner
user foo"),
: accessError(true, "functional"))
: .build();
nit: should be indented.
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 11
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer:
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/14106 )
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
Patch Set 6:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java
File fe/src/main/java/org/apache/impala/authorization/Authorizable.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@62
PS6, Line 62: return getName().hashCode();
If we're comparing owner, we should update the hashCode() as well.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@70
PS6, Line 70: temp.getOwnerUser() == this.getOwnerUser()
Shouldn't this use equals()?
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@31
PS6, Line 31: private final String ownerUser_;
For consistency, annotate with @Nullable.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@34
PS6, Line 34: String ownerUser
I think it's more readable to also annotate the parameter with @Nullable.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java@33
PS6, Line 33: String ownerUser
Same comment about @Nullable parameter.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java@45
PS6, Line 45: /**
:* Creates a new instance of column {@link Authorizable} for a
given database name and
:* gives access to all tables and columns.
:*/
: Authorizable newColumnAllTbls(String dbName, @Nullable String
dbOwnerUser);
:
: /**
:* Creates a new instance of column {@link Authorizable} for
given database and table
:* names and gives access to all columns.
:*/
: Authorizable newColumnInTable(
: String dbName, String tableName, @Nullable String
tblOwnerUser);
:
: /**
:* Creates a new instance of column {@link Authorizable} for
given database, table, and
:* column names.
:*/
: Authorizable newColumnInTable(
: String dbName, String tableName, String columnName,
@Nullable String tblOwnerUser);
nit: update javadoc about the new owner parameter.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java@34
PS6, Line 34: String ownerUser
Same comment about @Nullable parameter.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@94
PS6, Line 94: resources.add(new RangerImpalaResourceBuilder()
: .database("*").table("*").column("*").build());
: resources.add(new RangerImpalaResourceBuilder()
: .database("*").function("*").build());
: resources.add(new
RangerImpalaResourceBuilder().uri("*").build());
Should we add .owner() here as well?
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/catalog/Db.java@544
PS6, Line 544: if (db == null) return null;
: return db.getOwnerName();
nit: can be inlined: return db == null ? null : db.getOwnerName()
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
[Impala-ASF-CR] IMPALA-8750: Fix profile observability tests
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13851 ) Change subject: IMPALA-8750: Fix profile observability tests .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I65a1e81870e808f0e261f8a6097efdcc6903912a Gerrit-Change-Number: 13851 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 18 Jul 2019 05:45:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8750: Fix profile observability tests
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13851 )
Change subject: IMPALA-8750: Fix profile observability tests
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13851/1/tests/common/impala_service.py
File tests/common/impala_service.py:
http://gerrit.cloudera.org:8080/#/c/13851/1/tests/common/impala_service.py@90
PS1, Line 90: 'flags'
nit: it's better to not mix quotes. For consistency, maybe use "" instead.
http://gerrit.cloudera.org:8080/#/c/13851/1/tests/common/impala_service.py@90
PS1, Line 90: for var in varz.get('flags'):
: if var["name"] == flag:
: return var["current"]
what should this method return when varz.get("flags") is empty?
http://gerrit.cloudera.org:8080/#/c/13851/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/13851/1/tests/query_test/test_observability.py@326
PS1, Line 326: 'lineage_event_log_dir'
nit: use double quotes for consistency
http://gerrit.cloudera.org:8080/#/c/13851/1/tests/query_test/test_observability.py@327
PS1, Line 327: print (lineage_event_log_dir_value)
remove debug information
--
To view, visit http://gerrit.cloudera.org:8080/13851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I65a1e81870e808f0e261f8a6097efdcc6903912a
Gerrit-Change-Number: 13851
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Comment-Date: Mon, 15 Jul 2019 04:50:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8734: Reload table schema on TBLPROPERTIES change
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13785 )
Change subject: IMPALA-8734: Reload table schema on TBLPROPERTIES change
..
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py@697
PS3, Line 697: self.execute_query_expect_success(self.client, "create table
{0}.{1} (c1 string)"
> Is there a reason for not creating a new test? The new code seems to be com
No reason, I thought it would be simpler, but yeah it seems creating a new test
is better. Done.
http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py@702
PS3, Line 702: ct_s
> Using a less "null like" value could make the test's intention clearer - it
I was following the test case, but yeah using a different value is more
readable. Done.
http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py@701
PS3, Line 701:file_data="\nfoo\n")
: self.execute_query_expect_success(self.client,
: "alter table {0}.{1} set
tblproperties"
:
"('serialization.null.format'='foo')"
: .format(unique_database,
tbl_name))
: result = self.execute_query_expect_success(self.client,
:"select * from
{0}.{1}"
:
.format(unique_database, tbl_name))
: assert len(result.data) == 2
: assert result.data[0] == ''
> First I thought that it would be nice to create a utility function for this
TIL. Done.
--
To view, visit http://gerrit.cloudera.org:8080/13785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad
Gerrit-Change-Number: 13785
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 02 Jul 2019 19:52:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8734: Reload table schema on TBLPROPERTIES change
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13785 ) Change subject: IMPALA-8734: Reload table schema on TBLPROPERTIES change .. IMPALA-8734: Reload table schema on TBLPROPERTIES change Prior to this patch, an INVALIDATE METADATA was required when altering the TBLPROPERTIES for the changes to take effect. With this patch the table schema is automatically reloaded on TBLPROPERTIES change. Testing: - Added a new test in test_ddl.py - Ran test_ddl.py Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_ddl.py 2 files changed, 20 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13785/4 -- To view, visit http://gerrit.cloudera.org:8080/13785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad Gerrit-Change-Number: 13785 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8734: Reload table schema on TBLPROPERTIES change
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13785 Change subject: IMPALA-8734: Reload table schema on TBLPROPERTIES change .. IMPALA-8734: Reload table schema on TBLPROPERTIES change Prior to this patch, an INVALIDATE METADATA was required when altering the TBLPROPERTIES for the changes to take effect. With this patch the table schema is automatically reloaded on TBLPROPERTIES change. Testing: - Added a new test in test_ddl.py - Ran test_ddl.py Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_ddl.py 2 files changed, 25 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13785/3 -- To view, visit http://gerrit.cloudera.org:8080/13785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad Gerrit-Change-Number: 13785 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13670 ) Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 02 Jul 2019 13:22:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 3 files changed, 101 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/5 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. Patch Set 5: Code-Review+2 Carry Bharath's +2. -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 01 Jul 2019 21:41:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@319 PS2, Line 319: // so that we know all the audit events generated by the temporary audit : // handler are contained to the given resource. : RangerBufferAuditHandler tmpAuditHandler = originalAuditHandler == null ? : null : new RangerBufferAuditHandler(originalAuditHandler); : for (Privilege impliedPrivilege: privile > I think you missed the comment. Done -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 01 Jul 2019 21:41:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 3 files changed, 95 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/3 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13744 )
Change subject: IMPALA-8716: Log a group of privileges into a single audit
event.
..
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@318
PS2, Line 318: fferAu
> nit: authorized ? (same below)
Done
http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@319
PS2, Line 319: null : new RangerBufferAuditHandler(originalAuditHandler);
: for (Privilege impliedPrivilege:
privilege.getImpliedPrivileges()) {
: if (authorizeResource(authzCtx, resource, user,
impliedPrivilege,
: tmpAuditHandler)) {
: authorized = true;
> I think this could use a comment. Also, create a copy c'tor? (same below)
Good idea about copy constructor. Done.
http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@385
PS2, Line 385: RangerBufferAuditHandler auditHandler) throws
InternalException {
> just checking, is the accessResult automatically inferred from newAuditEven
Yeah since we're assigning newAuditEvent to an existing instance and mutating
it, so it will have the accessResult.
http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
File
fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java:
http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java@141
PS2, Line 141: 1
> Mind adding a couple more tests?
Those test cases have already been been covered:
accessResult = 0, isAny = true in L193-L199
accessResult = 1, isAny = false in L63-L136 --> isAny = false is basically all
privileges in
https://github.com/apache/impala/blob/c353cf7a648651244ac39677d0cb028e704281d0/fe/src/main/java/org/apache/impala/authorization/Privilege.java#L28-L35
(anyOf_ = false)
--
To view, visit http://gerrit.cloudera.org:8080/13744
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca
Gerrit-Change-Number: 13744
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 01 Jul 2019 19:23:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@408 PS2, Line 408: RangerAccessResult result = plugin_.isAccessAllowed(request, auditHandler); > Instead of modifying entries in a tmpAuditHandler, can we make it simple by Tthe plugin._isAccessAllowed() does some logic that passes the correct RangerAccessResult including populating some default values. Calling it manually means we need to always keep the logic similar to the one implemented by isAccessAllowed. -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 01 Jul 2019 14:16:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13673 )
Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@367
PS3, Line 367: private static class RangerResourceResult {
: List server = new ArrayList<>();
: List uri = new ArrayList<>();
: List database = new ArrayList<>();
: List udf = new ArrayList<>();
: List table = new ArrayList<>();
: List column = new ArrayList<>();
make these private
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@392
PS3, Line 392: public RangerResourceResult addUdfResult(RangerResultRow
result) {
: udf.add(result);
: return this;
: }
:
: public RangerResourceResult addUriResult(RangerResultRow
result) {
: uri.add(result);
: return this;
: }
These two methods are unused, that means uri and udf will always be empty.
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@407
PS3, Line 407: public List getResultRows() {
: List results = new ArrayList<>();
:
: results.addAll(filterIfAll(server));
: results.addAll(filterIfAll(database));
: results.addAll(filterIfAll(table));
: results.addAll(filterIfAll(column));
: results.addAll(filterIfAll(udf));
: results.addAll(filterIfAll(uri));
:
: return results;
: }
I don't quite follow the logic why we have to filter ALL. A comment will be
helpful.
--
To view, visit http://gerrit.cloudera.org:8080/13673
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c
Gerrit-Change-Number: 13673
Gerrit-PatchSet: 3
Gerrit-Owner: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 28 Jun 2019 20:38:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 16:40:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 4: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@131 PS4, Line 131: tableName_.toString() Just an FYI usually it's better to not call toString() explicitly in the logger since toString() will be lazily called by the logger. For example if we had a very expensive toString() call, calling LOG.info() on a different log level such as WARN or ERROR would not incur an expensive toString() call. I don't think it matters for this, though. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 15:18:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Patch Set 2: > Patch Set 2: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4559/ Looks like flaky Maven issue. Retrying the merge. 02:24:28 [ERROR] Failed to execute goal on project impala-minimal-hive-exec: Could not resolve dependencies for project org.apache.impala:impala-minimal-hive-exec:jar:0.1-SNAPSHOT: Could not transfer artifact io.dropwizard.metrics:metrics-core:jar:3.1.0 from/to impala.cdp.repo (https://native-toolchain.s3.amazonaws.com/build/cdp_components/1153860/maven): Access denied to: https://native-toolchain.s3.amazonaws.com/build/cdp_components/1153860/maven/io/dropwizard/metrics/metrics-core/3.1.0/metrics-core-3.1.0.jar , ReasonPhrase:Forbidden. -> [Help 1] 02:24:28 [ERROR] 02:24:28 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 02:24:28 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 02:24:28 [ERROR] 02:24:28 [ERROR] For more information about the errors and possible solutions, please read the following articles: 02:24:28 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException 02:24:28 mvn -U -B install -DskipTests exited with code 0 02:24:28 shaded-deps/CMakeFiles/shaded-deps.dir/build.make:57: recipe for target 'shaded-deps/CMakeFiles/shaded-deps' failed 02:24:28 make[2]: *** [shaded-deps/CMakeFiles/shaded-deps] Error 1 02:24:28 CMakeFiles/Makefile2:14411: recipe for target 'shaded-deps/CMakeFiles/shaded-deps.dir/all' failed 02:24:28 make[1]: *** [shaded-deps/CMakeFiles/shaded-deps.dir/all] Error 2 02:24:28 Makefile:94: recipe for target 'all' failed 02:24:28 make: *** [all] Error -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 15:14:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 2 files changed, 87 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/2 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13745 Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Add ext-py/bitarray-0.9.0 in .gitignore Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 --- M shell/.gitignore 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/13745/1 -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13744 Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 2 files changed, 81 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/1 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 15 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 26 Jun 2019 18:41:37 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix incorrect timestamp units in the logs
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13731 ) Change subject: Fix incorrect timestamp units in the logs .. Patch Set 2: Code-Review+2 > Patch Set 2: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4547/ Unrelated flaky tests. Re-running the merge. -- To view, visit http://gerrit.cloudera.org:8080/13731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I132a007404dcf5291e70f2642ed3e9f995d694b1 Gerrit-Change-Number: 13731 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 26 Jun 2019 17:25:59 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix incorrect timestamp units in the logs
Fredy Wijaya has removed a vote on this change. Change subject: Fix incorrect timestamp units in the logs .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/13731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I132a007404dcf5291e70f2642ed3e9f995d694b1 Gerrit-Change-Number: 13731 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13508 )
Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java:
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@40
PS3, Line 40: Expr
Shouldn't this be DropTableOrViewStmt?
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@131
PS3, Line 131: "Ignoring TableLoadingException for " + dbName_ + "." +
tableName_
nit: it's usually preferable to use the SLF4J log format, e.g.
LOG.info("Ignoring TableLoadingException for {}.{}", dbName_, getTbl());
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@134
PS3, Line 134: tableName_
tableName_.toString() may print "db.tbl":
https://github.com/apache/impala/blob/572c1278566ca329f3e47e5c2d7e13dd0c7967f7/fe/src/main/java/org/apache/impala/analysis/TableName.java#L85-L92,
should we use getTbl() instead similar to L115 and L120?
--
To view, visit http://gerrit.cloudera.org:8080/13508
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952
Gerrit-Change-Number: 13508
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 26 Jun 2019 17:24:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13346 )
Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper
Sentry API
..
Patch Set 11:
(5 comments)
Thanks for the fixes! This is getting close. I'm ready to give a +2 once all
the comments are resolved.
http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java
File
fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java:
http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java@535
PS11, Line 535: public boolean isSentryAdmin(User requestingUser) throws
InternalException,
can we add javadoc?
http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java@535
PS11, Line 535: throws InternalException,
nit: I think it's more readable to put throws InternalException,
SentryUserException in the next line.
http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java@537
PS11, Line 537: SentryServiceClient client = new SentryServiceClient();
: try {
: return client.get().isAdmin(requestingUser.getName());
: } finally {
: client.close();
: }
nit: using try-resource is shorter, i.e.
try (SentryServiceClient client = new SentryServiceClient()) {
return client.get().isAdmin(requestingUser.getName());
}
http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/service/JniCatalog.java@304
PS11, Line 304: try {
: boolean isSentryAdmin =
((SentryCatalogdAuthorizationManager)
:
catalogOpExecutor_.getAuthzManager()).isSentryAdmin(user);
: response.setIs_admin(isSentryAdmin);
: } catch (SentryUserException e) {
: // When a user is not defined in Sentry, isAdmin() will
throw
: // SentryUserException, we will consider this requesting
user
: // as a non-Sentry administrator.
: response.setIs_admin(false);
: }
Sorry for the back and forth, but putting this logic in JniCatalog doesn't feel
quite right since JniCatalog is supposed to be a thin JNI wrapper. It makes
more sense to move the try/catch logic in the SentryProxy instead.
http://gerrit.cloudera.org:8080/#/c/13346/11/tests/authorization/test_sentry.py
File tests/authorization/test_sentry.py:
http://gerrit.cloudera.org:8080/#/c/13346/11/tests/authorization/test_sentry.py@51
PS11, Line 51: root
nit: Calling it non_admin makes it easier to understand.
--
To view, visit http://gerrit.cloudera.org:8080/13346
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82
Gerrit-Change-Number: 13346
Gerrit-PatchSet: 11
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 26 Jun 2019 03:57:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Fix incorrect timestamp units in the logs
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13731 ) Change subject: Fix incorrect timestamp units in the logs .. Patch Set 1: Code-Review+2 Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/13731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I132a007404dcf5291e70f2642ed3e9f995d694b1 Gerrit-Change-Number: 13731 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 26 Jun 2019 03:40:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13713 ) Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. Patch Set 8: Code-Review+2 (2 comments) Carry Bharath's +2. http://gerrit.cloudera.org:8080/#/c/13713/7/fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java File fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java: http://gerrit.cloudera.org:8080/#/c/13713/7/fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java@24 PS7, Line 24: /** > Add a class comment on how this is supposed to work? Done http://gerrit.cloudera.org:8080/#/c/13713/7/tests/custom_cluster/test_query_event_hooks.py File tests/custom_cluster/test_query_event_hooks.py: http://gerrit.cloudera.org:8080/#/c/13713/7/tests/custom_cluster/test_query_event_hooks.py@112 PS7, Line 112: > remove? Done -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 25 Jun 2019 18:24:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13713 ) Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. IMPALA-8589: Re-enable flaky test_query_event_hooks.py This patch fixes the flaky test_query_event_hooks.py. The patch also cuts down the waiting time for impalad timeout to 5 seconds from the default 60 seconds especially for those tests that will fail Impala startup. Testing: - Ran test_query_event_hooks.py in a loop. Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 --- M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java M fe/src/test/java/org/apache/impala/testutil/AlwaysErrorQueryEventHook.java M fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java M fe/src/test/java/org/apache/impala/testutil/PostQueryErrorEventHook.java M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_query_event_hooks.py 6 files changed, 126 insertions(+), 196 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/13713/8 -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13713 ) Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. Patch Set 7: Rebased and re-running the dry-run. -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 25 Jun 2019 15:51:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13713 Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. IMPALA-8589: Re-enable flaky test_query_event_hooks.py This patch fixes the flaky test_query_event_hooks.py. The patch also cuts down the waiting time for impalad timeout to 5 seconds from the default 60 seconds especially for those tests that will fail Impala startup. Testing: - Ran test_query_event_hooks.py in a loop. Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 --- M fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_query_event_hooks.py 3 files changed, 67 insertions(+), 142 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/13713/6 -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 22:00:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py@1733 PS1, Line 1733: if options.output_delimiter != "\\t": Can we not check if the delimiter is not empty then perform the validation? http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py@1738 PS1, Line 1738: sys.exit(1) Raise FatalShellException to be consistent with the rest of the code here. Can we also add/update the test to catch this issue? -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 20:24:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. IMPALA-8682: Add authorized proxy user/group test coverage with Ranger This patch adds a test coverage for authorized proxy user/group with Ranger. This patch also moves the authorized proxy tests into a separate file, test_authorized_proxy and refactors the tests to be more readable and reusable. Testing: - Added a new test_authorized_proxy.py - Ran all E2E authorization tests Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 --- M tests/authorization/test_authorization.py A tests/authorization/test_authorized_proxy.py 2 files changed, 275 insertions(+), 145 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/13679/5 -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py File tests/authorization/test_authorized_proxy.py: http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@195 PS4, Line 195: s > what are these commented-out lines? I meant to delete that. Done. http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@231 PS4, Line 231: > typo? Done -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 17:14:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 10: > Patch Set 10: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4500/ This looks like some flaky Maven issue. I'm re-running the merge. 23:59:12 ] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project query-event-hook-api: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.3 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-compiler-plugin:jar:3.3 -> org.codehaus.plexus:plexus-container-default:jar:1.5.5 -> org.apache.xbean:xbean-reflect:jar:3.4: Failed to read artifact descriptor for org.apache.xbean:xbean-reflect:jar:3.4: Could not transfer artifact org.apache.xbean:xbean-reflect:pom:3.4 from/to central (https://repo.maven.apache.org/maven2): /home/ubuntu/.m2/repository/org/apache/xbean/xbean-reflect/3.4/xbean-reflect-3.4.pom.part (No such file or directory) -> [Help 1] 23:59:12 ] [ERROR] 23:59:12 ] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 23:59:12 ] [ERROR] Re-run Maven using the -X switch to enable full debug logging. 23:59:12 ] [ERROR] 23:59:12 ] [ERROR] For more information about the errors and possible solutions, please read the following articles: 23:59:12 ] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException 23:59:12 ] mvn -B install -DskipTests exited with code 0 23:59:12 ] query-event-hook-api/CMakeFiles/query-event-hook-api.dir/build.make:57: recipe for target 'query-event-hook-api/CMakeFiles/query-event-hook-api' failed 23:59:12 ] make[3]: *** [query-event-hook-api/CMakeFiles/query-event-hook-api] Error 1 23:59:12 ] CMakeFiles/Makefile2:14640: recipe for target 'query-event-hook-api/CMakeFiles/query-event-hook-api.dir/all' failed 23:59:12 ] make[2]: *** [query-event-hook-api/CMakeFiles/query-event-hook-api.dir/all] Error 2 23:59:12 ] make[2]: *** Waiting for unfinished jobs -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 17:13:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py File tests/authorization/test_authorized_proxy.py: http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py@26 PS3, Line 26: from getpass import getuser > flake8: F811 redefinition of unused 'time' from line 21 Done http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py@120 PS3, Line 120: t > flake8: E131 continuation line unaligned for hanging indent Done http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py@140 PS3, Line 140: t > flake8: E131 continuation line unaligned for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 18:51:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. IMPALA-8682: Add authorized proxy user/group test coverage with Ranger This patch adds a test coverage for authorized proxy user/group with Ranger. This patch also moves the authorized proxy tests into a separate file, test_authorized_proxy and refactors the tests to be more readable and reusable. Testing: - Added a new test_authorized_proxy.py - Ran all E2E authorization tests Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 --- M tests/authorization/test_authorization.py A tests/authorization/test_authorized_proxy.py 2 files changed, 278 insertions(+), 145 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/13679/4 -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13679 Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. IMPALA-8682: Add authorized proxy user/group test coverage with Ranger This patch adds a test coverage for authorized proxy user/group with Ranger. This patch also moves the authorized proxy tests into a separate file, test_authorized_proxy and refactors the tests to be more readable and reusable. Testing: - Added a new test_authorized_proxy.py - Ran all E2E authorization tests Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 --- M tests/authorization/test_authorization.py A tests/authorization/test_authorized_proxy.py 2 files changed, 279 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/13679/3 -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 19 Jun 2019 18:28:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 22 files changed, 223 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/11 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13673 )
Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant
..
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG@7
PS3, Line 7: [IMPALA-8587]
nit: use IMPALA-8587 format
http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG@20
PS3, Line 20: They would see no results. After this change, the user will see
database
: level privileges when executing the previous statement. If a user
has
: SELECT privilege on DATABASE and on TABLE and issues a show grant
on
: TABLE, they will only see the SELECT privilege for TABLE. Users
will not
: see multiple instances of SELECT or any other privilege type in a
SHOW
: GRANT statemenet.
Show what the new output looks like. It'll be much easier to understand.
We also need more explanation about this especially since this is relates to
how the Ranger policy engine works.
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@330
PS3, Line 330: !=
Use equals(). != or == is for reference equality. Sometimes you get lucky
because string intern, but we shouldn't rely on that.
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@329
PS3, Line 329: for (RangerResultRow row : resultRows) {
: if (row.column_ != "*" && !row.column_.isEmpty()) {
: resourceResult.addColumnResult(row);
: } else if (row.table_ != "*" && !row.table_.isEmpty()) {
: resourceResult.addTableResult(row);
: } else if (row.database_ != "*" &&
!row.database_.isEmpty()) {
: resourceResult.addDatabaseResult(row);
: } else {
: resourceResult.addServerResult(row);
: }
: }
Can you add a comment for this logic? It's not quite clear to me what it's
trying to do.
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@422
PS3, Line 422:
nit: overly indented, use 4 spaces
--
To view, visit http://gerrit.cloudera.org:8080/13673
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c
Gerrit-Change-Number: 13673
Gerrit-PatchSet: 3
Gerrit-Owner: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 18 Jun 2019 21:08:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory...
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13670 )
Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless
memory...
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG@7
PS3, Line 7: ...
remove ...
http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc@135
PS3, Line 135: if (mem_tracker != NULL) {
It's less nested to do if (mem_tracker == nullptr) return;
--
To view, visit http://gerrit.cloudera.org:8080/13670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23
Gerrit-Change-Number: 13670
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 18 Jun 2019 18:56:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 8: Code-Review+2 Thanks for the patch! -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 8 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 18 Jun 2019 16:30:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 225 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/8 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13653 ) Change subject: IMPALA-8599: Create a Maven module for query event hook API .. Patch Set 3: > Patch Set 3: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4482/ Looks like network issue: 23:29:56 ] [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.6.201602180812:prepare-agent (prepare-jacoco-agent) on project impala-frontend: Execution prepare-jacoco-agent of goal org.jacoco:jacoco-maven-plugin:0.7.6.201602180812:prepare-agent failed: Plugin org.jacoco:jacoco-maven-plugin:0.7.6.201602180812 or one of its dependencies could not be resolved: Could not transfer artifact org.apache.maven.shared:maven-shared-io:jar:1.1 from/to central (https://repo.maven.apache.org/maven2): Failed to transfer file: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-io/1.1/maven-shared-io-1.1.jar. Return code is: 503 , ReasonPhrase:Maximum threads for service reached. -> [Help 1] 23:29:56 ] [ERROR] 23:29:56 ] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 23:29:56 ] [ERROR] Re-run Maven using the -X switch to enable full debug logging. 23:29:56 ] [ERROR] 23:29:56 ] [ERROR] For more information about the errors and possible solutions, please read the following articles: 23:29:56 ] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException I'm going to retry. -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 01:14:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/13663/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java: http://gerrit.cloudera.org:8080/#/c/13663/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@79 PS3, Line 79: // since RangerAuthorizationChecker is stateless and creating a new instance of > do we need to reinit the ranger plugin for explicit REFRESH AUTHORIZATION c No, we can use the existing plugin instance and call a refreshTagsAndPolicy API instead of re-initing. http://gerrit.cloudera.org:8080/#/c/13663/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@81 PS3, Line 81: // Ranger plugin. > this isn't safe under concurrency unless authzChecker is volatile. Should w Good idea. Done. -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 20:22:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update This patch fixes the issue where RangerAuthorizationChecker instance gets created on every catalog update, which can impact the performance due to the need to re-initialize the Ranger plugin. Unlike SentryAuthorizationChecker which is stateful (it uses AuthorizationPolicy), RangerAuthorizationChecker is stateless and we can reuse the same instance of it. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/13663/4 -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13663/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java: http://gerrit.cloudera.org:8080/#/c/13663/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@39 PS2, Line 39: authzChecker_ > where does this ever get set? Ah my bad, I pushed the too quick :( I should've checked first before publishing. Sorry about that. -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 20:12:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update This patch fixes the issue where RangerAuthorizationChecker instance gets created on every catalog update, which can impact the performance due to the need to re-initialize the Ranger plugin. Unlike SentryAuthorizationChecker which is stateful (it uses AuthorizationPolicy), RangerAuthorizationChecker is stateless and we can reuse the same instance of it. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/13663/3 -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13663 Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update This patch fixes the issue where RangerAuthorizationChecker instance gets created on every catalog update, which can impact the performance due to the need to re-initialize the Ranger plugin. Unlike SentryAuthorizationChecker which is stateful (it uses AuthorizationPolicy), RangerAuthorizationChecker is stateless and we can reuse the same instance of it. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/13663/2 -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 7: (2 comments) You need to rebase again to resolve the merge conflict. http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@232 PS7, Line 232: nit: overly indented, use 4 spaces for continued indentation. http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/util/EventSequence.java File fe/src/main/java/org/apache/impala/util/EventSequence.java: http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/util/EventSequence.java@44 PS7, Line 44: duration nit: duration in nano seconds. -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 17 Jun 2019 19:59:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java@43 PS5, Line 43:* @param sqlStmt the SQL statement to be logged for auditing > nit: add a @param for sessionState? Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java@59 PS5, Line 59: sqlStmt_ = Preconditions.checkNotNull(sqlStmt); > should you Preconditions.checkNotNull() for these? or Objects.firstNonNull( Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@367 PS5, Line 367: ddlRequest.getHeader(), ddlRequest.g > it seems like 'requestingUser' always is being constructed out of ddlReques Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@367 PS5, Line 367: ddlRequest.getHeader(), ddlRequest.g > it seems like 'requestingUser' always is being constructed out of ddlReques Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java File fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java@44 PS5, Line 44: InetAddresses.forString(session.getNetwork_address().getHostname()) : .getHostAddress(); > hrm, now I'm wondering why we can't just use getNetowkr_address().getHostna Ranger wants it into the IP address form. It maybe safe to provide Ranger what it needs in case it needs to do a reverse lookup or something. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 19:01:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 218 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/6 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13653 ) Change subject: IMPALA-8599: Create a Maven module for query event hook API .. Patch Set 2: > Patch Set 2: Code-Review+2 > > (1 comment) > > Looks fine. We should probably reorganize all this stuff so that the > different modules are submodules within a top-level directory with a pom to > tie them together (multi-module maven project) instead of depending on each > other as separate projects tied together by cmake, but I guess that's out of > scope for this patch. Any interest in taking that on? I agree, the existing project structure isn't the usual Maven project structure. Sure, I can take on fixing it. -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 18:03:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13653 ) Change subject: IMPALA-8599: Create a Maven module for query event hook API .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13653/2/query-event-hook-api/CMakeLists.txt File query-event-hook-api/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/13653/2/query-event-hook-api/CMakeLists.txt@18 PS2, Line 18: DEPENDS gen-deps impala-parent > qq: why is this needed? Looking at the hook, seems like a standalone thing. It depends on impala-parent because this module uses impala-parent. As for gen-deps, it's currently not needed, but if we need to use anything from Thrift in the future, then we'll need gen-deps. -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 05:04:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13653 Change subject: IMPALA-8599: Create a Maven module for query event hook API .. IMPALA-8599: Create a Maven module for query event hook API This patch moves the query event event hook API into a separate Maven module so that it can be easily consumed by other projects, such as Atlas. Testing: - Ran make fe Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c --- M CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M impala-parent/pom.xml A query-event-hook-api/CMakeLists.txt A query-event-hook-api/pom.xml R query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryCompleteContext.java R query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java 8 files changed, 92 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/13653/2 -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] Remove "Could not transfer" exclusion in mvn-quiet.sh
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13647 ) Change subject: Remove "Could not transfer" exclusion in mvn-quiet.sh .. Patch Set 1: > Patch Set 1: > > We could also consider removing the whole error suppression thing. I added it > a long time ago when I was trying to make Jenkins logs less noisy but I'm now > not sure that it was a good idea. The current mvn-quiet.sh doesn't suppress error though. So, I think we're good here. -- To view, visit http://gerrit.cloudera.org:8080/13647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide3367fd98abbbe11eec1fa86fbad8b32eeecb8d Gerrit-Change-Number: 13647 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Jun 2019 16:25:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove "Could not transfer" exclusion in mvn-quiet.sh
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13647 Change subject: Remove "Could not transfer" exclusion in mvn-quiet.sh .. Remove "Could not transfer" exclusion in mvn-quiet.sh "Could not transfer" warning messages are noisy. However, excluding "Could not transfer" words can lead to actual error messages that contain "Could not transfer" to not be shown in the stdout, which can make debugging difficult. This patch updates mvn-quiet.sh to show "Could not transfer" messages. Testing: - Ran FE build Change-Id: Ide3367fd98abbbe11eec1fa86fbad8b32eeecb8d --- M bin/mvn-quiet.sh 1 file changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/13647/1 -- To view, visit http://gerrit.cloudera.org:8080/13647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ide3367fd98abbbe11eec1fa86fbad8b32eeecb8d Gerrit-Change-Number: 13647 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java File fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java: http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java@47 PS3, Line 47: } > I guess I'm wondering: would this ever _not_ be a dotted-decimal address? I Good idea. Done. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 14 Jun 2019 01:48:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 211 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/5 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 217 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/4 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13601 )
Change subject: IMPALA-8658: Populate missing Ranger audit fields
..
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/13601/3/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/13601/3/common/thrift/JniCatalog.thrift@642
PS3, Line 642: // Parameters of DROP DATABASE commands
> Can you name this 'redacted_sql_stmt' or something so that it's clear this
Done
http://gerrit.cloudera.org:8080/#/c/13601/3/common/thrift/JniCatalog.thrift@640
PS3, Line 640: }
:
: // Parameters of DROP DATABASE commands
: struct TDropDbParams {
: // Name of the database to drop
: 1: required string db
> What do you think about moving these to TCatalogServiceRequestHeader? They
Done
http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java
File fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java:
http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java@111
PS3, Line 111:
:
> seems weird that this can throw. Is it trying to reverse-lookup or somethin
Yeah it's just Java checked exception. I also don't know a case where it'll
throw an exception. I'll change it to throw an unchecked exception in
getClientIpAddress instead.
http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@222
PS3, Line 222: }
: }
: retur
> similar to elsewhere, it seems like this should never happen if we just hav
Done
http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java
File fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java:
http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java@47
PS3, Line 47: return
InetAddress.getByName(session.getNetwork_address().getHostname())
> We could avoid the 'throws UnknownHostException' by just using String.forma
Ranger wants it to be an IP address, although technically it's just a string
and I'm sure it'll accept anything.
--
To view, visit http://gerrit.cloudera.org:8080/13601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8
Gerrit-Change-Number: 13601
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Thu, 13 Jun 2019 23:44:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java: http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java@117 PS2, Line 117: String resourceType = resource != null ? resource.getLeafName() : null; > this seems like the impala IP, not the client's IP. I'm not sure exactly wh Ah Impala client IP. I thought it was Ranger client IP, which is an Impala IP :) Done. http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/test/resources/ranger-hive-audit.xml File fe/src/test/resources/ranger-hive-audit.xml: http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/test/resources/ranger-hive-audit.xml@27 PS2, Line 27: ranger.plugin.hive.ambari.cluster.name > if this parameter isn't set by the end user, do we risk an uninterpretable I don't think this field is strictly required by the Ranger audit. It seems to work fine without it. It is okay to pass null when cluster name isn't available. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 13 Jun 2019 19:56:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 16 files changed, 125 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/3 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR](2.x) Ignore IMPALA-7316, IMPALA-7212 and IMPALA-5607
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13609 ) Change subject: Ignore IMPALA-7316, IMPALA-7212 and IMPALA-5607 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13609 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I5465fed79b8cb5969a50eceedfb9d147a2a12811 Gerrit-Change-Number: 13609 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 13 Jun 2019 16:30:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13601 Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/resources/ranger-hive-audit.xml 4 files changed, 26 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/2 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8654: Log the SQL statement in the Ranger audit log
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13590 ) Change subject: IMPALA-8654: Log the SQL statement in the Ranger audit log .. IMPALA-8654: Log the SQL statement in the Ranger audit log This patch logs the SQL statement to be authorized in the Ranger audit log since it was a required field prior to the fix in RANGER-2463 to avoid a NullPointerException in the Ranger admin that could cause the Ranger audit logs to not show up in the Ranger web UI. Testing: - Updated the RangerAuditLogTest - Tested the changes against Solr cluster - Ran all FE tests Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 10 files changed, 71 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13590/4 -- To view, visit http://gerrit.cloudera.org:8080/13590 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 Gerrit-Change-Number: 13590 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13353 )
Change subject: IMPALA-8443: Record time spent in authorization in the runtime
profile
..
Patch Set 3:
> Patch Set 3: Patch Set 2 was rebased
Hi Tamas, I saw that you just rebased the patch. Are you going to make a change
to make sure the authorization time logged here:
https://github.com/apache/impala/blob/ab908d54c22861967f693428ec7d9f6d7008607f/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java#L89
is the same as the one in timeline.markEvent("Authorization finished")?
--
To view, visit http://gerrit.cloudera.org:8080/13353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae
Gerrit-Change-Number: 13353
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Comment-Date: Wed, 12 Jun 2019 01:23:36 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8654: Log the SQL statement in the Ranger audit log
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13590 Change subject: IMPALA-8654: Log the SQL statement in the Ranger audit log .. IMPALA-8654: Log the SQL statement in the Ranger audit log This patch logs the SQL statement to be authorized in the Ranger audit log since it was a required field prior to the fix in RANGER-2463 to avoid a NullPointerException in the Ranger admin that could cause the Ranger audit logs to not show up in the Ranger web UI. Logging the SQL statement requires proper implementation of StatementBase::toSql() which unfortunately is not available in all SQL statements. See IMPALA-8655. Testing: - Updated the RangerAuditLogTest - Tested the changes against Solr cluster Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 10 files changed, 84 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13590/1 -- To view, visit http://gerrit.cloudera.org:8080/13590 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 Gerrit-Change-Number: 13590 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13586 ) Change subject: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/13586/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13586/1//COMMIT_MSG@14 PS1, Line 14: timedout typo: timed out -- To view, visit http://gerrit.cloudera.org:8080/13586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7 Gerrit-Change-Number: 13586 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Jun 2019 22:26:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13508/1/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/1/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@124 PS1, Line 124: analyzer.addAccessEvent(new TAccessEvent( > Should we log the exception? We should have an E2E test here by having a table that is dropped externally. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 11 Jun 2019 18:30:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 8: Code-Review+2 Rebased. Carry Todd's +2. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 11 Jun 2019 01:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/8 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py@572 PS6, Line 572: assert "Error granting a privilege in Ranger. Ranger error message: " \ > don't these asserts need to be updated? Sorry, I forgot to commit this change. Done. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:25:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/7 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/6 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13525 )
Change subject: IMPALA-8551: Make the grant/revoke error messages to be more
user friendly
..
Patch Set 6:
(2 comments)
> Patch Set 5:
>
> (2 comments)
>
> Looks pretty good modulo some nits. One thing, though: would it be possible
> to split this into one commit to bump the CDP_BUILD_NUMBER and one to change
> the error messages? I'm afraid when we look through lists of commits to
> backport across branches, we'll see this and think "oh it's just a CDP
> version bump, we can skip this one and pick up a later one" whereas in fact
> it also contains a functional change.
Split into two now.
https://gerrit.cloudera.org/c/13574/
https://gerrit.cloudera.org/c/13525/
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@173
PS5, Line 173: try (AutoFlush auditHandler =
RangerBufferAuditHandler.autoFlush()) {
> pre-existing issue, but: is it possible to add a DEBUG level log message he
Wouldn't be better to use ERROR log level instead?
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@174
PS5, Line 174: plugin_.get().grantAccess(request, auditHandler);
> nit: can we reword this to "in Ranger" instead of "to Ranger" -- the privil
Done
--
To view, visit http://gerrit.cloudera.org:8080/13525
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576
Gerrit-Change-Number: 13525
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 10 Jun 2019 16:17:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 1: Splitting the https://gerrit.cloudera.org/c/13525/ into two commits. -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 16:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13574 Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Bump CDP_BUILD_NUMBER to 1153860 This patch bumps the CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b --- M bin/impala-config.sh 1 file changed, 5 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13574/1 -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/4/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/4/tests/authorization/test_ranger.py@587 PS4, Line 587: H > flake8: E126 continuation line over-indented for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 07 Jun 2019 21:18:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 This patch bumps CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Added E2E test to test grant/revoke with invalid principal - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 3 files changed, 63 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/5 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 This patch bumps CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Added E2E test to test grant/revoke with invalid principal - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 3 files changed, 63 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/4 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py@578 PS2, Line 578: ent in ["grant > hrm. I agree that parsing the error message isn't super robust, but I think I can easily prefix this with "Error granting a privilege to Ranger. Ranger error message: HTTP 403: ...". This way we don't have to worry if Ranger changes their error message and it's pretty clear that the error message came from Ranger. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 07 Jun 2019 21:16:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8638: Fix flaky TestLineage::test create table timestamp
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13560 ) Change subject: IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp .. Patch Set 3: Code-Review+2 (1 comment) Carry Lars' +2. http://gerrit.cloudera.org:8080/#/c/13560/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13560/2//COMMIT_MSG@12 PS2, Line 12: to u > missing word Done -- To view, visit http://gerrit.cloudera.org:8080/13560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 Gerrit-Change-Number: 13560 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Fri, 07 Jun 2019 17:48:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8638: Fix flaky TestLineage::test create table timestamp
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13560 ) Change subject: IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp .. IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp This patch fixes the bug TestLineage::test_create_table_timestamp where it uses the same lineage log directory created by TestLineage::test_start_end_timestamp, which could fail the query ID assertion. The fix is to use a different lineage log directory than the one used by TestLineage::test_start_end_timestamp. Testing: - Ran test_lineage.py multiple times and it still passed. Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 --- M tests/custom_cluster/test_lineage.py 1 file changed, 11 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13560/3 -- To view, visit http://gerrit.cloudera.org:8080/13560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 Gerrit-Change-Number: 13560 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8638: Fix flaky TestLineage::test create table timestamp
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13560 Change subject: IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp .. IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp This patch fixes the bug TestLineage::test_create_table_timestamp where it uses the same lineage log directory created by TestLineage::test_start_end_timestamp, which could fail the query ID assertion. The fix is to a different lineage log directory than the one used by TestLineage::test_start_end_timestamp. Testing: - Ran test_lineage.py multiple times and it still passed. Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 --- M tests/custom_cluster/test_lineage.py 1 file changed, 11 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13560/2 -- To view, visit http://gerrit.cloudera.org:8080/13560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 Gerrit-Change-Number: 13560 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13525 )
Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py@566
PS2, Line 566: ctional.alltypes from user {0}"
: .format(getuser())]:
: result =
self.execute_query_expect_failure(invalid_impala_cli
> it's weird that you need two different clients, but you're passing the user
This is a known issue. I filed a bug here:
https://issues.apache.org/jira/browse/IMPALA-8640
http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py@578
PS2, Line 578:
> is there any way we can get rid of the HTTP error part of the error message
We can but we have to parse the error message and that can be somewhat error
prone. Ranger throws a generic exception, so we pretty much just rely on the
error message returned by them.
--
To view, visit http://gerrit.cloudera.org:8080/13525
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576
Gerrit-Change-Number: 13525
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Fri, 07 Jun 2019 16:30:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 This patch bumps CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Added E2E test to test grant/revoke with invalid principal - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M bin/impala-config.sh M tests/authorization/test_ranger.py 2 files changed, 44 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/3 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 This patch bumps CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Added E2E test to test grant/revoke with invalid principal - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M bin/impala-config.sh M tests/authorization/test_ranger.py 2 files changed, 42 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/2 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 1: Running dry-run first to make sure there's no regression. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 06 Jun 2019 01:00:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13525 Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 This patch bumps CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Added E2E test to test grant/revoke with invalid principal - Ran core tests - Ran USE_CDP_HIVE=true ./buildall.sh -format -testdata -notests Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M bin/impala-config.sh M tests/authorization/test_ranger.py 2 files changed, 42 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/1 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
