[
https://issues.apache.org/jira/browse/IMPALA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682930#comment-17682930
]
ASF subversion and git services commented on IMPALA-11845:
----------------------------------------------------------
Commit 0c1bd9eff37b9a2df2849e6bc8edc8035bb8aefe in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0c1bd9eff ]
IMPALA-11845: Fix incorrect check of struct STAR path in resolvePathWithMasking
resolvePathWithMasking() is a wrapper on resolvePath() to further
resolve nested columns inside the table masking view. When it was
added, complex types in the select list hadn't been supported yet. So
the table masking view can't expose complex type columns directly in the
select list. Any paths in nested types will be further resolved inside
the table masking view in resolvePathWithMasking().
Take the following query as an example:
select id, nested_struct.* from complextypestbl;
If Ranger column-masking/row-filter policies applied on the table, the
query is rewritten as
select id, nested_struct.* from (
select mask(id) from complextypestbl
where row-filtering-condition
) t;
Table masking view "t" can't expose the nested column "nested_struct".
So we further resolve "nested_struct" inside the inlineView to use the
masked table "complextypestbl". The underlying TableRef is expected to
be a BaseTableRef.
Paths that don't reference nested columns should be resolved and
returned directly (just like the original resolvePath() does). E.g.
select v.* from masked_view v
is rewritten to
select v.* from (
select mask(c1), mask(c2), ..., mask(cn)
from masked_view
where row-filtering-condition
) v;
The STAR path "v.*" should be resolved directly. However, it's treated
as a nested column unexpectedly. The code then tries to resolve it
inside the table "masked_view" and found "masked_view" is not a table so
throws the IllegalStateException.
These are the current conditions for identifying nested STAR paths:
- The destType is STRUCT
- And the resolved path is rooted at a valid tuple descriptor
They don't really recognize the nested struct columns because STAR paths
on table/view also match these conditions. When the STAR path is an
expansion on a catalog table/view, the root tuple descriptor is
exactly the output tuple of the table/view. The destType is the type of
the tuple descriptor which is always a StructType.
Note that STAR paths on other nested types, i.e. array/map, are invalid.
So the first condition matches for all valid cases. The second condition
also matches all valid cases since both the table/view and struct STAR
expansion have the path rooted at a valid tuple descriptor.
This patch fixes the check for nested struct STAR path by checking
the matched types instead. Note that if "v.*" is a table/view expansion,
the matched type list is empty. If "v.*" is a struct column expansion,
the matched type list contains the STRUCT column type.
Tests:
- Add missing coverage on STAR paths (v.*) on masked views.
Change-Id: I8f1e78e325baafbe23101909d47e82bf140a2d77
Reviewed-on: http://gerrit.cloudera.org:8080/19429
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Select STAR with table ref is not resolved correctly on column/row masked
> views
> -------------------------------------------------------------------------------
>
> Key: IMPALA-11845
> URL: https://issues.apache.org/jira/browse/IMPALA-11845
> Project: IMPALA
> Issue Type: Bug
> Components: Security
> Affects Versions: Impala 4.0.0, Impala 3.4.0, Impala 3.4.1, Impala 4.1.0,
> Impala 4.2.0, Impala 4.1.1
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> When a catalog view is applied with Ranger column-masking or row-filtering
> policies, select STAR on it doesn't work if the table ref is used, e.g. this
> doesn't work:
> {code:sql}
> select v.* from masked_view v;
> ERROR: IllegalStateException: null
> {code}
> Note that selecting STAR directly or selecting any columns still work:
> {code:sql}
> select * from masked_view;
> select c1, c2 from masked_view;
> {code}
> The stacktrace:
> {noformat}
> I0117 10:50:36.894714 21116 jni-util.cc:288]
> f940f87cdea03526:3c48d0f300000000] java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:486)
> at
> org.apache.impala.analysis.Analyzer.resolvePathWithMasking(Analyzer.java:1216)
> at
> org.apache.impala.analysis.Analyzer.resolvePathWithMasking(Analyzer.java:1177)
> at
> org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyzeStarPath(SelectStmt.java:785)
> at
> org.apache.impala.analysis.SelectStmt$SelectAnalyzer.collectStarExpandedPaths(SelectStmt.java:879)
> at
> org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyze(SelectStmt.java:333)
> at
> org.apache.impala.analysis.SelectStmt$SelectAnalyzer.access$100(SelectStmt.java:280)
> at org.apache.impala.analysis.SelectStmt.analyze(SelectStmt.java:272)
> at
> org.apache.impala.analysis.AnalysisContext.reAnalyze(AnalysisContext.java:622)
> at
> org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:553)
> at
> org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:468)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2059)
> at
> org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1967)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1789)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
> {noformat}
> Related code:
> {code:java}
> 1180 public Path resolvePathWithMasking(List<String> rawPath, PathType
> pathType,
> 1181 TimeTravelSpec timeTravelSpec) throws AnalysisException,
> TableLoadingException {
> 1182 Path resolvedPath = resolvePath(rawPath, pathType, timeTravelSpec);
> 1183 // Skip normal resolution cases that don't relate to nested types.
> 1184 if (pathType == PathType.TABLE_REF) {
> 1185 if (resolvedPath.destTable() != null ||
> !resolvedPath.isRootedAtTuple()) {
> 1186 return resolvedPath;
> 1187 }
> 1188 } else if (pathType == PathType.SLOT_REF) {
> 1189 if (!resolvedPath.getMatchedTypes().get(0).isStructType()) {
> 1190 return resolvedPath;
> 1191 }
> 1192 } else if (pathType == PathType.STAR) {
> 1193 if (!resolvedPath.destType().isStructType() ||
> !resolvedPath.isRootedAtTuple()) {
> 1194 return resolvedPath;
> 1195 }
> 1196 }
> 1197 // In this case, resolvedPath is resolved on a nested column. Check
> if it's resolved
> 1198 // on a table masking view. The root TableRef(table/view) could be
> at a parent query
> 1199 // block (correlated case, e.g. "t.int_array" in query
> 1200 // "SELECT ... FROM tbl t, (SELECT * FROM t.int_array) a" roots at
> "tbl t" which is
> 1201 // in the parent block), so we should find the parent block first
> then we can find
> 1202 // the root TableRef.
> 1203 TupleId rootTupleId = resolvedPath.getRootDesc().getId();
> 1204 Analyzer parentAnalyzer = findAnalyzer(rootTupleId);
> 1205 TableRef rootTblRef = parentAnalyzer.getTableRef(rootTupleId);
> 1206 Preconditions.checkNotNull(rootTblRef);
> 1207 if (!rootTblRef.isTableMaskingView()) return resolvedPath;
> 1208 // resolvedPath is resolved on a nested column of a table masking
> view. The view
> 1209 // won't produce results of nested columns. It just exposes the
> nested columns of the
> 1210 // underlying BaseTableRef in the fields of its output type. (See
> more in
> 1211 // InlineViewRef#createTupleDescriptor()). We need to resolve
> 'rawPath' inside the
> 1212 // view as if the underlying table is not masked. So the resolved
> path can point to
> 1213 // the real table and be used to create materialized slot in the
> TupleDescriptor of
> 1214 // the real table.
> 1215 InlineViewRef tableMaskingView = (InlineViewRef) rootTblRef;
> 1216 Preconditions.checkState(
> 1217 tableMaskingView.getUnMaskedTableRef() instanceof BaseTableRef);
> 1218 // Resolve rawPath inside the table masking view to point to the
> real table.
> 1219 Path maskedPath = tableMaskingView.inlineViewAnalyzer_.resolvePath(
> 1220 rawPath, pathType, timeTravelSpec);
> 1221 maskedPath.setPathBeforeMasking(resolvedPath);
> 1222 return maskedPath;
> 1223 }
> {code}
> https://github.com/apache/impala/blob/b0009db40b7a532694987a0f4280b322d72f07b7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java#L1217
> The 'pathType' is STAR and the 'rawPath' is the view ref. It's not a nested
> struct column so should be handled in the if-branches at the begining.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
