[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. Wrapped setIsStraightJoin() in else clause. Testing: Added a wrapper for testing the state of straight_join hint. Modified existing tests to use this wrapper for +ve and -ve test cases. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 26 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/9 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 9: Fixed formatting. -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 04 Oct 2018 20:29:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. Wrapped setIsStraightJoin() in else clause. Testing: Added a wrapper for testing the state of straight_join hint. Modified existing tests to use this wrapper for +ve and -ve test cases. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 28 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/8 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7484: Unrecognized hints are interpreted as straight join
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Unrecognized hints are interpreted as straight_join .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11568/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java: http://gerrit.cloudera.org:8080/#/c/11568/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1849 PS4, Line 1849: assertTrue(ctx.getAnalyzer().isStraightJoin()); > An improvement to this collection of tests would be to wrap the pattern of There are three things that need to be tested. When hint is unrecognized: 1) Appropriate warning should be added. 2) straight_join must not be set. When hint is straight_join: 3) Assert straight_join is set. I think wrapping these three things in a method is a good idea like you said. -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 03 Oct 2018 17:26:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7484: setIsStraightJoin is wrapped in an else clause.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11568 Change subject: IMPALA-7484: setIsStraightJoin is wrapped in an else clause. .. IMPALA-7484: setIsStraightJoin is wrapped in an else clause. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/1 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-7484: setIsStraightJoin is wrapped in an else clause.
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: setIsStraightJoin is wrapped in an else clause. .. IMPALA-7484: setIsStraightJoin is wrapped in an else clause. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 9 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/2 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7484: Unrecognized hints are interpreted as straight join
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Unrecognized hints are interpreted as straight_join .. IMPALA-7484: Unrecognized hints are interpreted as straight_join Call to setIsStraightJoin() is outside else clause in SelectList.java causing even unrecognized hints to be interpreted as straight_joins. Moved it into an else. Now it will be set only if the hint is a straight_join. Testing: Added two test cases to TestSelectListHints. 1) To assert straight_join is not set when hint is unrecognized and 2) To assert it is properly set when hint is indeed a straigh_join. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 13 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/4 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7484: Unrecognized hints are interpreted as straight join
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Unrecognized hints are interpreted as straight_join .. Patch Set 4: (5 comments) Thanks Vuk, submitted new patch. http://gerrit.cloudera.org:8080/#/c/11568/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11568/3//COMMIT_MSG@7 PS3, Line 7: IMPALA-7484: Unrecognized hints are interpreted as straight_join > add a newline Done http://gerrit.cloudera.org:8080/#/c/11568/3//COMMIT_MSG@8 PS3, Line 8: > wrap this appropriately. Done http://gerrit.cloudera.org:8080/#/c/11568/3//COMMIT_MSG@10 PS3, Line 10: causing even unrecognized hints to be interpreted as straight_joins. > same here, please wrap the lines. Done http://gerrit.cloudera.org:8080/#/c/11568/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java: http://gerrit.cloudera.org:8080/#/c/11568/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1845 PS3, Line 1845: // If hint is straight join > add a space after '//' Done http://gerrit.cloudera.org:8080/#/c/11568/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1851 PS3, Line 1851: // Unrecognized hint > same here, add a space. Done -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 03 Oct 2018 07:20:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7484: Unrecognized hints are interpreted as straight join Call to setIsStraightJoin() is outside else clause in SelectList.java causing even unrecognized hints to be interpreted
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Unrecognized hints are interpreted as straight_join Call to setIsStraightJoin() is outside else clause in SelectList.java causing even unrecognized hints to be interpreted as straight_joins. Moved it into an else. Now it will be called only i .. Patch Set 3: (5 comments) Thanks Fredy for the inputs. Uploaded new patch set. Please take a look. http://gerrit.cloudera.org:8080/#/c/11568/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11568/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-7484: Unrecognized hints are interpreted as straight_join > Our convention for the commit message is: Done http://gerrit.cloudera.org:8080/#/c/11568/2/fe/src/main/java/org/apache/impala/analysis/SelectList.java File fe/src/main/java/org/apache/impala/analysis/SelectList.java: http://gerrit.cloudera.org:8080/#/c/11568/2/fe/src/main/java/org/apache/impala/analysis/SelectList.java@89 PS2, Line 89: analyzer.setIsStraightJoin(); > nit: move L89 to L88 Done http://gerrit.cloudera.org:8080/#/c/11568/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java: http://gerrit.cloudera.org:8080/#/c/11568/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1844 PS2, Line 1844: ctx = createAnaly > nit: rename to ctx Done http://gerrit.cloudera.org:8080/#/c/11568/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1847 PS2, Line 1847: "select %sstraight_join%s * from functional.alltypes", prefix, suffix), : ctx); > nit: fix indentation Done http://gerrit.cloudera.org:8080/#/c/11568/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1849 PS2, Line 1849: ssertTrue(ctx.getAnalyzer().isStraightJoin()); > We also need another test case where assertFalse(ctx.getAnalyzer().isStraig Done -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Oct 2018 04:10:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7484: Unrecognized hints are interpreted as straight join Call to setIsStraightJoin() is outside else clause in SelectList.java causing even unrecognized hints to be interpreted
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Unrecognized hints are interpreted as straight_join Call to setIsStraightJoin() is outside else clause in SelectList.java causing even unrecognized hints to be interpreted as straight_joins. Moved it into an else. Now it will be called only i .. IMPALA-7484: Unrecognized hints are interpreted as straight_join Call to setIsStraightJoin() is outside else clause in SelectList.java causing even unrecognized hints to be interpreted as straight_joins. Moved it into an else. Now it will be called only if the hintis a straight_join. Testing: Added two test cases to TestSelectListHints. 1) To make assert straight_join is not set when hint is unrecognized and 2) To assert it is properly set when hint is indeed a straigh_join. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 13 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/3 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. Wrapped setIsStraightJoin() in else clause. Testing: Added tests for +ve and -ve test cases. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 21 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/5 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. Wrapped setIsStraightJoin() in else clause. Testing: Added a wrapper for testing the state of straight_join hint. Modified existing tests to use this wrapper for +ve and -ve test cases. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 27 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/6 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/11568 )
Change subject: IMPALA-7484: Do not interpret unrecognized hints as
straight_join hints.
..
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/11568/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11568/4//COMMIT_MSG@7
PS4, Line 7: Do not interpret unrecognized hints as
> We typically describe the fix rather than the problem, something like
Done
http://gerrit.cloudera.org:8080/#/c/11568/4//COMMIT_MSG@9
PS4, Line 9:
> Remove. No need to describe the commit. Better to keep the commit message h
Done
http://gerrit.cloudera.org:8080/#/c/11568/4//COMMIT_MSG@14
PS4, Line 14: Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04
:
:
> Same as above. Maybe just say something like, added unit tests for positive
Done
http://gerrit.cloudera.org:8080/#/c/11568/4/fe/src/main/java/org/apache/impala/analysis/SelectList.java
File fe/src/main/java/org/apache/impala/analysis/SelectList.java:
http://gerrit.cloudera.org:8080/#/c/11568/4/fe/src/main/java/org/apache/impala/analysis/SelectList.java@86
PS4, Line 86: if (hint.is("straight_join")) {
: analyzer.setIsStraightJoin();
: } else {
: analyzer.addWarning("PLAN hint not recognized: " + hint);
: }
> nit: How about inverting it? I think that is more readable.
Done
http://gerrit.cloudera.org:8080/#/c/11568/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:
http://gerrit.cloudera.org:8080/#/c/11568/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1849
PS4, Line 1849:
> An improvement to this collection of tests would be to wrap the pattern of
Done
--
To view, visit http://gerrit.cloudera.org:8080/11568
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04
Gerrit-Change-Number: 11568
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 03 Oct 2018 23:01:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 6: (5 comments) Changed existing tests to use the wrapper to test the state of straight_join hint for each test. http://gerrit.cloudera.org:8080/#/c/11568/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java: http://gerrit.cloudera.org:8080/#/c/11568/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1830 PS5, Line 1830: An > I was expecting all of these cases to use the wrapper. We'll get more cover Done http://gerrit.cloudera.org:8080/#/c/11568/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1852 PS5, Line 1852: null)); > pls reword this so its clear that this does not test the hint (just returns Done http://gerrit.cloudera.org:8080/#/c/11568/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1855 PS5, Line 1855: %sstr > replace "check" with "has" Done http://gerrit.cloudera.org:8080/#/c/11568/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1855 PS5, Line 1855: "select %sstraight_join,straight_join%s * from functional.alltypes", > put the helper before its use, e.g., on L1824 Done http://gerrit.cloudera.org:8080/#/c/11568/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1858 PS5, Line 1858: > what's this comment for? Done -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 03 Oct 2018 23:42:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12889 )
Change subject: IMPALA-7971: Add support for insert events in event processor.
..
Patch Set 5:
(24 comments)
Thanks for the comments Vihang. I cleaned up code in CatalogOpExecutor. Still
figuring out a way to write tests to verify if tables are refreshed because of
insert events.
http://gerrit.cloudera.org:8080/#/c/12889/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12889/4//COMMIT_MSG@24
PS4, Line 24: Existing self-events logic cannot be used for insert events since
:firing insert event does not allow us to modify
> I think it is more appropriate to say existing self-events logic cannot be
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/12889/4/be/src/service/client-request-state.cc@1106
PS4, Line 1106: // is_overwrite is used to know the type of insert in FE.
> add a comment here explaining why this is needed.
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@839
PS4, Line 839: HashSet<>(f
> would be appropriate to intialize the set with the capacity fdList.size() s
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@842
PS4, Line 842:
> suggest you to use Path.SEPARATOR instead of "/"
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@159
PS4, Line 159: refresh
> refresh on a table/partition
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@587
PS4, Line 587: public static class InsertEvent extends MetastoreTableEvent {
> IIUC, the reason you are extending TableInvalidatingEvent is because you wa
You are right. Changing it back to old TableInvalidatingEvent.
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@588
PS4, Line 588:
> nit, add new line above the constructor. Add a javadoc
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@606
PS4, Line 606: }
> add a // TODO : to handle self-events for insert case
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@613
PS4, Line 613:
> nit, just saying refresh is good enough. No need to say reload here.
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@616
PS4, Line 616:
> same as above. Just say refresh since I don't think reload means anything e
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@623
PS4, Line 623: }
> Add a // TODO : One way to do this would be to change hive source code to r
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@965
PS4, Line 965: t object parameters used for self-
> Why do we need to rename? Currently, all the implementations of this sub-cl
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@118
PS4, Line 118:
> Ignore to keep it consistent with other entries of this table
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@118
PS4, Line 118: |||
: * | INSERT EVENT| Refres
> Just use Refresh unless Reload means something else.
Done
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/12889/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3554
PS4, Line 3554: FeCatalogUtils.loadAllPartitions((HdfsTable) table);
: // Map of partition ids to file names of all existing
partitions touched by the
: // insert
>
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. The event-processor will refresh tables using this event. Both insert into and overwrite are supported for tables/partitions. Also, renamed TableInvalidatingEvent class to TableInvalidatingOrRefreshingEvent to reflect new behaviour. Known Issues: 1. There is an unnecessary table invalidate when insert is done in Hive as the insert operation creates an ALTER and an INSERT notification event. Currently there is no way for the Event Processor to identify and prevent the unnecessary invalidate. IMPALA-7973 may potentially solve this issue. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/custom_cluster/test_event_processing.py 7 files changed, 282 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/5 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 1: Code-Review+1 (2 comments) Apart from nits, LGTM. Will let others take a look. http://gerrit.cloudera.org:8080/#/c/12940/1/common/thrift/CatalogObjects.thrift File common/thrift/CatalogObjects.thrift: http://gerrit.cloudera.org:8080/#/c/12940/1/common/thrift/CatalogObjects.thrift@474 PS1, Line 474: // Set iff this is a table needs access storage. Maybe say "Set if this table needs storage access" ? http://gerrit.cloudera.org:8080/#/c/12940/1/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/12940/1/tests/query_test/test_observability.py@573 PS1, Line 573: def test_query_profile_storge_load(self): How about having the method signature as test_query_profile_storage_load_time(self) ? -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 05 Apr 2019 21:19:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Hello Bharath Vissapragada, Paul Rogers, Vihang Karajgaonkar, Bharath Krishna, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12889 to look at the new patch set (#2). Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. The event-processor will refresh tables using this event. Both insert into and overwrite are supported for tables/partitions. Also, renamed TableInvalidatingEvent class to TableInvalidatingOrRegreshingEvent to reflect new behaviour. Known Issues: 1. There is an unnecessary table invalidate when insert is done in Hive as the insert operation creates an ALTER and an INSERT notification event. Currently there is no way for the Event Processor to identify and prevent the unnecessary invalidate. IMPALA-7973 may potentially solve this issue. 2. Detection of self-events does not work for inserts currently because of the way the self-event checks are implemented. The flags added to test for self events have no way to persist in HMS with just an insert operation. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/custom_cluster/test_event_processing.py 7 files changed, 301 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/2 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. The event-processor will refresh tables using this event. Both insert into and overwrite are supported for tables/partitions. Also, renamed TableInvalidatingEvent class to TableInvalidatingOrRefreshingEvent to reflect new behaviour. Known Issues: 1. There is an unnecessary table invalidate when insert is done in Hive as the insert operation creates an ALTER and an INSERT notification event. Currently there is no way for the Event Processor to identify and prevent the unnecessary invalidate. IMPALA-7973 may potentially solve this issue. 2. Detection of self-events does not work for inserts currently because of the way the self-event checks are implemented. The flags added to test for self events have no way to persist in HMS with just an insert operation. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: Wrote new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/custom_cluster/test_event_processing.py 7 files changed, 302 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/4 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12889 Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. The event-processor will refresh tables using this event. Both insert into and overwrite are supported for tables/partitions. Also, renamed TableInvalidatingEvent class to TableInvalidatingOrRegreshingEvent to reflect new behaviour. Known Issues: 1. There is an unnecessary table invalidate when insert is done in Hive as the insert operation creates an ALTER and an INSERT notification event. Currently there is no way for the Event Processor to identify and prevent the unnecessary invalidate. IMPALA-7973 may potentially solve this issue. 2. Detection of self-events does not work for inserts currently because of the way the self-event checks are implemented. The flags added to test for self events have no way to persist in HMS with just an insert operation. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: Wrote new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/custom_cluster/test_event_processing.py 7 files changed, 301 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/1 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. The event-processor will refresh tables using this event. Both insert into and overwrite are supported for tables/partitions. Also, renamed TableInvalidatingEvent class to TableInvalidatingOrRefreshingEvent to reflect new behaviour. Known Issues: 1. There is an unnecessary table invalidate when insert is done in Hive as the insert operation creates an ALTER and an INSERT notification event. Currently there is no way for the Event Processor to identify and prevent the unnecessary invalidate. IMPALA-7973 may potentially solve this issue. 2. Detection of self-events does not work for inserts currently because of the way the self-event checks are implemented. The flags added to test for self events have no way to persist in HMS with just an insert operation. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: Wrote new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/custom_cluster/test_event_processing.py 7 files changed, 299 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/3 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After this change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 45 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/5 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(18 comments)
http://gerrit.cloudera.org:8080/#/c/12601/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12601/4//COMMIT_MSG@12
PS4, Line 12: this
> typo
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@327
PS4, Line 327: ex) {
> nit: can be shortened to ex
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@328
PS4, Line 328: // No need to change the EventProcessor state to error
since we want the
> How many times does it retry until it becomes unsuccessful? Will HMS throw
There is no bound on number of retries. It will keep retrying
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java
File
fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java:
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@25
PS4, Line 25:
> nit: extra \n
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@27
PS4, Line 27:
:
:
> I'd rephrase this to something like, events processor that simulates HMS fa
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@30
PS4, Line 30:
> I don't think this warrants a separate class. A private static inner class
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@31
PS4, Line 31:
> indentation off
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@36
PS4, Line 36:
> nit: rename to counter_ and move this before constructor.
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@36
PS4, Line 36:
> class members use _ suffix. Move to the top. Add a comment
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@42
PS4, Line 42:
> This looks weird. How about throw an error randomly 50% of the times?
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@42
PS4, Line 42:
:
:
> This code is also not generic enough. If the goal of this class to be reusa
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@45
PS4, Line 45:
> nit: else isn't necessary after a throw or return.
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@45
PS4, Line 45:
> not needed.
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@528
PS4, Line 528:
> nit: space before {
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@529
PS4, Line 529:
> nit: we usually don't use _ for non-member variables.
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@529
PS4, Line 529: Processor {
> local variables don't need a suffix _
Done
http://gerrit.cloudera.org:8080/#/c/12601/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@536
PS4, Line 536: storeNotificationFetchExcep
> Should we also have a test where EventProessorState becomes ERROR?
For FetchExceptions, it will never become ERROR since the EventProcessor needs
to trying.
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. IMPALA-7935: Disable /catalog_object in local catalog mode. getTCatalogObject() is not supported in local catalog mode since metadata is partially fetched on demand. Removed hyperlinks to the /catalog_object endpoints when local_catalog_mode is enabled. Testing: Added a negetive and a positive test to verify /catalog_mode endpoint is disabled when in local catalog mode and vice-versa. Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c --- M be/src/service/impala-http-handler.cc M tests/custom_cluster/test_local_catalog.py M tests/webserver/test_web_pages.py M www/catalog.tmpl 4 files changed, 53 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12443/5 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12443 )
Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode.
..
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/12443/4/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
http://gerrit.cloudera.org:8080/#/c/12443/4/be/src/service/impala-http-handler.cc@103
PS4, Line 103: if(!FLAGS_use_local_catalog) {
> I think you missed the comment part. Add a quick comment?
Done
http://gerrit.cloudera.org:8080/#/c/12443/4/be/src/service/impala-http-handler.cc@541
PS4, Line 541: Value table_obj(kObjectType);
> Is this needed? The template file uses fqtn only when use_local_catalog is
Yes, fqtn is only needed to call the /catalog_obj endpoint.
http://gerrit.cloudera.org:8080/#/c/12443/4/be/src/service/impala-http-handler.cc@549
PS4, Line 549: table_obj.AddMember("name", table_name,
document->GetAllocator());
: Value use_local_catalog(FLAGS_use_local_catalog);
: table_obj.AddMember("use_loca
> Why this?
This is needed because, mushtashe templates seem to not recognize variables in
parent scope. Hence, it is needed to add the local_catalog variable at both
Databases and Tables level to render two different parts of the template.
--
To view, visit http://gerrit.cloudera.org:8080/12443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c
Gerrit-Change-Number: 12443
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 04 Mar 2019 03:14:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
> I'm sorry, not sure what you mean here. The test class makes sure that exce
I think I see your point here. I will make the change. Thanks for the comments.
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:31:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@534
PS5, Line 534: @Override
> nit: newline
Done
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
> right. But since the exception is thrown at random, it is possible that the
Understood. Thanks for clarifying.
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:37:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After this change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 52 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/6 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After this change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 50 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/8 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
> Looks like this can even pass when no exceptions are thrown. I think we sho
I'm sorry, not sure what you mean here. The test class makes sure that
exceptions are thrown roughly half of the times this method is called..This is
actually an overkill. I want to test the status has not changed to ERROR after
exceptions. This was the behavior before the change - Even a single exception
would change the status to ERROR which would prevent the EventProcessor to run
the next time.
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:20:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@38 PS6, Line 38: import org.apache.hadoop.hive.metastore.api.CurrentNotificationEventId; > nit, unused import. RRest looks good to me. Done http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@38 PS6, Line 38: import org.apache.hadoop.hive.metastore > remove Done http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@542 PS6, Line 542: T > nit: space if ( Done -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 01:18:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12601 Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After the change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 16 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/1 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After thise change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java A fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 65 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/2 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/12601/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/12601/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@328
PS1, Line 328: // No need to change the EventProcessor state to error
since we want the
> Can you make the error log state that the metastore might be unavailable, a
Done
http://gerrit.cloudera.org:8080/#/c/12601/3/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java
File
fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java:
http://gerrit.cloudera.org:8080/#/c/12601/3/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@25
PS3, Line 25:
> java doc explaining what it does would be great
Done
http://gerrit.cloudera.org:8080/#/c/12601/3/fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java@40
PS3, Line 40: throws MetastoreNotificationFetchException {
> format to inline with }
Done
http://gerrit.cloudera.org:8080/#/c/12601/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@531
PS3, Line 531: catalog_, eventsProcessor_.getCurrentEventId(), 2L);
> Since this is used only this method, you don't need to declare the variable
Done
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Tue, 26 Feb 2019 23:54:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After thise change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java A fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 67 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/4 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After thise change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java A fe/src/test/java/org/apache/impala/catalog/events/HMSFetchNotificationsEventProcessorForTests.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 66 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/3 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8163: Add banner showing for local catalog mode when enabled.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12413 Change subject: IMPALA-8163: Add banner showing for local catalog mode when enabled. .. IMPALA-8163: Add banner showing for local catalog mode when enabled. Added a banner on impalad web UI for visibility of local catalog (V2) mode. Testing: Added observability test in test_webpage.py to test if banner appears in V2 mode. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_web_pages.py M www/root.tmpl 3 files changed, 24 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/1 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-8163: Add banner showing for local catalog mode when enabled.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing for local catalog mode when enabled. .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/12413/1/tests/custom_cluster/test_web_pages.py File tests/custom_cluster/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12413/1/tests/custom_cluster/test_web_pages.py@47 PS1, Line 47: def test_observability(self): > Last we discussed, I thought we wanted to add this in test_local_catalog.py test_cache_metrics is confined in TestCompactCatalogUpdates which did not seem like a place for observability tests. Also, test_web_pages.py already does something close to observability test. Is there a performance implication for doing it this way? http://gerrit.cloudera.org:8080/#/c/12413/1/www/root.tmpl File www/root.tmpl: http://gerrit.cloudera.org:8080/#/c/12413/1/www/root.tmpl@30 PS1, Line 30: (V2) > Thinking about this again, probably omit this, since that does not make sen Okay, omitting V2. -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 Feb 2019 21:03:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8163: Add banner showing for local catalog mode when enabled.
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing for local catalog mode when enabled. .. IMPALA-8163: Add banner showing for local catalog mode when enabled. Added a banner on impalad web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when V2 mode is on. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 87 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/2 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8163: Add banner showing for local catalog mode when enabled.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing for local catalog mode when enabled. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12413/1/tests/custom_cluster/test_web_pages.py File tests/custom_cluster/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12413/1/tests/custom_cluster/test_web_pages.py@47 PS1, Line 47: > Looking at it again, I don't think test_cache_metrics has anything to do wi Done -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 09 Feb 2019 00:31:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8163: Add banner showing for local catalog mode when enabled.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing for local catalog mode when enabled. .. IMPALA-8163: Add banner showing for local catalog mode when enabled. Added a banner on impalad web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when V2 mode is on. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 87 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/3 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8163: Add banner showing for local catalog mode when enabled.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing for local catalog mode when enabled. .. IMPALA-8163: Add banner showing for local catalog mode when enabled. Added a banner on impalad web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when V2 mode is on. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 87 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/5 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. IMPALA-7935: Disable /catalog_object in local catalog mode. getTCatalogObject() is not supported in local catalog mode since metadata is partially fetched on demand. Removed hyperlinks to the /catalog_object endpoints when local_catalog_mode is enabled. Testing: Added a test in test_local_catalog to verify /catalog_mode endpoint is disabled. Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c --- M be/src/service/impala-http-handler.cc M tests/custom_cluster/test_local_catalog.py M www/catalog.tmpl 3 files changed, 34 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12443/3 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. IMPALA-7935: Disable /catalog_object in local catalog mode. getTCatalogObject() is not supported in local catalog mode since metadata is partially fetched on demand. Removed hyperlinks to the /catalog_object endpoints when local_catalog_mode is enabled. Testing: 1. Added a positive test in test_local_catalog to verify /catalog_mode endpoint is disabled. 2. Added a negative test to test_web_pages.py to check /catalog_object is working when local_catalog mode is not set. Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c --- M be/src/service/impala-http-handler.cc M tests/custom_cluster/test_local_catalog.py M tests/custom_cluster/test_web_pages.py M www/catalog.tmpl 4 files changed, 62 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12443/4 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object endpoint when local catalog mode is enabled.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12443 Change subject: IMPALA-7935: Disable /catalog_object endpoint when local_catalog_mode is enabled. .. IMPALA-7935: Disable /catalog_object endpoint when local_catalog_mode is enabled. Remove hyperlinks pointing to the /catalog_object endpoints when local_catalog_mode is enabled. Testing: Added a test in test_local_catalog to verify hyperlinks to /catalog_object are removed. Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c --- M be/src/service/impala-http-handler.cc M tests/custom_cluster/test_local_catalog.py M www/catalog.tmpl 3 files changed, 27 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12443/1 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-8163: Add banner showing local catalog mode on web UI when enabled.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. .. IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. Added a banner on coordinator web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when local catalog mode is enabled. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 88 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/6 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8163: Add banner showing local catalog mode on web UI when enabled.
Anurag Mantripragada has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. .. IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. Added a banner on coordinator web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when local catalog mode is enabled. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 72 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/9 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8163: Add banner showing local catalog mode on web UI when enabled.
Anurag Mantripragada has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. .. IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. Added a banner on coordinator web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when local catalog mode is enabled. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 88 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/7 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8163: Add banner showing local catalog mode on web UI when enabled.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. .. IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. Added a banner on coordinator web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when local catalog mode is enabled. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 72 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/8 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8163: Add banner showing local catalog mode on web UI when enabled.
Anurag Mantripragada has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/12413 ) Change subject: IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. .. IMPALA-8163: Add banner showing local catalog mode on web UI when enabled. Added a banner on coordinator web UI for visibility of local catalog mode. Testing: 1. Added a test in test_local_catalog to scrape webpages and search for strings. Tested banner appears when local catalog mode is enabled. 2. Refactored parts of test_local_catalog into TestObservability class. Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 --- M be/src/util/default-path-handlers.cc M tests/custom_cluster/test_local_catalog.py M www/root.tmpl 3 files changed, 72 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/12413/10 -- To view, visit http://gerrit.cloudera.org:8080/12413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207 Gerrit-Change-Number: 12413 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 518 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/6 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 6: (19 comments) http://gerrit.cloudera.org:8080/#/c/12889/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12889/5//COMMIT_MSG@14 PS5, Line 14: partitions. : : Known Issues: > This may not be applicable anymore Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@413 PS5, Line 413: ts(dbName_, tb > I think this call is not correct since this will be a no-op if the table is Thanks for this catch. Used reloadTable() instead which forces a reload every time. reloadPartition() http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@601 PS5, Line 601:* Metastore event for INSERT events. > These following two lines can be Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@204 PS5, Line 204: > Typo sofar Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@38 PS5, Line 38: import org.apache.hadoop.hive.metastore.PartitionDropOptions; > unused? Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3551 PS5, Line 3551: // partition key will be empty. > nit : comma after tables. Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3557 PS5, Line 3557: InsertForTable = new HashSet<>(); > may be a better name would be to suggest that this contains files before in Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3558 PS5, Line 3558: eConf = new HiveConf(this. > is this unused? Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3580 PS5, Line 3580: rtition.isMarkedCached()) > Is there a concern here of running into CastException? I see that FeFsParti This code path is taken by only HDFSTables as you can see on L3522. http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3673 PS5, Line 3673: // For > may be do a else if(catalog.isExternalEventProcessingEnabled()) here so tha Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3674 PS5, Line 3674: Preconditions.checkState(parts.size() == 1); > May be add a Preconditions.checkState(parts.size == 1); here to make sure t Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3676 PS5, Line 3676: BeforeInsertForTable = ((( > same as above, Do we need to handle LocalFsPartition as well? The code path is taken by HdfsTable, so we do not need to handle LocalFsPartition. http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3699 PS5, Line 3699: createInsertEve > May be you can create 2 methods, one for partitioned case and another for n Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3716 PS5, Line 3716: } > Add to the description that this method is a no-op if event processing is d Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3719 PS5, Line 3719: nsert and ca > nit, change the name to isInsertOverwrite Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3736 PS5, Line 3736: List newFiles = new ArrayList<>(); > I think it would be helpful to add info log here which says how many new fi Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3750 PS5, Line 3750: } > add a info level log here which prints how many new files were added into t Done
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 514 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/8 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 514 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/9 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 538 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/10 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 10: (3 comments) http://gerrit.cloudera.org:8080/#/c/12889/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/12889/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3823 PS9, Line 3823: > This is interesting because based on the thrift definition of InsertEventRe Done http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@450 PS6, Line 450: testInsertEvents > Does this test inserOverwrite case as well? Seems important enough not to b >From Impala side, we are not using the overwrite flag so I thought it was too >trivial to test. But I see your point. Adding a test that calls a process on >overwrite event. http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@463 PS6, Line 463: String parentPathString = : "/test-warehouse/" + dbName +".db/" + tblName; : String filePathString = isPartitionInsert ? "/p1=testPartVal/testFile.0" : : "/testFile.0"; > I am not a big fan of hardcoding paths in the tests esp. if the root locati Thanks for this suggestion. The load table at L489 is after a partition has been added. catalog_.getTable().getMetastoreTable().getSd().getLocation() is null for some reason. I will investigate more. I'm trying to avoid getOrLoad(). For now let's keep this. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 12 Apr 2019 03:51:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 540 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/12 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 12 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 539 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/11 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/12889/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12889/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@516 PS12, Line 516: true, > you probably meant to use true here? Yes. Corrected. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 13 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 12 Apr 2019 06:14:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 540 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/13 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 13 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 528 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/16 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 16 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 525 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/17 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 17 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 569 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/18 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 18 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 18: Thanks for the pointers Bharath. I tried to make the wait for event processing more deterministic by scraping the /events page for last_sync_event_id to know if the events have been processed. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 18 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 20:59:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 19: (2 comments) Thanks Bharath. Changed the tests to account for slower builds. http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py File tests/custom_cluster/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py@122 PS18, Line 122: time.sleep(bu > Lets use a longer timeout for slower builds like ASAN. How about 2 for regu Done http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py@133 PS18, Line 133: > Breaks if someone updates the page. Parse it into keyvalue pairs and pick u Done -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 19 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 00:57:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 575 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/19 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 19 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 548 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/14 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 14 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12889 )
Change subject: IMPALA-7971: Add support for insert events in event processor.
..
Patch Set 14:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/12889/13/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/12889/13/common/thrift/CatalogService.thrift@193
PS13, Line 193: // True if the update corresponds to an "insert overwrite"
operation
> nit: I think we should say "True if this update corresponds to an 'insert o
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@396
PS13, Line 396: /**
: * Util method to issue invalidate on a given table on the
catalog. This method
: * atomically invalidates the table if it exists in the
catalog. No-op if the table
: * does not exist
: */
: p
> don't think this needs a separate method. Inline it at the caller?
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@582
PS13, Line 582:
> ..handler..Also, add some more color to it? Like it handles the inserts at
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@586
PS13, Line 586:
: InsertEven
> instead say. Null if the table is unprartitioned...or something like that?
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@614
PS13, Line 614:
> this is obvious, remove?
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@619
PS13, Line 619: **
: * Process partition inserts
: */
: private void processPartit
> braces
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@630
PS13, Line 630: Preconditions.checkState(fsList.size() ==
partVals.size());
> Preconditions.checkNotNull(insertPartition_);
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@664
PS13, Line 664: d
> unpartitioned ..
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@667
PS13, Line 667: try {
> Preconditions.checkArgument(partition == null)
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3583
PS13, Line 3583: Collection parts =
: FeCatalogUtils.loadAllParti
> Remove, this is obv?
Done
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3589
PS13, Line 3589: List existingPartitionsTouchedByInsert =
new ArrayList<>();
> I suggest to refactor the code like this. Lemme know what you think. I thin
Thanks for the suggestion. Sounds good. Refactored code according to your
suggestion. However, tracking files for partitions is slightly different from
that with table inserts. non-partitioned tables change ids after load, so we
cannot track using a map. Hence the if...else for calculating files.
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3767
PS13, Line 3767: * fireInsertEvent() if external event processing is enabled.
This is no-op otherwise.
:*
> shouldn't this be done only for the affected partitions?
The filesBeforeInsertForPartitions map contains only the affected partitions.
http://gerrit.cloudera.org:8080/#/c/12889/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3823
PS13, Line 3823: * insert. In case of partitioned table, this event also
contains the partition
:* values of existing partitions which were touched by the
insert.
:*/
: private void fireInsertEvent(Table tbl, List
partVals,
:
> insertData.setReplace(isOverwrite)
Done
--
To view, visit http://gerrit.cloudera.org:8080/12889
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id:
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12889 )
Change subject: IMPALA-7971: Add support for insert events in event processor.
..
Patch Set 15:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3587
PS14, Line 3587: Map> filesBeforeInsertForPartitions = new
HashMap<>();
: Set filesBeforeInsertForTable = new HashSet<>();
> Both of them can be handled in a single map? Also, rename to filesBeforeIns
Here we track fileNames of all affectedPartitions using a Map. Later, this map is used to retrieve and compare fileNames to
calculate deltas using partitionIds. However, partitionId cannot be used for a
non-partitioned table because they change during load().
One solution was to use partitionNames instead of partitionIds. We decided to
go with Ids because using partitionNames was too error prone.
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3589
PS14, Line 3589: affectedExistingPartitions = new
> just say affectedPartitions ?
Changed it to affectedExistingPartitions. Want to stress on 'existing' as these
are existing partitions that are involved in this insert event. There could be
some new partitions created in the insert which are not stored here.
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3608
PS14, Line 3608: if (catalog_.isExternalEventProcessingEnabled()) {
> not needed.
We don't wish to do this if event processing is disabled right?
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3704
PS14, Line 3704: if (catalog_.isExternalEventProcessingEnabled()) {
> not needed.
We don't wish this path be taken if event processor is disabled. Right?
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3736
PS14, Line 3736: filesBeforeInsertForTable = ((HdfsPartition)
Iterables
> Like I mentioned above, I don't see why we should use two different datastr
As mentioned before, we cannot keep track of filenames using partitionIds for
non-partitioned tables as we drop and create the single partition during load.
(partitionId changes.). Hence two datastructures.
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3743
PS14, Line 3743: // After loading metadata, fire insert event if external
event processing is
: // enabled.
: if (table.getNumClusteringCols() > 0) {
: createInsertEventsForPartitions(table,
filesBeforeInsertForPartitions,
: update.is_overwrite);
: } else {
: createInsertEventForTable(table,
filesBeforeInsertForTable, update.is_overwrite);
: }
> How about changing the signature to
As mentioned above this is not possible because of two different data
structures for filesBeforeInsert for partition case and non-partitioned case
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3787
PS14, Line 3787: deltaFiles = ((HdfsPartition)part).getFileNames();
> shouldn't this go before if? tests didn't catch this?
This looks right to me. We want deltaFiles to be empty when it is an overwrite.
We only calculate the new files if its not an overwrite.
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3791
PS14, Line 3791: deltaFiles.size
> need this?
Removed.
http://gerrit.cloudera.org:8080/#/c/12889/14/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3813
PS14, Line 3813: deltaFiles = ((HdfsPartition)singlePart).getFileNames();
> same question, shouldn't deltaFiles go before if?
We want delta files to be empty if it is an overwrite.
--
To view, visit http://gerrit.cloudera.org:8080/12889
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52
Gerrit-Change-Number: 12889
Gerrit-PatchSet: 15
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Sat, 13 Apr 2019 08:19:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 548 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/15 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 15 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 4: (1 comment) Thanks for creating a test case for non-existent entities. Patch looks good to me other than a minor comment below. http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@755 PS4, Line 755: createFakeInsertEvent I see you are creating the insert events by calling the HMS API. You can probably remove this method? -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 10 Jun 2019 20:46:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7947: script to push images to docker repo
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13698 ) Change subject: IMPALA-7947: script to push images to docker repo .. Patch Set 1: Code-Review+1 (1 comment) Looks good to me. http://gerrit.cloudera.org:8080/#/c/13698/1/docker/push-images.sh File docker/push-images.sh: http://gerrit.cloudera.org:8080/#/c/13698/1/docker/push-images.sh@54 PS1, Line 54: required Nit: Typo -- To view, visit http://gerrit.cloudera.org:8080/13698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0996b090f513351b58c801ed7149f80c4188f903 Gerrit-Change-Number: 13698 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 21 Jun 2019 02:44:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8559 : Support config validation for event processor on HMS-3
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13372 ) Change subject: IMPALA-8559 : Support config validation for event processor on HMS-3 .. Patch Set 3: Code-Review+1 The patch looks good to me. I will let one other set of eyes to have a look at it. -- To view, visit http://gerrit.cloudera.org:8080/13372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I35b3dd93f4a90c103a402349e8b9cd36db39a259 Gerrit-Change-Number: 13372 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 May 2019 20:00:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 212 insertions(+), 79 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/8 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2115 PS7, Line 2115: @throws > typo: @throws Done http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@551 PS7, Line 551: TPart > TPartiton is the name of the struct, so it's more readable if we don't shor Done http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@568 PS7, Line 568: P > nit: this should be an upper case P Done -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 17:22:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 24: Rebased after CDH change. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 24 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 15:26:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/resources/hive-site.xml.py A tests/custom_cluster/test_event_processing.py 10 files changed, 592 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/24 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 24 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1046268.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13140 Change subject: Bump CDH_BUILD_NUMBER to 1046268. .. Bump CDH_BUILD_NUMBER to 1046268. This change brings in the CDH version which fixes an issue preventing hive from firing insert events for insert overwrites into dynamically loaded partitions. This was affecting an incoming impala patch to support insert events processing IMPALA-7971. Testing: Ran a full exaustive build using imapala-private-parametrized job. Change-Id: I7b367d803d396196988842f3fb77896725145ff2 --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13140/1 -- To view, visit http://gerrit.cloudera.org:8080/13140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Gerrit-Change-Number: 13140 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 205 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/4 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 6: Resolved merge conflicts with IMPALA-8454. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 22:31:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 201 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/6 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 5: Rebased after insert events patch was merged. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 21:37:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 200 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/5 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 212 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/7 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/resources/hive-site.xml.py A tests/custom_cluster/test_event_processing.py 10 files changed, 576 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/21 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 21 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 21: Build failed as the HMS flag that enables insert notification was not set by default. Added hive.metastore.dml.events=true config in hive-site.xml.py -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 21 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 22:13:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 21: This python file generates the templates for hive-site.xml - Every mini-cluster will have this config set. This is just like the event processing flags are set. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 21 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 23:03:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13860 Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. createInsertEvents() uses partition ids to keep track of the files added to the partitions by the insert by finding the delta of files in a partition before and after load() call. However, if partitions are marked dirty (for eg.: partitions created by RECOVER PARTITIONS), load() will drop and re-create them which will change the partition ids. createInsertEvents() then cannot find these parittions and fails with exception. In this patch, partitions are tracked by partition names instead of partition ids so drop + reload will not affect the logic. Testing: 1. Ran TestRecoverPartitions.test_post_invalidate() which was failing. 2. Ran MetastoreEventProcessorTest FE tests. Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 60 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13860/1 -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13111 Change subject: IMPALA-7973: Add support for fine grained updates at partition level. .. IMPALA-7973: Add support for fine grained updates at partition level. This patch adds suport for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be very expensive for large tables. Here, we refresh partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction. We throw MetastoreNotificationNeedsInvalidateException if any of the partition refreshes fails. Testing: Modified pre-existing tests for partition events to insead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 182 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/1 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained updates at partition level. .. IMPALA-7973: Add support for fine grained updates at partition level. This patch adds suport for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be very expensive for large tables. Here, we refresh partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction. We throw MetastoreNotificationNeedsInvalidateException if any of the partition refreshes fails. Testing: Modified pre-existing tests for partition events to insead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 183 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/2 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 512 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/7 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12889 )
Change subject: IMPALA-7971: Add support for insert events in event processor.
..
Patch Set 7:
(31 comments)
Thanks Vihang and Bharath for the comments. Is there a better way to handle
finally block in MetastoreEventsProcessorTest?
http://gerrit.cloudera.org:8080/#/c/12889/6/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/12889/6/be/src/service/client-request-state.cc@1106
PS6, Line 1106:
> nit: I think this is kind of obv (and also there is a comment in the thrift
Removed.
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2063
PS6, Line 2063: erwise. Throws CatalogE
> Looks like it throws DbNotFoundEx if the db doesn't exist.
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2065
PS6, Line 2065: */
: public boolean refreshTableIfExists(String dbName
> nit: Just say, returns true if reload is successful, false otherwise? Remov
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2077
PS6, Line 2077:
> nit, use "after insert events" to be more specific
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2077
PS6, Line 2077: Db doesn't exist.
> I feel the entire statement can be omitted. Callers may change that makes t
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2082
PS6, Line 2082: if (table == null || table instanceof IncompleteTable)
return false;
> same comments as above javadoc.
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@835
PS6, Line 835: s
> nit: Returns..
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@398
PS6, Line 398: */
: protected String getPartitionKeyValuesAsString(Map partSpec) {
: return
Joiner.on(",").withKeyValueSeparator("=").join(partSpec);
: }
:
: /**
:
> Use Joiner from guava
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@423
PS6, Line 423: value of the table
> since this method is not doing anything else apart from calling catalog_ref
I left this here because I wanted to add metrics of "Tables Refreshed" in a
separate patch. Not very sure if this is any useful to users. Would like to
know your thoughts. For now, I have removed it.
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@431
PS6, Line 431: Boolean tblProperty = getHmsSyncProperty(msTbl_);
> same as above, The method is doing nothing other than calling another metho
Same as earlier reply. Wanted to have metric for partitionsRefreshed here.
Removed for now.
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@630
PS6, Line 630: Map partSpec = new HashMap<>();
> Any reason why?
Added the reason for this
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@635
PS6, Line 635: Preconditions.checkState(fsList.size() == partVals.size());
> can we break this into two helpers? processPartitionedInsert() and processN
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@641
PS6, Line 641: // refresh fails.
> probably worth while to check the following conditions:
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@649
PS6, Line 649:
> nit, remove Automatic. kind of redundant since debugLog prints the event in
Done
http://gerrit.cloudera.org:8080/#/c/12889/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@673
PS6, Line 673: nt in the
> nit, remove
Done
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affetced in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 208 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/3 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
..
Patch Set 3:
(15 comments)
Thanks for your comments Vihang and Bharath. Please review my comments.
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7973: Add support for fine grained events processing for
> nit: mention about "events" in the title? Otherwise it looks very general .
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@13
PS2, Line 13:
> nit: ..affected..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@14
PS2, Line 14: refresh affected partitions
: in case of add/d
> Not sure I understand this, can you please clarify?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@19
PS2, Line 19: on to
> nit:typo
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085:
> nit: ..it..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085: s. Returns true if reload of the partition succeeds,
:* false othe
> Instead of referring to specific methods, rephrase it to something like re
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2089
PS2, Line 2089: reloadP
> nit:call "reload" to be consistent ?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1102
PS2, Line 1102: per(catalog, metrics, event);
:
Preconditions.checkState(eventType_.equals(MetastoreEventType.ADD_PARTITION));
: if (event.getMessage() == null) {
: throw new IllegalStateException(debugString("Event
messag
> nit: Not super clear what this is. How are HMS transactional semantics rela
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1106
PS2, Line 1106:
> Would be good if you can log a useful information about the event like numb
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1107
PS2, Line 1107: {
: AddPartitionMessage addPartitionMessage_ =
: MetastoreEventsProcessor.getMessageFactory()
: .getDeserializer()
: .getAddPartitionMessage(event.getMessage());
: addedPartitions_ =
:
Lists.newArrayList(addPartitionMessage_.getPartitionObjs());
: Preconditions.checkState(addedPartitions_.size() > 0);
: //
> We can avoid one unnecessary conversion from Partition -> partSpecMap -> TP
Created static method in base class to do this.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1118
PS2, Line 1118:
> nit: include the partition name too
This scenario is only if the table is not present in the catalog, Next load of
the table will refresh all the partitions anyway. Do you think logging which
partition failed is useful to the user in this case?
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1125
PS2, Line 1125:
> include the partition name
In success case, it can be a really log list of partitions. (eg:- dynamically
loaded partitions)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1194
PS2, Line 1194: */
: private AlterPartitionEvent(CatalogServiceCatalog catalog,
Metrics metrics,
: NotificationEvent event) throws
MetastoreNotificationException {
: super(catalog, metrics, event);
:
Preconditions.checkState(eventType_.equals(MetastoreEventType.ALTER_PARTITION));
: Preconditions.checkNotNull(event.getMessage());
: AlterPartitionMessage alterPartitionMessage =
:
MetastoreEventsProcessor.getMessageFactory().getDeserializer()
:
[Impala-ASF-CR] IMPALA-8734: Reload table schema on TBLPROPERTIES change
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13785 ) Change subject: IMPALA-8734: Reload table schema on TBLPROPERTIES change .. Patch Set 3: Code-Review+1 Looks good to me -- To view, visit http://gerrit.cloudera.org:8080/13785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad Gerrit-Change-Number: 13785 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Jul 2019 17:50:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7975 : Improve supportability of the automatic invalidate feature
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13947 ) Change subject: IMPALA-7975 : Improve supportability of the automatic invalidate feature .. Patch Set 1: (1 comment) Congratulations on your first contribution. I left a comment below. http://gerrit.cloudera.org:8080/#/c/13947/1/be/src/util/event-metrics.h File be/src/util/event-metrics.h: http://gerrit.cloudera.org:8080/#/c/13947/1/be/src/util/event-metrics.h@59 PS1, Line 59: IntGauge Is there a reason why we want to use "guage" instead of a "counter"? IIUC, "guage" can increase or decrease over time whereas "counter" can only increase. Is there a case for LAST_SYNCED_EVENT_ID to decrease? -- To view, visit http://gerrit.cloudera.org:8080/13947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib113a5b9458dcf483b183e927544a6c6d46e1af3 Gerrit-Change-Number: 13947 Gerrit-PatchSet: 1 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Jul 2019 06:37:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12443 )
Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode.
..
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12443/8/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/12443/8/tests/custom_cluster/test_local_catalog.py@386
PS8, Line 386: in impalad.service.read_debug_webpage('/catalog_object')
> nit: 4 indent spaces.
Done
http://gerrit.cloudera.org:8080/#/c/12443/8/www/catalog.tmpl
File www/catalog.tmpl:
http://gerrit.cloudera.org:8080/#/c/12443/8/www/catalog.tmpl@42
PS8, Line 42:
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. IMPALA-7935: Disable /catalog_object in local catalog mode. getTCatalogObject() is not supported in local catalog mode since metadata is partially fetched on demand. Removed hyperlinks to the /catalog_object endpoints when local_catalog_mode is enabled. Testing: Added a test to test_local_catalog::TestObservability to verify /catalog_mode endpoint is disabled when in local catalog mode. Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c --- M be/src/service/impala-http-handler.cc M tests/custom_cluster/test_local_catalog.py M www/catalog.tmpl 3 files changed, 45 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12443/9 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. IMPALA-7935: Disable /catalog_object in local catalog mode. getTCatalogObject() is not supported in local catalog mode since metadata is partially fetched on demand. Removed hyperlinks to the /catalog_object endpoints when local_catalog_mode is enabled. Testing: Added a test to test_local_catalog::TestObservability to verify /catalog_mode endpoint is disabled when in local catalog mode. Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c --- M be/src/service/impala-http-handler.cc M tests/custom_cluster/test_local_catalog.py M www/catalog.tmpl 3 files changed, 43 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12443/8 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 8: (2 comments) http://gerrit.cloudera.org:8080/#/c/12443/7/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/12443/7/be/src/service/impala-http-handler.cc@109 PS7, Line 109: // The /catalog_object endpoint is disabled if local_catalog_mode is used > nit: add why Done. http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@610 PS7, Line 610: > Can you also scrape the webpage source and make sure it doesn't have a "cat In fact, this test is redundant. There is already a test for checking /catalog_obj endpoint is working for both loaded and unloaded tables. Removing this. -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 16 Aug 2019 18:12:11 + Gerrit-HasComments: Yes
