[Impala-ASF-CR] IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10680 ) Change subject: IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare() .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc@915 PS3, Line 915: llvm::Function* LlvmCodeGen::ReplaceCallSitesRecursively(llvm::Function* caller, > I believe your analysis is right - I spent some time looking at it after I Sorry my 2nd argument is invalid. I was thinking if IR entry point is A, A calls B, B calls a crafted IR, sometimes you probably don't want to inline B into A, and may get rid of ALWAYS_INLINE and give LLVM the choice. That dosn't make sense. -- To view, visit http://gerrit.cloudera.org:8080/10680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4657ac09daf20408797856d94521d417d8cf171 Gerrit-Change-Number: 10680 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 29 Mar 2019 17:05:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10680 ) Change subject: IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare() .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc@915 PS3, Line 915: llvm::Function* LlvmCodeGen::ReplaceCallSitesRecursively(llvm::Function* caller, > If you have A->B B->A A->C, C in F, entry point = A. I believe you need to The additional benefit of this is that, when you clone and replace, LLVM can choose what to inline and what not to. That would reduce the codegen time and reduce the size of generated code. -- To view, visit http://gerrit.cloudera.org:8080/10680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4657ac09daf20408797856d94521d417d8cf171 Gerrit-Change-Number: 10680 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 29 Mar 2019 01:50:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10680 ) Change subject: IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare() .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc@915 PS3, Line 915: llvm::Function* LlvmCodeGen::ReplaceCallSitesRecursively(llvm::Function* caller, > This patch is too big for me to adopt at the moment but it would be nice to If you have A->B B->A A->C, C in F, entry point = A. I believe you need to clone B in this case because you need to call A'. When you visit A->C in a plain DFS you've already finished visiting B. How can you know that B needs to be cloned with plain DFS? A 2nd pass? -- To view, visit http://gerrit.cloudera.org:8080/10680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4657ac09daf20408797856d94521d417d8cf171 Gerrit-Change-Number: 10680 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 29 Mar 2019 01:35:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7047. Refreshing partitions should not make an RPC per file
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11227 ) Change subject: IMPALA-7047. Refreshing partitions should not make an RPC per file .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/11227/7/fe/src/test/java/org/apache/impala/catalog/CatalogTest.java File fe/src/test/java/org/apache/impala/catalog/CatalogTest.java: http://gerrit.cloudera.org:8080/#/c/11227/7/fe/src/test/java/org/apache/impala/catalog/CatalogTest.java@347 PS7, Line 347: // - one listLocatedStatus() per partition, to get the file info Any idea where this is called? -- To view, visit http://gerrit.cloudera.org:8080/11227 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2051b96599206164aaa06ecbdf64374c46eda956 Gerrit-Change-Number: 11227 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 03 Dec 2018 21:12:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7870: Increase the timeout in test v1 catalog
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11997 Change subject: IMPALA-7870: Increase the timeout in test_v1_catalog .. IMPALA-7870: Increase the timeout in test_v1_catalog TestAutomaticCatalogInvalidation.test_v1_catalog need to wait for a predefined time for the invalidation to take effect. The test is flaky recently because of it. This patch increates the timeout by 2.5x. Change-Id: If7d37a6109b2e8de1473d42d699b8c7057d0b29b --- M tests/custom_cluster/test_automatic_invalidation.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11997/1 -- To view, visit http://gerrit.cloudera.org:8080/11997 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If7d37a6109b2e8de1473d42d699b8c7057d0b29b Gerrit-Change-Number: 11997 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has abandoned this change. ( http://gerrit.cloudera.org:8080/11641 ) Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 00:49:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml File docs/topics/impala_config_options.xml: http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml@356 PS1, Line 356: invalidate invalidates http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml@366 PS1, Line 366: When : the size of the garbage collection of catalogd : exceeds a certain threshold, Impala invalidates a certain fraction of : the least recently used tables. "size of the garbage collection" isn't accurate, how about: "When the memory pressure is high after a Java garbage collection in catalogd, Impala invalidates ... " -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 00:42:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/11641 )
Change subject: IMPALA-7670: Avoid getting the latest tables in
bulkAlterPartitions()
..
Patch Set 1:
> Patch Set 1:
>
> (1 comment)
>
> ya I don't think it is specific to V2 (updated the jira description).
I've been running concurrent invalidate and drop stats with this diff and
haven't reproduced it.
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
b/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
index e9ef929103..7058a384e2 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
@@ -424,6 +424,7 @@ public interface FeFsTable extends FeTable {
break;
}
}
+try { Thread.sleep(100);} catch (Exception e) {}
if (matchFound) return partition;
}
return null;
diff --git a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
index 88c1fd5138..892be0a5a7 100644
--- a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
@@ -792,6 +792,7 @@ public class HdfsTable extends Table implements FeFsTable {
HdfsPartition partition = createPartition(msPartition.getSd(),
msPartition,
permCache);
addPartition(partition);
+try { Thread.sleep(100);} catch (Exception e) {}
// If the partition is null, its HDFS path does not exist, and it was
not added
// to this table's partition list. Skip the partition.
if (partition == null) continue;
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 36725df84c..e2fcc785a4 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -3275,6 +3275,7 @@ public class CatalogOpExecutor {
MetastoreShim.alterPartitions(msClient.getHiveClient(), dbName,
tableName,
hmsPartitionsSubList);
// Mark the corresponding HdfsPartition objects as dirty
+ try {Thread.sleep(3000);} catch (Exception e) {}
for (org.apache.hadoop.hive.metastore.api.Partition msPartition:
hmsPartitionsSubList) {
try {
--
To view, visit http://gerrit.cloudera.org:8080/11641
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897
Gerrit-Change-Number: 11641
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 12 Oct 2018 21:38:28 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11670 ) Change subject: IMPALA-7622: adds profile metrics for incremental stats .. Patch Set 1: Code-Review+2 This was reverted only to revert IMPALA-7527, right? -- To view, visit http://gerrit.cloudera.org:8080/11670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273 Gerrit-Change-Number: 11670 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 12 Oct 2018 19:07:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11641 ) Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. Patch Set 1: > Can we add a targeted stress test to reproduce this? The JIRA also > seems to indicate this is a v2-only issue but I can't understand > how that would be the case. Are we sure it's a regression? We haven't been able to reproduce it. It only happened once on V2 and it is not necessarily a V2-only issue. If my theory is correct it shouldn't be a regression. -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Oct 2018 21:14:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11641 ) Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11641/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11641/1//COMMIT_MSG@11 PS1, Line 11: without locking > Just a theory. I have an impression that the table in tablecache can be rep For example CatalogServiceCatalog.invalidateTable doesn't lock the table before removing it from catalog. If bulkAlterPartitions is called concurrently it might observe a different table without locking. -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Oct 2018 21:11:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11641 ) Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11641/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11641/1//COMMIT_MSG@11 PS1, Line 11: without locking > Hmm. If that is the case, some call path is violating the locking requireme Just a theory. I have an impression that the table in tablecache can be replaced even if it's locked. Let me look at that part of the code again. -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Oct 2018 17:43:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11641 ) Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11641/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11641/1//COMMIT_MSG@11 PS1, Line 11: without locking > Doesn't the caller of the bulkAlterPartitions() hold the table lock? For ex I think the table is locked but bulkAlterPartitions is fetching the table again from table cache, which might be a different table object. -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Oct 2018 02:35:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11641 Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() bulkAlterPartitions() needs to mark altered partitions as dirty. It currently gets the latest table using table names and mark partitions in them as dirty without locking, which could lead to concurrent modifications of a table. This patch changes it into marking the tables it operates on dirty. Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 21 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/11641/1 -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11603 ) Change subject: IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC .. IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC UpdateTableUsage() is logically a one-way RPC and the status object in TUpdateTableUsageResponse is set only if there is an error at RPC layer. This patch fixes the incorrect error handling that leads to NullPointerException in ImpaladTableUsageTracer. Change-Id: Iccba4c6f4696ef08bc8a614ae13f62b5e445917b --- M be/src/service/fe-support.cc M fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java 2 files changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/11603/2 -- To view, visit http://gerrit.cloudera.org:8080/11603 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iccba4c6f4696ef08bc8a614ae13f62b5e445917b Gerrit-Change-Number: 11603 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11603 Change subject: IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC .. IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC UpdateTableUsage() is logically a one-way RPC and the status object in TUpdateTableUsageResponse is set only if there is an error at RPC layer. This patch fixes the incorrect error handling that leads to NullPointerException in ImpaladTableUsageTracer. Change-Id: Iccba4c6f4696ef08bc8a614ae13f62b5e445917b --- M be/src/service/fe-support.cc M fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java 2 files changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/11603/1 -- To view, visit http://gerrit.cloudera.org:8080/11603 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iccba4c6f4696ef08bc8a614ae13f62b5e445917b Gerrit-Change-Number: 11603 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7661: Increase the sleep time in test reconnect
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11589 Change subject: IMPALA-7661: Increase the sleep time in test_reconnect .. IMPALA-7661: Increase the sleep time in test_reconnect test_reconnect is flaky in ASAN because the time waited for impala shell to connect to impalad is not enough. This patch increases the sleep time from 2 secs to 5 secs. Change-Id: Ia009808adac0da1cfa00b9e9dd41cc276d49c6eb --- M tests/shell/test_shell_interactive.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/11589/1 -- To view, visit http://gerrit.cloudera.org:8080/11589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia009808adac0da1cfa00b9e9dd41cc276d49c6eb Gerrit-Change-Number: 11589 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7532: Add catalogd client backoff time into impalad CLI options
Tianyi Wang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/11543 ) Change subject: IMPALA-7532: Add catalogd client backoff time into impalad CLI options .. IMPALA-7532: Add catalogd client backoff time into impalad CLI options Impala may fail queries or fail to start if the connection to catalogd cannot be estabilished. Impala already has a retrial mechanism but the backoff time is currently 0. This patch adds an option "catalog_client_rpc_retry_interval_ms" for it, defaulting to 10 seconds. Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 --- M be/src/runtime/exec-env.cc 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/11543/4 -- To view, visit http://gerrit.cloudera.org:8080/11543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 Gerrit-Change-Number: 11543 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7532: Add catalogd client backoff time into impalad CLI options
Tianyi Wang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/11543 ) Change subject: IMPALA-7532: Add catalogd client backoff time into impalad CLI options .. IMPALA-7532: Add catalogd client backoff time into impalad CLI options Impala may fail queries or fail to start if the connection to catalogd cannot be estabilished. Impala already has a retrial mechanism but the backoff time is currently 0. This patch adds an option for it, defaulting to 10 seconds. Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 --- M be/src/runtime/exec-env.cc 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/11543/3 -- To view, visit http://gerrit.cloudera.org:8080/11543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 Gerrit-Change-Number: 11543 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7532: Add catalogd client backoff time into impalad CLI options
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11543 Change subject: IMPALA-7532: Add catalogd client backoff time into impalad CLI options .. IMPALA-7532: Add catalogd client backoff time into impalad CLI options Impala may fail queries or fail to start if the connection to catalogd cannot be estabilished. Impala already has a retrial mechanism but the backoff time is currently 0. This patch adds an option for it, defaulting to 10 seconds. Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 --- M be/src/runtime/exec-env.cc 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/11543/2 -- To view, visit http://gerrit.cloudera.org:8080/11543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 Gerrit-Change-Number: 11543 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7632: fix erasure coding build for custom cluster tests
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11536 ) Change subject: IMPALA-7632: fix erasure coding build for custom cluster tests .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11536 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d809faedc0c45417519f13c73559efb6c54154e Gerrit-Change-Number: 11536 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 27 Sep 2018 21:01:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11512 ) Change subject: IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11512/4/tests/custom_cluster/test_automatic_invalidation.py File tests/custom_cluster/test_automatic_invalidation.py: http://gerrit.cloudera.org:8080/#/c/11512/4/tests/custom_cluster/test_automatic_invalidation.py@100 PS4, Line 100: openjdk 1.8. > might be good to verify this for 7 since some tests still run with this afa Someone checked the source code and it does. https://stackoverflow.com/questions/11390665/can-jmap-histo-trigger-full-garbage-collection#15747911 It's not a documented feature BTW. -- To view, visit http://gerrit.cloudera.org:8080/11512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Gerrit-Change-Number: 11512 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 27 Sep 2018 00:47:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11512 ) Change subject: IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/11512/2/fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java File fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java: http://gerrit.cloudera.org:8080/#/c/11512/2/fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java@157 PS2, Line 157: NotificationEmitter > any thoughts on how to test this? it seemed to pass manual tests, so wonder added a test using jmap to trigger a GC http://gerrit.cloudera.org:8080/#/c/11512/2/tests/custom_cluster/test_automatic_invalidation.py File tests/custom_cluster/test_automatic_invalidation.py: http://gerrit.cloudera.org:8080/#/c/11512/2/tests/custom_cluster/test_automatic_invalidation.py@81 PS2, Line 81: > nit: a Done http://gerrit.cloudera.org:8080/#/c/11512/2/tests/custom_cluster/test_automatic_invalidation.py@81 PS2, Line 81: > nit: don't Done http://gerrit.cloudera.org:8080/#/c/11512/2/tests/custom_cluster/test_automatic_invalidation.py@82 PS2, Line 82: led to > nit: for Done -- To view, visit http://gerrit.cloudera.org:8080/11512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Gerrit-Change-Number: 11512 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 26 Sep 2018 23:58:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator
Tianyi Wang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/11512 ) Change subject: IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator .. IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator CatalogdTableInvalidator detects if a table is in a normal state using Table.isLoaded() function. This is wrong because if there is an error during the loading of a table, isLoaded() returns true. This patch checks if the table is an IncompleteTable instead. Also fixed a bug in tryInstallGcListener(). A test is added to test the memory-based invalidation. Change-Id: If938a40434b00af516445152f88832ef55d0d0ce --- M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M tests/custom_cluster/test_automatic_invalidation.py 2 files changed, 36 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/11512/4 -- To view, visit http://gerrit.cloudera.org:8080/11512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Gerrit-Change-Number: 11512 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11512 ) Change subject: IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator .. IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator CatalogdTableInvalidator detects if a table is in a normal state using Table.isLoaded() function. This is wrong because if there is an error during the loading of a table, isLoaded() returns true. This patch checks if the table is an IncompleteTable instead. Also fixed a bug in tryInstallGcListener(). Change-Id: If938a40434b00af516445152f88832ef55d0d0ce --- M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M tests/custom_cluster/test_automatic_invalidation.py 2 files changed, 19 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/11512/2 -- To view, visit http://gerrit.cloudera.org:8080/11512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Gerrit-Change-Number: 11512 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11512 Change subject: IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator .. IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator CatalogdTableInvalidator detects if a table is in a normal state using Table.isLoaded() function. This is wrong because if there is an error during the loading of a table, isLoaded() returns true. This patch checks if the table is an IncompleteTable instead. Change-Id: If938a40434b00af516445152f88832ef55d0d0ce --- M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/11512/1 -- To view, visit http://gerrit.cloudera.org:8080/11512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Gerrit-Change-Number: 11512 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7594: Fix the waiting time in test automatic invalidation
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11471 ) Change subject: IMPALA-7594: Fix the waiting time in test_automatic_invalidation .. IMPALA-7594: Fix the waiting time in test_automatic_invalidation IMPALA-7593 increased the table invalidation timeout but the time to wait for the effect should also be increased. Change-Id: Ibb41e615e42712f9f75a4180f55270f8d4159668 --- M tests/custom_cluster/test_automatic_invalidation.py 1 file changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/11471/2 -- To view, visit http://gerrit.cloudera.org:8080/11471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibb41e615e42712f9f75a4180f55270f8d4159668 Gerrit-Change-Number: 11471 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7594: Fix the waiting time in test automatic invalidation
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11471 ) Change subject: IMPALA-7594: Fix the waiting time in test_automatic_invalidation .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11471/1/tests/custom_cluster/test_automatic_invalidation.py File tests/custom_cluster/test_automatic_invalidation.py: http://gerrit.cloudera.org:8080/#/c/11471/1/tests/custom_cluster/test_automatic_invalidation.py@35 PS1, Line 35: timeout = 20 if IMPALAD_BUILD.runs_slowly() or (not IS_HDFS and not IS_LOCAL) else 10 > Basic question, for my understanding. Why does this param depend on the bui The tests will run a query and assumes the table used in the query is loaded after the query finishes. If the query runs slowly the table might have been invalidated when the query finishes. http://gerrit.cloudera.org:8080/#/c/11471/1/tests/custom_cluster/test_automatic_invalidation.py@53 PS1, Line 53: self.timeout * 2 > qq, 2*timeout because we want to make sure it is invalidated timeout before It only needs to be a little larger than timeout. 2* is empirical -- To view, visit http://gerrit.cloudera.org:8080/11471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb41e615e42712f9f75a4180f55270f8d4159668 Gerrit-Change-Number: 11471 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 19 Sep 2018 20:05:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7594: Fix the waiting time in test automatic invalidation
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11471 Change subject: IMPALA-7594: Fix the waiting time in test_automatic_invalidation .. IMPALA-7594: Fix the waiting time in test_automatic_invalidation IMPALA-7593 increased the table invalidation timeout but the time to wait for the effect should also be increased. Change-Id: Ibb41e615e42712f9f75a4180f55270f8d4159668 --- M tests/custom_cluster/test_automatic_invalidation.py 1 file changed, 5 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/11471/1 -- To view, visit http://gerrit.cloudera.org:8080/11471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibb41e615e42712f9f75a4180f55270f8d4159668 Gerrit-Change-Number: 11471 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7580: Increase timeout int test automatic invalidation
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11453 Change subject: IMPALA-7580: Increase timeout int test_automatic_invalidation .. IMPALA-7580: Increase timeout int test_automatic_invalidation The query in test_automatic_invalidation might finish after the table expires and renders the test flaky. This patch increases the timeout to 10 seconds on regular builds and 20 seconds on slow builds. Change-Id: Ifcf8fabdbfedf310b452bbc9b913a0a85c4b18f1 --- M tests/custom_cluster/test_automatic_invalidation.py 1 file changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/11453/1 -- To view, visit http://gerrit.cloudera.org:8080/11453 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifcf8fabdbfedf310b452bbc9b913a0a85c4b18f1 Gerrit-Change-Number: 11453 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout_s" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation are added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Reviewed-on: http://gerrit.cloudera.org:8080/11224 Tested-by: Impala Public Jenkins Reviewed-by: Tianyi Wang --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 27 files changed, 829 insertions(+), 25 deletions(-) Approvals: Impala Public Jenkins: Verified Tianyi Wang: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 11 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 13 Sep 2018 22:54:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout_s" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation are added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 27 files changed, 829 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/10 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 10 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. Patch Set 9: clang-tidy complained about codes I didn't change. I'll fix those warnings. -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 9 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 13 Sep 2018 19:14:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout_s" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation are added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 27 files changed, 812 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/9 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 9 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout_s" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation are added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 28 files changed, 809 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/8 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout_s" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation are added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 28 files changed, 804 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/7 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation is added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 25 files changed, 788 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/6 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 6 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation is added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 25 files changed, 788 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/5 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 5 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "invalidate_tables_timeout" is invalidated from catalogd. 2. If the old GC generation is almost full, a certain percentage of LRU tables are invalidated. This can be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are used during planning. Tests on time-based invalidation is added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 26 files changed, 787 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/4 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6758: Add metric for current catalog version to catalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11293 ) Change subject: IMPALA-6758: Add metric for current catalog version to catalog .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11293 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff7e158292ca9e5a17663e5bfc74931cc57c0328 Gerrit-Change-Number: 11293 Gerrit-PatchSet: 3 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Fri, 24 Aug 2018 20:58:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "unused_table_ttl_sec" is invalidated from catalogd. 2. If the old GC generation is almost full, a centen percentage of LRU tables are invalidated. This could be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are unsed during planning. Tests on time-based invalidation is added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A tests/custom_cluster/test_automatic_invalidation.py 27 files changed, 734 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/3 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11224 ) Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. There are two invalidation strategies: 1. Periodically the HDFS tables that are not used in a configured period "unused_table_ttl_sec" is invalidated from catalogd. 2. If the old GC generation is more than 70% full, 10% LRU tables are invalidated. This could be enabled by backend flag "invalidate_tables_on_memory_pressure". The table usage is reported by impalad to catalogd when the tables are unsed during planning. Tests on time-based invalidation is added. It is manually verified that the GC callback is called if strings are randomly stuffed into catalogd. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogdTableShrinker.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/ImpaladTableUsageTracker.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogdTableShrinkerTest.java A tests/custom_cluster/test_automatic_invalidation.py 25 files changed, 584 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/2 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7448: Invalidate recently unused tables from catalogd
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11224 Change subject: IMPALA-7448: Invalidate recently unused tables from catalogd .. IMPALA-7448: Invalidate recently unused tables from catalogd This patch implements an automatic invalidation mechanism in catalogd. This feature can be enabled with an catalogd CLI arg --unused_table_ttl_sec. Periodically the HDFS tables that are not used in unused_table_ttl_sec is invalidated from catalogd. The table usage is reported by impalad to catalogd when a query/DML finishes its execution. This patch also enables manual eviction from catalog webpage /invalidate_lru_table for experiment purposes. Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A tests/custom_cluster/test_automatic_invalidation.py 21 files changed, 332 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/11224/1 -- To view, visit http://gerrit.cloudera.org:8080/11224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib549717abefcffb14d9a3814ee8cf0de8bd49e89 Gerrit-Change-Number: 11224 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7347: Ignore numFilesErasureCoded in TestShowCreateTable
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11192 Change subject: IMPALA-7347: Ignore numFilesErasureCoded in TestShowCreateTable .. IMPALA-7347: Ignore numFilesErasureCoded in TestShowCreateTable This table properties only exist for HDFS tables. To get the test work on local tables, it needs to be ignored. Change-Id: Icc8494fb91c4777cee662a97f750486aa8e79a8e --- M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test M tests/metadata/test_show_create_table.py 2 files changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/11192/1 -- To view, visit http://gerrit.cloudera.org:8080/11192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icc8494fb91c4777cee662a97f750486aa8e79a8e Gerrit-Change-Number: 11192 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7383: Configurable HMS and Sentry policy DB
Tianyi Wang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable HMS and Sentry policy DB .. IMPALA-7383: Configurable HMS and Sentry policy DB Some developers keep multiple impala repos on their disk. Isolating METASTORE_DB and SENTRY_POLICY_DB may help with switching between those repos without reloading the data. This patch makes those DB names configurable and default to an escaped IMPALA_HOME path. Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Reviewed-on: http://gerrit.cloudera.org:8080/11104 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/create-test-configuration.sh M bin/impala-config.sh M fe/src/test/resources/sentry-site.xml.template 3 files changed, 6 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7383: Configurable HMS and Sentry policy DB
Tianyi Wang has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable HMS and Sentry policy DB .. IMPALA-7383: Configurable HMS and Sentry policy DB Some developers keep multiple impala repos on their disk. Isolating METASTORE_DB and SENTRY_POLICY_DB may help with switching between those repos without reloading the data. This patch makes those DB names configurable and default to an escaped IMPALA_HOME path. Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 --- M bin/create-test-configuration.sh M bin/impala-config.sh M fe/src/test/resources/sentry-site.xml.template 3 files changed, 6 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/11104/6 -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 6 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7383: Configurable HMS and Sentry policy DB
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable HMS and Sentry policy DB .. Patch Set 5: It passes the check now https://jenkins.impala.io/job/gerrit-code-review-checks/255/ -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 5 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 08 Aug 2018 20:59:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7383: Configurable HMS and Sentry policy DB
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable HMS and Sentry policy DB .. Patch Set 4: Thanks. It's strange in that the regular build succeeded. I'm going to run clang-tidy build on my machine. -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Aug 2018 23:43:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ ONLY
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10974 ) Change subject: IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ_ONLY .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dd81100ae73fcabdbfaf679c20cea7dc102cd13 Gerrit-Change-Number: 10974 Gerrit-PatchSet: 9 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 07 Aug 2018 23:35:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7383: Configurable HMS and Sentry policy DB
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable HMS and Sentry policy DB .. Patch Set 4: Not sure why this failed in the clang-tidy check: https://jenkins.impala.io/job/clang-tidy-ub1604/2482/console -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Tue, 07 Aug 2018 22:43:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7383: Configurable METASTORE DB defaulting to escaped IMPALA HOME
Tianyi Wang has restored this change. ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable METASTORE_DB defaulting to escaped IMPALA_HOME .. Restored -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: restore Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7383: Configurable HMS and Sentry policy DB
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable HMS and Sentry policy DB .. Patch Set 3: > Can you do similar logic for the sentry db? > https://github.com/apache/impala/blob/master/bin/create-test-configuration.sh#L109 Done -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Mon, 06 Aug 2018 21:15:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7383: Configurable HMS and Sentry policy DB
Tianyi Wang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable HMS and Sentry policy DB .. IMPALA-7383: Configurable HMS and Sentry policy DB Some developers keep multiple impala repos on their disk. Isolating METASTORE_DB and SENTRY_POLICY_DB may help with switching between those repos without reloading the data. This patch makes those DB names configurable and default to an escaped IMPALA_HOME path. Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 --- M bin/create-test-configuration.sh M bin/impala-config.sh M fe/src/test/resources/sentry-site.xml.template 3 files changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/11104/3 -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7383: Configurable METASTORE DB defaulting to escaped IMPALA HOME
Tianyi Wang has abandoned this change. ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable METASTORE_DB defaulting to escaped IMPALA_HOME .. Abandoned submitted the wrong commit -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7383: Configurable METASTORE DB defaulting to escaped IMPALA HOME
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11104 ) Change subject: IMPALA-7383: Configurable METASTORE_DB defaulting to escaped IMPALA_HOME .. IMPALA-7383: Configurable METASTORE_DB defaulting to escaped IMPALA_HOME Some developers keep multiple impala repos on their disk. Isolating METASTORE_DB may help with switching between those repos without reloading the data. This patch makes METASTORE_DB configurable and default to an escaped IMPALA_HOME path. Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 --- M bin/create-test-configuration.sh M bin/impala-config.sh 2 files changed, 8 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/11104/2 -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7258: Support querying HBase tables in LocalCatalog
Tianyi Wang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/11079 ) Change subject: IMPALA-7258: Support querying HBase tables in LocalCatalog .. IMPALA-7258: Support querying HBase tables in LocalCatalog This is a straightforward port expecting no behavior change. All of the HBase E2E tests pass with this patch. Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java A fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java A fe/src/main/java/org/apache/impala/catalog/local/LocalHbaseTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 20 files changed, 923 insertions(+), 750 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/11079/4 -- To view, visit http://gerrit.cloudera.org:8080/11079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 Gerrit-Change-Number: 11079 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7258: Support querying HBase tables in LocalCatalog
Tianyi Wang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/11079 ) Change subject: IMPALA-7258: Support querying HBase tables in LocalCatalog .. IMPALA-7258: Support querying HBase tables in LocalCatalog This is a straightforward port expecting no behavior change. All of the HBase E2E tests pass with this patch. Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java A fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java A fe/src/main/java/org/apache/impala/catalog/local/LocalHbaseTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 20 files changed, 922 insertions(+), 750 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/11079/3 -- To view, visit http://gerrit.cloudera.org:8080/11079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 Gerrit-Change-Number: 11079 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7308. Support Avro tables in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10970 ) Change subject: IMPALA-7308. Support Avro tables in LocalCatalog .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4b86c8203271b773a711ed77558ec3e3070cb69 Gerrit-Change-Number: 10970 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 00:10:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7384: Move /var/lib/hadoop-hdfs into IMPALA HOME
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11105 ) Change subject: IMPALA-7384: Move /var/lib/hadoop-hdfs into IMPALA_HOME .. Patch Set 1: HDFS requires every parent of this socket path to be either owned by root or non-writable by its group. Absurd! I will put it into $(mktemp) instead. -- To view, visit http://gerrit.cloudera.org:8080/11105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79501fbc762176674bb1c0dde4196a592aee49b2 Gerrit-Change-Number: 11105 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 02 Aug 2018 21:04:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7203. Support UDFs in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11053 ) Change subject: IMPALA-7203. Support UDFs in LocalCatalog .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6130d07b9c641525382a618a9f8da048c7ae75ed Gerrit-Change-Number: 11053 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 02 Aug 2018 01:32:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ ONLY
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10974 ) Change subject: IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ_ONLY .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dd81100ae73fcabdbfaf679c20cea7dc102cd13 Gerrit-Change-Number: 10974 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 02 Aug 2018 01:32:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7384: Move /var/lib/hadoop-hdfs into IMPALA HOME
Tianyi Wang has abandoned this change. ( http://gerrit.cloudera.org:8080/11105 ) Change subject: IMPALA-7384: Move /var/lib/hadoop-hdfs into IMPALA_HOME .. Abandoned Oops, it works on my machine but not on jenkins. -- To view, visit http://gerrit.cloudera.org:8080/11105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I79501fbc762176674bb1c0dde4196a592aee49b2 Gerrit-Change-Number: 11105 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7347: Update tests to accomodate HIVE-18118
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11108 Change subject: IMPALA-7347: Update tests to accomodate HIVE-18118 .. IMPALA-7347: Update tests to accomodate HIVE-18118 HIVE-18118 adds 'numFilesErasureCoded' to table properties. This patch addes it to test_show_create_table to work with the latest Hive. Change-Id: I6aae402dd38374de90b35c32166a9507e6eb29f9 --- M bin/impala-config.sh M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test 2 files changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/11108/1 -- To view, visit http://gerrit.cloudera.org:8080/11108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6aae402dd38374de90b35c32166a9507e6eb29f9 Gerrit-Change-Number: 11108 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7384: Move /var/lib/hadoop-hdfs into IMPALA HOME
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11105 Change subject: IMPALA-7384: Move /var/lib/hadoop-hdfs into IMPALA_HOME .. IMPALA-7384: Move /var/lib/hadoop-hdfs into IMPALA_HOME Currently the mini-cluster uses /var/lib/hadoop-hdfs as HDFS shortcut socket dir. It's a hard-coded path, requires root privilege to create, is a leaky resource, and is one of the many reasons preventing developers to run multiple mini-clusters concurrently. We should put it into the testdata/cluster directory instead. Change-Id: I79501fbc762176674bb1c0dde4196a592aee49b2 --- M bin/bootstrap_system.sh M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl 3 files changed, 3 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/11105/1 -- To view, visit http://gerrit.cloudera.org:8080/11105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I79501fbc762176674bb1c0dde4196a592aee49b2 Gerrit-Change-Number: 11105 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7383: Configurable METASTORE DB defaulting to escaped IMPALA HOME
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11104 Change subject: IMPALA-7383: Configurable METASTORE_DB defaulting to escaped IMPALA_HOME .. IMPALA-7383: Configurable METASTORE_DB defaulting to escaped IMPALA_HOME Some developers keep multiple impala repos on their disk. Isolating METASTORE_DB may help with switching between those repos without reloading the data. This patch makes METASTORE_DB configurable and default to an escaped IMPALA_HOME path. Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/11104/1 -- To view, visit http://gerrit.cloudera.org:8080/11104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I190d657cb95dfdf73ebd05e5dd24ef2a8e3156b8 Gerrit-Change-Number: 11104 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7258: Support querying HBase tables in LocalCatalog
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11079 ) Change subject: IMPALA-7258: Support querying HBase tables in LocalCatalog .. IMPALA-7258: Support querying HBase tables in LocalCatalog This is a straightforward porting expecting no behavior change. All of the HBase E2E tests passes with this patch. Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java A fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java A fe/src/main/java/org/apache/impala/catalog/local/LocalHbaseTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 21 files changed, 924 insertions(+), 739 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/11079/2 -- To view, visit http://gerrit.cloudera.org:8080/11079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 Gerrit-Change-Number: 11079 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ ONLY
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10974 ) Change subject: IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ_ONLY .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10974/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/10974/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@470 PS5, Line 470: / No explicit partition was specified. Need to ensure that write access is available : // to all partitions. Why isn't hasWriteAccessToBaseDir needed here? -- To view, visit http://gerrit.cloudera.org:8080/10974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dd81100ae73fcabdbfaf679c20cea7dc102cd13 Gerrit-Change-Number: 10974 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 30 Jul 2018 23:50:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7307 (part 1). Support stats extrapolation in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10971 ) Change subject: IMPALA-7307 (part 1). Support stats extrapolation in LocalCatalog .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/10971/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/10971/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@197 PS5, Line 197: public static Modifier 'public' is redundant for interface fields Modifier 'static' is redundant for interface fields -- To view, visit http://gerrit.cloudera.org:8080/10971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I479b7f517091dd558768601e1e0704a1902b78a5 Gerrit-Change-Number: 10971 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 27 Jul 2018 23:55:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7308. Support Avro tables in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10970 ) Change subject: IMPALA-7308. Support Avro tables in LocalCatalog .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/10970/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10970/5//COMMIT_MSG@39 PS5, Line 39: if an Avro partition is added to a non-Avro table, and that partition : has a schema that isn't compatible with the table's schema, an error : will occur on read. Can we test this yet? Or is the plan to test existing and local catalog together after IMPALA-7309? http://gerrit.cloudera.org:8080/#/c/10970/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10970/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1558 PS5, Line 1558: ; Why don't we return here? Is there any need to reconcile a inferred schema? http://gerrit.cloudera.org:8080/#/c/10970/5/fe/src/main/java/org/apache/impala/util/AvroSchemaUtils.java File fe/src/main/java/org/apache/impala/util/AvroSchemaUtils.java: http://gerrit.cloudera.org:8080/#/c/10970/5/fe/src/main/java/org/apache/impala/util/AvroSchemaUtils.java@47 PS5, Line 47: HdfsTable AvroSchemaUtils http://gerrit.cloudera.org:8080/#/c/10970/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java File fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java: http://gerrit.cloudera.org:8080/#/c/10970/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@239 PS5, Line 239: // TODO(todd): do we have any tables which are mixed format? alltypesmixedformat: https://github.com/apache/impala/blob/b5608264b4552e44eb73ded1e232a8775c3dba6b/testdata/bin/load-dependent-tables.sql#L62 -- To view, visit http://gerrit.cloudera.org:8080/10970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4b86c8203271b773a711ed77558ec3e3070cb69 Gerrit-Change-Number: 10970 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 27 Jul 2018 23:30:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7203. Support UDFs in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11053 ) Change subject: IMPALA-7203. Support UDFs in LocalCatalog .. Patch Set 1: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/11053/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11053/1//COMMIT_MSG@1 PS1, Line 1: Parent: 0af4661c (IMPALA-7320. Avoid calling getFileStatus() for each partition when table is loaded) In my understanding the transient UDF is a legacy feature and is not widely used. Is that correct? http://gerrit.cloudera.org:8080/#/c/11053/1/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/11053/1/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@166 PS1, Line 166: MetaException No need to declare MetaException because it extends TException. There are many other instances of this redundancy. -- To view, visit http://gerrit.cloudera.org:8080/11053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6130d07b9c641525382a618a9f8da048c7ae75ed Gerrit-Change-Number: 11053 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 27 Jul 2018 18:03:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10912 ) Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 18:59:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() if the partition is removed from HMS. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 49 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/12 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Tue, 24 Jul 2018 17:59:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10912 ) Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. Patch Set 3: (1 comment) LGTM. Will +2 after the small fixes are applied. http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java: http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@140 PS3, Line 140: kuduTableName_ instead of name below > The line of code there is accessing KuduTableDescriptor::table_name() (set You are right. It's only used for logging in the backend. -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 20 Jul 2018 22:55:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10912 ) Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/KuduTable.java File fe/src/main/java/org/apache/impala/catalog/KuduTable.java: http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@258 PS3, Line 258: public static List loadPartitionByParams( > I think eventually they could become default methods in the FeKuduTable int Another option is to put them in a class FeKuduTable.Common. It would be closer to the Java 8 code structure. http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java: http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@140 PS3, Line 140: kuduTableName_ instead of name below > I think that was a bug in the old implementation. The name that shows up in It's eventually used here: https://github.com/apache/impala/blob/master/be/src/exec/kudu-scan-node-base.cc#L103. Why shouldn't it be the table name in Kudu? -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 20 Jul 2018 21:26:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: The problem is that dropPartition() is also called on dirty partitions, in which case we should not change anything in the namenode. Maybe we should further differentiate these two cases, but that's non-trivial code change. -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 19 Jul 2018 22:41:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7252: Backport rate limiting of fadvise calls into toolchain glog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10965 ) Change subject: IMPALA-7252: Backport rate limiting of fadvise calls into toolchain glog .. Patch Set 2: hit IMPALA-7316 -- To view, visit http://gerrit.cloudera.org:8080/10965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41fd855fbf0e9ec58845ac0d2eb96a87b0172152 Gerrit-Change-Number: 10965 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Jul 2018 01:23:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Tianyi Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/10912 )
Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog
..
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@369
PS3, Line 369: for (KuduPartitionParam partitionParam:
table.getPartitionBy()) {
Why putting them here instead of KuduTable?
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/KuduTable.java
File fe/src/main/java/org/apache/impala/catalog/KuduTable.java:
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@258
PS3, Line 258: public static List loadPartitionByParams(
What's the future plan for those static methods shared with local table?
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java:
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@59
PS3, Line 59: public static LocalTable loadFromKudu(LocalDb db, Table
msTable) {
package-private? It's for the local package only.
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@139
PS3, Line 139: {
long line
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@140
PS3, Line 140: kuduTableName_ instead of name below
Why can't we do the same here?
http://gerrit.cloudera.org:8080/#/c/10912/3/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@147
PS3, Line 147: // TODO(todd)
please explain
--
To view, visit http://gerrit.cloudera.org:8080/10912
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd
Gerrit-Change-Number: 10912
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 19 Jul 2018 01:20:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7252: Backport rate limiting of fadvise calls into toolchain glog
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10965 Change subject: IMPALA-7252: Backport rate limiting of fadvise calls into toolchain glog .. IMPALA-7252: Backport rate limiting of fadvise calls into toolchain glog This patch bumps glog version to 0.3.4-p3 to include the patch limiting fadvise calls. Change-Id: I41fd855fbf0e9ec58845ac0d2eb96a87b0172152 --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/10965/1 -- To view, visit http://gerrit.cloudera.org:8080/10965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I41fd855fbf0e9ec58845ac0d2eb96a87b0172152 Gerrit-Change-Number: 10965 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() to fix this bug. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 29 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/6 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 6 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() to fix this bug. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 30 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/5 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 5 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[native-toolchain-CR] IMPALA-7252. Backport rate limiting of fadvise calls for glog
Tianyi Wang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10877 ) Change subject: IMPALA-7252. Backport rate limiting of fadvise calls for glog .. IMPALA-7252. Backport rate limiting of fadvise calls for glog This backports a fix from glog[1] which limits the calling of advise to at most once every 2MB written. Tested that glog builds properly with this fix. [1] https://github.com/google/glog/commit/dacd29679633c9b845708e7015bd2c79367a6ea2 Change-Id: I9f889343ffa458f86c3430b6a8cb2c0d5c14254a Reviewed-on: http://gerrit.cloudera.org:8080/10877 Reviewed-by: Tianyi Wang Tested-by: Tianyi Wang --- M buildall.sh A source/glog/glog-0.3.4-patches/0003-rate-limit-calls-to-posix_fadvise.patch 2 files changed, 110 insertions(+), 1 deletion(-) Approvals: Tianyi Wang: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10877 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9f889343ffa458f86c3430b6a8cb2c0d5c14254a Gerrit-Change-Number: 10877 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tianyi Wang
[native-toolchain-CR] IMPALA-7252. Backport rate limiting of fadvise calls for glog
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10877 ) Change subject: IMPALA-7252. Backport rate limiting of fadvise calls for glog .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10877 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f889343ffa458f86c3430b6a8cb2c0d5c14254a Gerrit-Change-Number: 10877 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 06 Jul 2018 18:27:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() to fix this bug. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 28 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/4 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 4: (6 comments) http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG@7 PS3, Line 7: if a partition is dropped externally > may be say external table drops? Done http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG@9 PS3, Line 9: > Add some context about what happens when it is dropped from Hive. Done http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG@13 PS3, Line 13: table/database, the cache directive won't be removed even if the table > Could you add a test for this in test_hdfs_caching? Done http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1158 PS3, Line 1158:* HdfsPartition that was dropped or null if the partition does not exist. > Update that this drops the cache directive if its cached. Done http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1191 PS3, Line 1191: // If there are multiple partition ids corresponding to a literal, remove > I think this should only run on the Catalog server? I checked the callers of this function and it can only be called from catalogd. http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1193 PS3, Line 1193: if (partitionIds.size() > 1) partitionIds.remove(partitionId); > Do we need to remove a similar check from CatalogOpEx#alterTableDropPartiti Done -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 04 Jul 2018 00:54:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3040: Remove cache directives during background partition dropping
Tianyi Wang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives during background partition dropping .. IMPALA-3040: Remove cache directives during background partition dropping HdfsTable.dropPartition() doesn't uncache the partition right now. If the table is later dropped, the partition won't be uncached either because it has been removed then. This patch removes the cache directive in dropPartition() to fix this bug. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/3 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES
Tianyi Wang has restored this change. ( http://gerrit.cloudera.org:8080/10857 ) Change subject: IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES .. Restored -- To view, visit http://gerrit.cloudera.org:8080/10857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: restore Gerrit-Change-Id: Ife1e791541e3f4fed6bec00945390c7d7681e824 Gerrit-Change-Number: 10857 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES
Tianyi Wang has abandoned this change. ( http://gerrit.cloudera.org:8080/10857 ) Change subject: IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/10857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ife1e791541e3f4fed6bec00945390c7d7681e824 Gerrit-Change-Number: 10857 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10857 Change subject: IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES .. IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES This patch adds a missing "break" statement in a switch statement changed by IMPALA-7102. Also fixes an non-deterministic test case. Change-Id: Ife1e791541e3f4fed6bec00945390c7d7681e824 --- M be/src/service/query-options.cc M testdata/workloads/functional-query/queries/QueryTest/hdfs-erasure-coding.test 2 files changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/10857/1 -- To view, visit http://gerrit.cloudera.org:8080/10857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ife1e791541e3f4fed6bec00945390c7d7681e824 Gerrit-Change-Number: 10857 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directive before dropping a table
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directive before dropping a table .. Patch Set 2: > Do you know what in test_caching_ddl() is calling this drop (drop db > cascade/drop table etc.) ? https://github.com/apache/impala/blob/master/tests/query_test/test_hdfs_caching.py#L207 BTW, The specific case I looked into failed in test_caching_ddl_drop_database. > Also, thinking a bit more about your theory, are you able to reproduce it by > adding Thread.sleep() s in the required places? The tricky part is that reloadTable() calls into HMS to get the table before loading partitions and we need to let that one succeed. Plus load() is called multiple times and the exact timing becomes unclear. I spent some time on it but no luck so far. The fix in this patch should still work. -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 29 Jun 2018 02:14:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directive before dropping a table
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directive before dropping a table .. Patch Set 2: The exception thrown is: E0614 17:03:05.768528 17538 HdfsTable.java:909] Encountered an error loading block metadata for table: cachedb.cach ed_tbl_part Java exception follows: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File does not exist: /test-warehouse/cached b.db/cached_tbl_part/j=2/b14eab6ad3ac682a-1338d1ba_385360643_data.0. at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2157) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2127) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2040) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:583) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(Authorizatio nProviderProxyClientProtocol.java:94) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenod eProtocolServerSideTranslatorPB.java:377) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod (ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1080) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272) at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.impala.catalog.HdfsTable.loadMetadataAndDiskIds(HdfsTable.java:904) at org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(HdfsTable.java:1403) at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1253) at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1199) at org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(CatalogServiceCatalog.java:1460) at org.apache.impala.catalog.TableLoadingMgr.execAsyncRefreshWork(TableLoadingMgr.java:320) at org.apache.impala.catalog.TableLoadingMgr.access$500(TableLoadingMgr.java:48) at org.apache.impala.catalog.TableLoadingMgr$1.call(TableLoadingMgr.java:175) at org.apache.impala.catalog.TableLoadingMgr$1.call(TableLoadingMgr.java:171) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: File does not exist: /test-warehouse/cachedb.db/cached_tbl_part/j=2/b14ea b6ad3ac682a-1338d1ba_385360643_data.0. at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2157) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2127) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2040) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:583) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(Authorizatio nProviderProxyClientProtocol.java:94) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenod eProtocolServerSideTranslatorPB.java:377) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod (ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1080) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274) at java.security.AccessController.doPrivileged(Native Method) at
[Impala-ASF-CR] IMPALA-6816: minimise calls to GetMinSubscriberTopicVersion()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10705 ) Change subject: IMPALA-6816: minimise calls to GetMinSubscriberTopicVersion() .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/10705/4/tests/statestore/test_statestore.py File tests/statestore/test_statestore.py: http://gerrit.cloudera.org:8080/#/c/10705/4/tests/statestore/test_statestore.py@635 PS4, Line 635: assert min_subscriber_topic_version <= min_to_version,\ > Let me go over my reasoning again, I'm not totally confident now. We have 3 It might save people's time by explicitly mentioning that the assertion is not absolute. -- To view, visit http://gerrit.cloudera.org:8080/10705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ee7cb2355ba1049b9081e0df344ac41aa4ebeb1 Gerrit-Change-Number: 10705 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 26 Jun 2018 20:11:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7149: Disable some tests in the EC build
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10804 ) Change subject: IMPALA-7149: Disable some tests in the EC build .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10804 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0975b1a28b318625f853b612bdfea3a8adcd776e Gerrit-Change-Number: 10804 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Mon, 25 Jun 2018 23:19:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directive before dropping a table
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directive before dropping a table .. IMPALA-3040: Remove cache directive before dropping a table One way to hit IMPALA-3040 is to drop a table while the catalog is loading it. The problematic test drops the cached table/database and then checks if the cache directive has been removed. When the table is dropped, the HMS metadata will be removed first. If a concurrent table loading operation is running, it will fail because it cannot find the table in HMS. When the loading procedure throws, the old partition objects have already been cleared from the table catalog object, so the catalog won't remove the cache directives because the metadata has gone. There are several potential solutions: - Lock the tables and the databases before dropping. We don't currently have database lock so this is not trivial. - Fix the table loading procedure so that it loads and replaces existing partitions atomically. - Remove the cache directives first. This patch takes the last approach. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 7 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/2 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-6816: minimise calls to GetMinSubscriberTopicVersion()
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10705 ) Change subject: IMPALA-6816: minimise calls to GetMinSubscriberTopicVersion() .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/10705/4/tests/statestore/test_statestore.py File tests/statestore/test_statestore.py: http://gerrit.cloudera.org:8080/#/c/10705/4/tests/statestore/test_statestore.py@635 PS4, Line 635: assert min_subscriber_topic_version >= min_to_version - 1,\ If the update for the consumer comes late for some reason, won't the to_version of it advance by two at a time and break this assertion? -- To view, visit http://gerrit.cloudera.org:8080/10705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ee7cb2355ba1049b9081e0df344ac41aa4ebeb1 Gerrit-Change-Number: 10705 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 25 Jun 2018 22:31:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6816: minimise calls to GetMinSubscriberTopicVersion()
Tianyi Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/10705 )
Change subject: IMPALA-6816: minimise calls to GetMinSubscriberTopicVersion()
..
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/10705/3/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/10705/3/be/src/scheduling/admission-controller.cc@245
PS3, Line 245: StatestoreSubscriber::UpdateCallback cb = [this](
Nit: auto should be used as the type for lambda. Declaring it as std::function
would invoke std::function's copy-constructor once. (Fun fact: It's not
move-constructed).
http://gerrit.cloudera.org:8080/#/c/10705/3/be/src/statestore/statestore.cc
File be/src/statestore/statestore.cc:
http://gerrit.cloudera.org:8080/#/c/10705/3/be/src/statestore/statestore.cc@735
PS3, Line 735: if (deltas_needing_min_version.size() > 0) {
nit: !deltas_needing_min_version.empty()
http://gerrit.cloudera.org:8080/#/c/10705/3/be/src/statestore/statestore.cc@737
PS3, Line 737: typedef map TopicDeltaMap;
No longer used
http://gerrit.cloudera.org:8080/#/c/10705/3/common/thrift/StatestoreService.thrift
File common/thrift/StatestoreService.thrift:
http://gerrit.cloudera.org:8080/#/c/10705/3/common/thrift/StatestoreService.thrift@147
PS3, Line 147: 3: optional bool populate_min_subscriber_topic_version = false;
Why optional? It's treated as required throughout this patch.
http://gerrit.cloudera.org:8080/#/c/10705/3/tests/statestore/test_statestore.py
File tests/statestore/test_statestore.py:
http://gerrit.cloudera.org:8080/#/c/10705/3/tests/statestore/test_statestore.py@614
PS3, Line 614: update_counts
index into last_to_versions
http://gerrit.cloudera.org:8080/#/c/10705/3/tests/statestore/test_statestore.py@615
PS3, Line 615: update_count
unused
--
To view, visit http://gerrit.cloudera.org:8080/10705
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8ee7cb2355ba1049b9081e0df344ac41aa4ebeb1
Gerrit-Change-Number: 10705
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Sat, 23 Jun 2018 02:03:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3040: Remove cache directive before dropping a table
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10792 Change subject: IMPALA-3040: Remove cache directive before dropping a table .. IMPALA-3040: Remove cache directive before dropping a table One way to hit IMPALA-3040 is to drop a table while the catalog is loading it. If the HDFS files of a partition are removed when the partition is being loaded, the catalog object will be in an inconsistent state and the catalog will fail to recognize some cached partitions and not remove the cache directives. This patch removes the cache directives first to avoid this race condition. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 7 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/1 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 ) Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD .. Patch Set 2: > Patch Set 2: > > We can remerge this, right? Since it's a flaky test fix. I think so. (The fix has been merged) -- To view, visit http://gerrit.cloudera.org:8080/10742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8 Gerrit-Change-Number: 10742 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Jun 2018 22:05:08 + Gerrit-HasComments: No
