[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-28 Thread Daniel Becker (Code Review) 
Daniel Becker has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..

IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds

The above codegen cache tests were introduced by IMPALA-12260. They run
two queries and the first query produces two codegen cache entries. The
tests aim to bring about the following scenario:

1. both codegen cache entries from the first query fit in the cache
AND
2. both entries from the first query are evicted during the second
   query.

The parameters that can be tuned are the following:
1. the size of the codegen cache entries of the first query
2. the size of the codegen cache entries of the second query
3. the size of the codegen cache.

If the parameters are chosen badly or the sizes of the codegen cache
entries change because of other Impala changes (e.g. codegen
optimisations), the conditions may not be satisfied and the tests may
fail like they did now.

This change makes the tests more robust by
 - increasing the cache footprint of the second query (from 487.40 KB to
   663.68 KB)
 - choosing the size of the codegen cache so as to leave as much margin
   on each side as possible. At present
 - the minimal codegen cache size so that both entries from the
   first query fit the cache is around 2.4 MB
 - the maximal cache size so that both entries from the first query
   are evicted during the second query is around 4.1 MB
   Therefore we choose a cache size of 3.25 MB, which lies in the middle.

Experience has shown that this setup is fragile and breaks easily when
new commits are added to Impala. Therefore this change relaxes some of
the assertions in the tests as a temporary measure to prevent build
failures. For this and other reasons IMPALA-12306 was opened to make
these tests more robust.

Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Reviewed-on: http://gerrit.cloudera.org:8080/20224
Reviewed-by: Yida Wu 
Tested-by: Impala Public Jenkins 
---
M tests/custom_cluster/test_codegen_cache.py
1 file changed, 18 insertions(+), 13 deletions(-)

Approvals:
  Yida Wu: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Fri, 28 Jul 2023 03:08:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9536/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 22:57:12 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/13657/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 22:52:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/13656/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 21:25:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 4: Verified-1

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/13654/ : ABORTED


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 20:49:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Yida Wu (Code Review) 
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 4: Code-Review+2

Thanks Daniel for fixing this. To enhance the robustness of the testcase, we 
could follow up on IMPALA-12306.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 18:20:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Daniel Becker (Code Review) 
Daniel Becker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20224/3/tests/custom_cluster/test_codegen_cache.py
File tests/custom_cluster/test_codegen_cache.py:

http://gerrit.cloudera.org:8080/#/c/20224/3/tests/custom_cluster/test_codegen_cache.py@189
PS3, Line 189: cache_entries_in_use > 0
> would it be better to be > 0? Seems no cache entries in use quite an abnorm
You're right, it should have been >0. Corrected.



--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 16:38:55 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Daniel Becker (Code Review) 
Daniel Becker has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..

IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds

The above codegen cache tests were introduced by IMPALA-12260. They run
two queries and the first query produces two codegen cache entries. The
tests aim to bring about the following scenario:

1. both codegen cache entries from the first query fit in the cache
AND
2. both entries from the first query are evicted during the second
   query.

The parameters that can be tuned are the following:
1. the size of the codegen cache entries of the first query
2. the size of the codegen cache entries of the second query
3. the size of the codegen cache.

If the parameters are chosen badly or the sizes of the codegen cache
entries change because of other Impala changes (e.g. codegen
optimisations), the conditions may not be satisfied and the tests may
fail like they did now.

This change makes the tests more robust by
 - increasing the cache footprint of the second query (from 487.40 KB to
   663.68 KB)
 - choosing the size of the codegen cache so as to leave as much margin
   on each side as possible. At present
 - the minimal codegen cache size so that both entries from the
   first query fit the cache is around 2.4 MB
 - the maximal cache size so that both entries from the first query
   are evicted during the second query is around 4.1 MB
   Therefore we choose a cache size of 3.25 MB, which lies in the middle.

Experience has shown that this setup is fragile and breaks easily when
new commits are added to Impala. Therefore this change relaxes some of
the assertions in the tests as a temporary measure to prevent build
failures. For this and other reasons IMPALA-12306 was opened to make
these tests more robust.

Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
---
M tests/custom_cluster/test_codegen_cache.py
1 file changed, 18 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/20224/4
--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Yida Wu (Code Review) 
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 3: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20224/3/tests/custom_cluster/test_codegen_cache.py
File tests/custom_cluster/test_codegen_cache.py:

http://gerrit.cloudera.org:8080/#/c/20224/3/tests/custom_cluster/test_codegen_cache.py@189
PS3, Line 189: cache_entries_in_use >= 0
would it be better to be > 0? Seems no cache entries in use quite an abnormal 
case.



--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 15:23:45 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/13652/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 14:25:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Daniel Becker (Code Review) 
Daniel Becker has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..

IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds

The above codegen cache tests were introduced by IMPALA-12260. They run
two queries and the first query produces two codegen cache entries. The
tests aim to bring about the following scenario:

1. both codegen cache entries from the first query fit in the cache
AND
2. both entries from the first query are evicted during the second
   query.

The parameters that can be tuned are the following:
1. the size of the codegen cache entries of the first query
2. the size of the codegen cache entries of the second query
3. the size of the codegen cache.

If the parameters are chosen badly or the sizes of the codegen cache
entries change because of other Impala changes (e.g. codegen
optimisations), the conditions may not be satisfied and the tests may
fail like they did now.

This change makes the tests more robust by
 - increasing the cache footprint of the second query (from 487.40 KB to
   663.68 KB)
 - choosing the size of the codegen cache so as to leave as much margin
   on each side as possible. At present
 - the minimal codegen cache size so that both entries from the
   first query fit the cache is around 2.4 MB
 - the maximal cache size so that both entries from the first query
   are evicted during the second query is around 4.1 MB
   Therefore we choose a cache size of 3.25 MB, which lies in the middle.

Experience has shown that this setup is fragile and breaks easily when
new commits are added to Impala. Therefore this change relaxes some of
the assertions in the tests as a temporary measure to prevent build
failures. For this and other reasons IMPALA-12306 was opened to make
these tests more robust.

Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
---
M tests/custom_cluster/test_codegen_cache.py
1 file changed, 18 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/20224/3
--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-27 Thread Daniel Becker (Code Review) 
Daniel Becker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2:

I'll do that. First I'll rebase it, then set the codegen cache size to a value 
that works right now with the original constraint and loosen the constraint so 
we can get rid of the build failure.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 27 Jul 2023 12:45:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-26 Thread Yida Wu (Code Review) 
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2:

Maybe the recent llvm compilation changes the size a bit, and we are using a 
quite strict sizing in the testcase, therefore lead to a different result. 
Because we are using the testcase to create a crash, maybe the metrics don't 
have to be that accurate. How about if we use a looser assertion, like,
self.execute_query_expect_success(self.client, q1, exec_options)
assert self.get_metric('impala.codegen-cache.entries-in-use') > 0
assert self.get_metric('impala.codegen-cache.hits') == 0
self.execute_query_expect_success(self.client, q2, exec_options)
assert self.get_metric('impala.codegen-cache.entries-evicted') > 0
assert self.get_metric('impala.codegen-cache.hits') == 0
Then focus on IMPALA-12306 for the next.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Wed, 26 Jul 2023 19:41:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-26 Thread Daniel Becker (Code Review) 
Daniel Becker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2:

Yida Wu yes, that is related. Now I ran it after a rebase and got a different 
kind of error. It seems this test is very sensitive to changes. Maybe we should 
set the cache size adaptively. Or do you have any other idea how to ensure that 
the two cached execution engines from the first query fit in the cache but are 
both ejected during the second query?

If we decide to do it adaptively, I think it can be done but it will take some 
time, maybe we should temporarily disable these tests in the meantime. What do 
you think?


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Wed, 26 Jul 2023 11:55:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-24 Thread Yida Wu (Code Review) 
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2:

Got a failure running the testcase. Could you please check if that is the case?
custom_cluster/test_codegen_cache.py:151: in 
test_codegen_cache_with_asm_module_dir
self._test_codegen_cache_with_symbol_emitter(vector)
custom_cluster/test_codegen_cache.py:188: in 
_test_codegen_cache_with_symbol_emitter
assert self.get_metric('impala.codegen-cache.entries-in-use') == 2
E   assert 1 == 2
E+  where 1 = >('impala.codegen-cache.entries-in-use')
E+where > = 
.get_metric


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Tue, 25 Jul 2023 00:44:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-24 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Mon, 24 Jul 2023 16:21:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-24 Thread Yida Wu (Code Review) 
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Mon, 24 Jul 2023 14:24:53 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-24 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9525/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Mon, 24 Jul 2023 12:14:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-24 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/13623/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Mon, 24 Jul 2023 11:02:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-24 Thread Daniel Becker (Code Review) 
Daniel Becker has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..

IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds

The above codegen cache tests were introduced by IMPALA-12260. They run
two queries and the first query produces two codegen cache entries. The
tests aim to bring about the following scenario:

1. both codegen cache entries from the first query fit in the cache
AND
2. both entries from the first query are evicted during the second
   query.

The parameters that can be tuned are the following:
1. the size of the codegen cache entries of the first query
2. the size of the codegen cache entries of the second query
3. the size of the codegen cache.

If the parameters are chosen badly or the sizes of the codegen cache
entries change because of other Impala changes (e.g. codegen
optimisations), the conditions may not be satisfied and the tests may
fail like they did now.

This change makes the tests more robust by
 - increasing the cache footprint of the second query (from 487.40 KB to
   663.68 KB)
 - choosing the size of the codegen cache so as to leave as much margin
   on each side as possible. At present
 - the minimal codegen cache size so that both entries from the
   first query fit the cache is around 1.16 MB
 - the maximal cache size so that both entries from the first query
   are evicted during the second query is around 2.05 MB
   Therefore we choose a cache size of 1.7 MB, which lies in the middle.

If the tests fail in the future because of the reasons mentioned above,
we can try an adaptive approach where the tests first search for the
appropriate codegen cache size.

Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
---
M tests/custom_cluster/test_codegen_cache.py
1 file changed, 16 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/20224/2
--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-23 Thread Yida Wu (Code Review) 
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 1:

Thanks Daniel for investigating it. So it seems the modified testcase can still 
catch the IMPALA-12260 even not 100%. It will be nice if we can have a better 
solution in this patch. But if the optimization of the testcase is non-trivial, 
it is also good that we file a jira for it and only fix the build failure.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Mon, 24 Jul 2023 03:43:09 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-21 Thread Daniel Becker (Code Review) 
Daniel Becker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 1:

If I compile it in ASAN mode, it's the other way round, it crashes without 
IMPALA-12260 if the first query is
  select int_col from functional_parquet.alltypessmall order by int_col desc 
limit 20;

As dereferencing a NULL pointer is undefined behaviour, it's actually ok, but 
we'll have to find a way to test it well, we can't rely on the crash. I'll 
check if we can use debug actions.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Fri, 21 Jul 2023 16:16:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-20 Thread Daniel Becker (Code Review) 
Daniel Becker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 1:

Thanks Yida. Interestingly it only hits a crash if I include "tinyint_col" in 
the select list of the first query. I'll try to find out why this is before 
merging this.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 20 Jul 2023 14:11:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-19 Thread Yida Wu (Code Review) 
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 1: Code-Review+1

It looks good to me. Just to confirm that whether the changed testcase still 
hits a crash if IMPALA-12260 hasn't been fixed.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Wed, 19 Jul 2023 15:26:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-19 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20224 )

Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/13582/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Wed, 19 Jul 2023 14:59:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12292: TestCodegenCache.{test codegen cache with asm module dir,test codegen cache with perf map} fail in builds

2023-07-19 Thread Daniel Becker (Code Review) 
Daniel Becker has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/20224


Change subject: IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds
..

IMPALA-12292: 
TestCodegenCache.{test_codegen_cache_with_asm_module_dir,test_codegen_cache_with_perf_map}
 fail in builds

The above codegen cache tests were introduced by IMPALA-12260. They run
two queries and the first query produces two codegen cache entries. The
tests aim to bring about the following scenario:

1. both codegen cache entries from the first query fit in the cache
AND
2. both entries from the first query are evicted during the second
   query.

The parameters that can be tuned are the following:
1. the size of the codegen cache entries of the first query
2. the size of the codegen cache entries of the second query
3. the size of the codegen cache.

If the parameters are chosen badly or the sizes of the codegen cache
entries change because of other Impala changes (e.g. codegen
optimisations), the conditions may not be satisfied and the tests may
fail like they did now.

This change makes the tests more robust by
 - decreasing the cache footprint of the first query (though only
   marginally, from 234.71 KB to 233.13 KB)
 - increasing the cache footprint of the second query (from 487.40 KB to
   663.68 KB)
 - choosing the size of the codegen cache so as to leave as much margin
   on each side as possible. At present
 - the minimal codegen cache size so that both entries from the
   first query fit the cache is 1.16 MB
 - the maximal cache size so that both entries from the first query
   are evicted during the second query is 2.05 MB
   Therefore we choose a cache size of 1.7 MB, which lies in the middle.

If the tests fail in the future because of the reasons mentioned above,
we can try an adaptive approach where the tests first search for the
appropriate codegen cache size.

Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
---
M tests/custom_cluster/test_codegen_cache.py
1 file changed, 17 insertions(+), 12 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/20224/1
--
To view, visit http://gerrit.cloudera.org:8080/20224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I15320b8c0d06f4d93927b19731c11bd4e15b3690
Gerrit-Change-Number: 20224
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker