[Impala-ASF-CR] IMPALA-12559 (part 2): Fix build issue for different versions of openssl
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21432 ) Change subject: IMPALA-12559 (part 2): Fix build issue for different versions of openssl .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I62b9f0c00f91c2b13be30c415e3f1ebd0e1bd2bc Gerrit-Change-Number: 21432 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 16 May 2024 00:24:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 27: (2 comments) http://gerrit.cloudera.org:8080/#/c/21382/26/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java: http://gerrit.cloudera.org:8080/#/c/21382/26/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@161 PS26, Line 161: (tempJwk > Don't need to make this change. Instead, set createJWKSForWebServer_ as fal Restored. Thanks. http://gerrit.cloudera.org:8080/#/c/21382/26/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@566 PS26, Line 566: fals > Set as false Done -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 27 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 23:46:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#27). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 346 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/27 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 27 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#26). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 347 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/26 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 26 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#25). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 346 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/25 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 25 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] CDPD-68003: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#24). Change subject: CDPD-68003: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. CDPD-68003: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 346 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/24 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 24 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 23: (2 comments) http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util-test.cc File be/src/util/jwt-util-test.cc: http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util-test.cc@322 PS20, Line 322: kid_x5c > I think the value of this constant still needs to be "internal-gateway-jwt. In the latest version, the jwks is updated to also have kid: "kid_x5c". If they do not match then the verify will fail. http://gerrit.cloudera.org:8080/#/c/21382/22/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java: http://gerrit.cloudera.org:8080/#/c/21382/22/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@567 PS22, Line 567: String jwksFilename = > nit: it's unused Removed. Thanks. -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 23 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 17:10:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#23). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 346 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/23 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 23 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 20:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java:
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@573
PS21, Line 573: ate_frequency_s=1 --jwt_allow_w
> If read jwks from local file, then don't need these three parameters, and d
Done
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@585
PS21, Line 585: Client
> Should be 'jwt-cpp.example.localhost'
Done
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@598
PS21, Line 598: verifyJwtAuthMetrics(3, 0);
:
: // Update JWKS in the root directory of Web server.
: createTempJWKSInWebServerRootDir("jwks_x5c_rs256.json");
: // Sleep long enough for coordinator to update JWKS from
Web server.
: Thread.sleep(3000);
: // Authenticate fails due JWT verification failure since
the RS256 public key cannot
: // be found in the JWKS.
: t
> This negative test is dependent on the JWKS change in running time. It shou
Done
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 20
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Tue, 14 May 2024 16:05:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#22). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 347 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/22 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 22 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 21:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util-test.cc
File be/src/util/jwt-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util-test.cc@460
PS20, Line 460: kid_x5c)
> This code is using the string "kid_x5c" but should be using the variable ki
Done
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc@157
PS20, Line 157: if (!values[0].empty()) {
> This statement needs an "else" clause.
Added. Thanks.
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc@158
PS20, Line 158: = values[0];
> enclose this line with { } to follow coding style.
Done
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc@258
PS20, Line 258:
> Line 251 allow different json type, here assume it's string. Please keep co
In Line 251, we assert that the json_value is of type array. and here we are
extracting a string since value is of type cpp_type which is *string or
[]string.
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc@395
PS20, Line 395: algorithm = "rs384";
> If the certificate algorithm is not supported, then the users will get an e
Added. Thanks.
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java:
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@110
PS20, Line 110:
> This function currently use fixed json file. You may add new input paramete
Reading the jwks directly from the local file instead of jwks url now.
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@575
PS20, Line 575: lient transport = new THttpClient("http://local
> add a new parameter to pass jwks_x5c_rs256.json
Reading the jwks from local file instead of the jwks url now.
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@600
PS20, Line 600: transport.setCustomHeaders(headers);
: try {
: openResp = client.OpenSession(openReq);
: } catch (Exception e) {
: verifyJwtAuthMetrics(3, 1);
: assertEquals(e.getMessage(), "HTTP Response code: 401");
: }
: }
:
: /**
:* Asserts that the specified string is present in the
impalad.ERROR file within the
:* specified log directory.
:*
> This function block seems copying from testJwtAuthWithJwksHttpUrl(). In tha
Removed. Now loading the jwks from local file directly.
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 21
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Mon, 13 May 2024 22:23:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#21). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 356 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/21 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 21 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#20). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 354 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/20 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 20 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 19:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/21382/14//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21382/14//COMMIT_MSG@26
PS14, Line 26: sha512WithRSAEncryption => rs512
> In the existing code, we support algorithms PS256, PS384, and PS512 but do
I printed the complete list of signatures but I could not locate the PS algo
class. Here's the list for reference.
Digest: id-rsassa-pkcs1-v1_5-with-sha3-512
Digest: sha512WithRSAEncryption
Digest: sha384WithRSAEncryption
Digest: sha224WithRSAEncryption
Digest: RSA-SHA512/224
Digest: sha512-224
Digest: md5WithRSAEncryption
Digest: sm3
Digest: sha512
Digest: sha384
Digest: sha224
Digest: md4
Digest: blake2b512
Digest: RSA-SHA256
Digest: RSA-SHA224
Digest: ssl3-sha1
Digest: id-rsassa-pkcs1-v1_5-with-sha3-256
Digest: ripemd160
Digest: ssl3-md5
Digest: sha512-224WithRSAEncryption
Digest: sha256
Digest: sha512-256
Digest: RSA-SM3
Digest: shake128
Digest: RSA-RIPEMD160
Digest: whirlpool
Digest: id-rsassa-pkcs1-v1_5-with-sha3-384
Digest: RSA-SHA3-512
Digest: rmd160
Digest: RSA-SHA1
Digest: md5-sha1
Digest: RSA-SHA512/256
Digest: md4WithRSAEncryption
Digest: sm3WithRSAEncryption
Digest: blake2s256
Digest: sha3-256
Digest: RSA-MD5
Digest: RSA-SHA512
Digest: shake256
Digest: ripemd
Digest: RSA-SHA1-2
Digest: RSA-SHA384
Digest: RSA-SHA3-256
Digest: sha512-256WithRSAEncryption
Digest: sha3-224
Digest: id-rsassa-pkcs1-v1_5-with-sha3-224
Digest: sha3-512
Digest: sha3-384
Digest: md5
Digest: ripemd160WithRSA
Digest: sha256WithRSAEncryption
Digest: RSA-SHA3-224
Digest: sha1
Digest: sha1WithRSAEncryption
Digest: RSA-SHA3-384
Digest: RSA-MD4
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util-test.cc
File be/src/util/jwt-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util-test.cc@1245
PS14, Line 1245: auto token =
> Some common code between these new tests. Please refactor it out into a se
Created a static function: CreateJwtX5cToken() with the common code.
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util-test.cc@1264
PS14, Line 1264: << " Actual error: " << status.GetDetail();
> Use "kid_x5c" variable instead of string literal.
Done
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util-test.cc@1268
PS14, Line 1268: // Verify JWT token with x5c certificate.
> Nit: unnecessary
Removed.
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util-test.cc@1300
PS14, Line 1300: status = jwt_helper.Verify(decoded_token.get());
> Use "kid_x5c" variable instead of string literal.
Done
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util-test.cc@1304
PS14, Line 1304: } // namespace impala
> Nit: unnecessary
Removed.
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util.cc@148
PS14, Line 148: _key
> Nit: it's cleaner to not initialize the array members at all. Then, down i
Good point. Thanks.
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util.cc@153
PS14, Line 153: if (NameOfTypeOfJsonValue(json_value) == ARRAY_TYPE) {
> This condition needs to check that the key is "x5c" otherwise any array fie
True, but I think we should generic pick any other array claims as well for
future work or changes in the RFC spec.
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util.cc@375
PS14, Line 375: const char *data = pub_key_str.c_str();
> According to the RFC, if the same data is represented in multiple ways, the
Since the x5c certificate mapping has limited values(4) and also we dont have
coverage for all RSA types(ps256, ps384, ps512 etc), I think we should
prioritize "alg" over the derived value from certificate ?
Also, the certificate validity should be the liability of the ID provider
(aws/azure etc).
Thoughts ?
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util.cc@390
PS14, Line 390: algorithm = "rs256";
> Shouldn't need to declare sslbuf, just directly construct sigalg from OBJ_n
Done
http://gerrit.cloudera.org:8080/#/c/21382/14/be/src/util/jwt-util.cc@393
PS14, Line 393: } else if (sigalg == "sha512WithRSAEncryption") {
> We can drop the parsing of this algorithm since it's not handled in the if
Done
http://gerrit.cloudera.org:8080/#/c/21382/18/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/18/be/src/util/jwt-util.cc@157
PS18, Line 157: if (!values[0].empty())
> Need an "else" clause for this statement. Currently, if the statement eval
Moved the string k, v, values[MAX_X5C_CERTIFICATES] inside the for loop for
better context and scope.
--
To view,
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#19). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 354 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/19 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 19 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#18). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 368 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/18 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 18 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Jason Fehr, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#17). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 368 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/17 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 17 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#16). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java A testdata/jwt/jwks_x5c_rs256.json 4 files changed, 377 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/16 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 16 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#15). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java M testdata/jwt/jwks_rs256.json 4 files changed, 376 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/15 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 15 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#14). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java M testdata/jwt/jwks_rs256.json 4 files changed, 378 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/14 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 14 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG@22 PS6, Line 22: > Please add end-end test case. Added e2e test testJwtAuthWithJwksX5cHttpUrl -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 13 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 08 May 2024 21:37:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#13). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". * added e2e test testJwtAuthWithJwksX5cHttpUrl to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java M testdata/jwt/jwks_rs256.json 4 files changed, 378 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/13 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 13 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc File be/src/util/jwt-util.cc: http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc@403 PS10, Line 403: al > nit: extra spaces Done -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 12 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 08 May 2024 18:12:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#12). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 286 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/12 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 12 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#11). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 267 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/11 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 11 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 10:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@356
PS8, Line 356: if (it_n == kv_map.end() || it_e == kv_map.end()) {
: return Status("'n' and 'e' properties are required");
: } else if (it_n->second.empty() || it_e->second.empty()) {
: return Status("'n' and 'e' properties must be a non-empty
string");
: }
> Are "n" and "e" optional when 'x5c' is provided?
As per the RFC document, they are not optional when x5c is provided.
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@381
PS8, Line 381: O_puts(bio, data);
> add { } around return statement.
good point. The man page for PEM_read_bio_X509() doesn't mention whether it
frees it not and also no examples in openssl docs. I think its good to free it
if PEM_read_bio_X509 errors out.
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@407
PS8, Line 407: pu
> nit: extra indent spaces.
Done
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 10
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Wed, 08 May 2024 15:11:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#10). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 259 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/10 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#9). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 256 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/9 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@226
PS6, Line 226: }
> nit: extra indent spaces
Done
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@256
PS6, Line 256:
\
: for (size_t i = 0; i < json_value.Size() && i <
MAX_X5C_CERTIFICATES; i++) { \
:
> The for loop block should be indented with 4 spaces.
Done. Thx.
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@323
PS6, Line 323: try {
> Is it possible there is no 'k' property when 'x5c' is provided?
Its not specified in the RFC:
https://datatracker.ietf.org/doc/html/rfc7517#section-4.4
It only says that x5c can be optional.
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Tue, 07 May 2024 21:28:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#8). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 276 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/8 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter in JSON Web Keys
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter in JSON Web Keys .. Patch Set 7: (8 comments) http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc File be/src/util/jwt-util.cc: http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@51 PS6, Line 51: x5c certificate > Is there hard limit for number of x5s certificate in RFC? As per RFC, there can be a chain of certificates, however at this point, we do not know how to process mulitiple certificates and/or have the examples of jwks with multiple x5c certificates. We plan to add that in the next patch. http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@150 PS6, Line 150: r()]; > nit: define a constant Done http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@200 PS6, Line 200: ing& nam > this variable is unused Removed. Thanks. http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@209 PS6, Line 209: re > nit: extra indent spaces Fixed. Thanks. http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@245 PS6, Line 245: error message upon > function return Status, not boolean value Updated the comment. http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@256 PS6, Line 256: value[i] = json_value[i].GetString(); \ : } \ : > nit: indent Done http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@323 PS6, Line 323: if (it_x5c != kv_map.end()) > since 'x5c' has priority over 'k', could we check 'x5c' before checking 'k' We check each key regardless to validate them. If "k" is invalid, then the whole jwks is incorrect and we will error out. Only at the time of consuming the values, we make a decision as to which one to prioritize. http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@385 PS6, Line 385: if (it_x5c != kv_map.end()) : pub_key_str = jwt::helper::convert_base64_der_to_pem(it_x5c->second); : else : pub_key_str = pub_key; : > since 'x5c' has priority over 'n' and 'e', could we check 'x5c' first? We check each key regardless to validate them. If "n" and "e" is invalid, then the whole jwks key is incorrect and we will error out. Only at the time of consuming the values, we make a decision as to which one to prioritize. -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 02 May 2024 15:12:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter in JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#7). Change subject: IMPALA-12559: Support x5c Parameter in JSON Web Keys .. IMPALA-12559: Support x5c Parameter in JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the jwks key. The x5c claim can be part of the jwks key either as a string or an array. This patch only supports a single x5c certificate per jwks key. If x5c is present on the jwk, then it is given priority over other mandatory fields like "n", "e", "x", "y" etc (depending on the algorithm type), to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 199 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/7 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter in JSON Web Keys
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#6). Change subject: IMPALA-12559: Support x5c Parameter in JSON Web Keys .. IMPALA-12559: Support x5c Parameter in JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the jwks key. The x5c claim can be part of the jwks key either as a string or an array. This patch only supports a single x5c certificate per jwks key. If x5c is present on the jwk, then it is given priority over other mandatory fields like "n", "e", "x", "y" etc (depending on the algorithm type), to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 202 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/6 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 6 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#5). Change subject: WIP .. WIP IMPALA-12559: Support x5c Parameter in JSON Web Keys This enables the jwt verification using the x5c certificates in the jwks key. The x5c claim is part of the jwks either as a string or an array. This patch only supports one x5c certificate per public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 202 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/5 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#4). Change subject: WIP .. WIP IMPALA-12559: Support x5c Parameter in JSON Web Keys This enables the jwt verification using the x5c certificates in the jwks key. The x5c claim is part of the jwks either as a string or an array. This patch only supports one x5c certificate per public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 191 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/4 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 4 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#3). Change subject: WIP .. WIP IMPALA-12559: Support x5c Parameter in JSON Web Keys This enables the jwt verification using the x5c certificates in the jwks key. The x5c claim is part of the jwks either as a string or an array. This patch only supports one x5c certificate per public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 200 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/3 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 3 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#2). Change subject: WIP .. WIP IMPALA-12559: Support x5c Parameter in JSON Web Keys This enables the jwt verification using the x5c certificates in the jwks key. The x5c claim is part of the jwks either as a string or an array. This patch only supports one x5c certificate per public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 197 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/2 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 2 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter in JSON Web Keys
gaurav singh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21382 Change subject: IMPALA-12559: Support x5c Parameter in JSON Web Keys .. IMPALA-12559: Support x5c Parameter in JSON Web Keys This enables the jwt verification using the x5c certificates in the jwks key. The x5c claim is part of the jwks either as a string or an array. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 199 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/1 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 1 Gerrit-Owner: gaurav singh
[Impala-ASF-CR] IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21187 ) Change subject: IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I83dc32c8d0fec1cdfdfe06e720561b2ae1adf5df Gerrit-Change-Number: 21187 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 25 Mar 2024 23:28:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21187 ) Change subject: IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21187/2/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java: http://gerrit.cloudera.org:8080/#/c/21187/2/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@479 PS2, Line 479: properties.put(key, "**"); Should the number of astreix(s) be equal to the length of the password instead of fixed size ? -- To view, visit http://gerrit.cloudera.org:8080/21187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I83dc32c8d0fec1cdfdfe06e720561b2ae1adf5df Gerrit-Change-Number: 21187 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 25 Mar 2024 21:04:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 15: Updated the planner test and rebased with latest master. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 15 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Fri, 01 Mar 2024 20:04:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#15). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. Binary SCAN predicates involving timestamp literals are pushed down to remote Database. The current logic assumes ISO 8601 (SQL standard) format for timestamp literals - '-mm-dd hh:mm:ss.ms' Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. - Added Planner tests to check predicate pushdown for date/timestamp literals, date/timestamp functions and CASTs Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 454 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/15 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 15 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#14). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. Binary SCAN predicates involving timestamp literals are pushed down to remote Database. The current logic assumes ISO 8601 (SQL standard) format for timestamp literals - '-mm-dd hh:mm:ss.ms' Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. - Added Planner tests to check predicate pushdown for date/timestamp literals, date/timestamp functions and CASTs Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 454 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/14 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 14 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 12: Commit message updated. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 12 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 27 Feb 2024 16:16:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#12). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. Binary SCAN predicates involving timestamp literals are pushed down to remote Database. The current logic assumes ISO 8601 (SQL standard) format for timestamp literals - '-mm-dd hh:mm:ss.ms' Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. - Added Planner tests to check predicate pushdown for date/timestamp literals, date/timestamp functions and CASTs Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 460 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/12 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 12 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#11). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 460 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/11 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 11 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 10: Extended planner test to include cast() and to_timestamp() functions. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 27 Feb 2024 00:04:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#10). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 460 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/10 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#9). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 446 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/9 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 8: Rebased from latest master. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 17:23:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#8). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 461 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/8 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 6: Added planner test for both date and timestamp SCAN predicates. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 6 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 16:35:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#7). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 446 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/7 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG@16 PS5, Line 16: postgre > nit: Postgres Done http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG@17 PS5, Line 17: remote > nit: remote impala Done http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG@19 PS5, Line 19: timestam > nit: timestamp Done http://gerrit.cloudera.org:8080/#/c/21015/5/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21015/5/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@33 PS5, Line 33: import org.apache.impala.analysis.Expr; > nit: keep alphabet order Thanks. Moved it to above the TupleDescriptor. http://gerrit.cloudera.org:8080/#/c/21015/5/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test File testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test: http://gerrit.cloudera.org:8080/#/c/21015/5/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test@275 PS5, Line 275: between > it's better to use count(*) for 'between', like select count(*) from alltyp Done. Thanks. Changed it for mysql and postgres as well. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 6 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 22 Feb 2024 17:25:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#6). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 4 files changed, 432 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/6 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 6 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@148 PS4, Line 148: * Literals will finally be converted to string for pushdown. > It's probably okay to use string representation here as all Literals will a Thanks. Updated comment. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 21 Feb 2024 17:40:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#5). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for impala, mysql and impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timetamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 4 files changed, 450 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/5 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@148 PS4, Line 148: return new TColumnValue().setString_val( > The timestamp format used by most databases is ISO 8601. Most databases should accept multiple timestamp formats. ISO 8601 should be an acceptable format for most (if not the default one). -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 4 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 15 Feb 2024 16:08:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@148 PS4, Line 148: return new TColumnValue().setString_val( > timestamp_val is defined as 16 byte binary array. When calling setTimestamp The timestamp format used by most databases is ISO 8601. >From the postgres documentation: The output format of the date/time types can >be set to one of the four styles ISO 8601, SQL (Ingres), traditional POSTGRES >(Unix date format), or German. The default is the ISO format. ISO ISO 8601, SQL standard 1997-12-17 07:37:16-08 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 4 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 15 Feb 2024 15:48:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/21015/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21015/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-12815: Support timestamp for scan predicates > Much cleaner to create a separate JIRA for timestamp support Created a separate jira: IMPALA-12815 http://gerrit.cloudera.org:8080/#/c/21015/2//COMMIT_MSG@10 PS2, Line 10: The timestamp is of the ISO 8601 format(SQL standard): > We should be more specific here, we're only talking about SCAN predicates h Done. Updated the commit message about timestamp format ISO 8601 (sql standard) http://gerrit.cloudera.org:8080/#/c/21015/2//COMMIT_MSG@14 PS2, Line 14: Testing: > Would be good to be more specific about what kind of tests: e2e, unit, cust specified custom cluster tests. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 4 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 14 Feb 2024 21:40:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#4). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for impala, mysql and impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timetamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 4 files changed, 448 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/4 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 4 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503 PART-2: Support timestamp for predicates for external data source table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12503 PART-2: Support timestamp for predicates for external data source table .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/21015/2/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21015/2/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@145 PS2, Line 145: > rename timestamp_col to timestampVal Thanks. Changed to getStringValue() The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. So far, I checked its followed by most major databases. http://gerrit.cloudera.org:8080/#/c/21015/2/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@150 PS2, Line 150: ((TimestampLiteral) expr).getStringValue()); > No need to call toSql() and extract the string value by parsing it. You cou Done http://gerrit.cloudera.org:8080/#/c/21015/2/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test File testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test: http://gerrit.cloudera.org:8080/#/c/21015/2/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test@278 PS2, Line 278: > remove extra space Done http://gerrit.cloudera.org:8080/#/c/21015/2/testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test File testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test: http://gerrit.cloudera.org:8080/#/c/21015/2/testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test@408 PS2, Line 408: > remove extra space Done http://gerrit.cloudera.org:8080/#/c/21015/2/testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test File testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test: http://gerrit.cloudera.org:8080/#/c/21015/2/testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test@408 PS2, Line 408: > remove extra space Done -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 3 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 14 Feb 2024 19:20:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503 PART-2: Support timestamp for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#3). Change subject: IMPALA-12503 PART-2: Support timestamp for predicates for external data source table .. IMPALA-12503 PART-2: Support timestamp for predicates for external data source table This patch adds support for timestamp as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 4 files changed, 449 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/3 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 3 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12503 PART-2: Support timestamp for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#2). Change subject: IMPALA-12503 PART-2: Support timestamp for predicates for external data source table .. IMPALA-12503 PART-2: Support timestamp for predicates for external data source table This patch adds support for timestamp as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 4 files changed, 368 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/2 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 2 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12503 PART-2: WIP
gaurav singh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21015 Change subject: IMPALA-12503 PART-2: WIP .. IMPALA-12503 PART-2: WIP Support timestamp for predicates for external data source table This patch adds support for datatype timestamp as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test 2 files changed, 133 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/1 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 1 Gerrit-Owner: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#18). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh A testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 13 files changed, 556 insertions(+), 60 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/18 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 18 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/20915 )
Change subject: IMPALA-12503: Support date data type for predicates for
external data source table
..
Patch Set 17:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20915/16/fe/src/test/java/org/apache/impala/service/FrontendTest.java
File fe/src/test/java/org/apache/impala/service/FrontendTest.java:
http://gerrit.cloudera.org:8080/#/c/20915/16/fe/src/test/java/org/apache/impala/service/FrontendTest.java@146
PS16, Line 146: 6
> change 5 to 6 in line 146 since we add one more table 'alltypes_with_date'
Done
http://gerrit.cloudera.org:8080/#/c/20915/16/fe/src/test/java/org/apache/impala/service/FrontendTest.java@156
PS16, Line 156:
resp.rows.get(4).colVals.get(2).string_val.toLowerCase());
> add one assert as:
Done
http://gerrit.cloudera.org:8080/#/c/20915/16/testdata/bin/load-ext-data-sources.sh
File testdata/bin/load-ext-data-sources.sh:
http://gerrit.cloudera.org:8080/#/c/20915/16/testdata/bin/load-ext-data-sources.sh@106
PS16, Line 106: IMPALAD=${IM
> change to:
Done. Thanks.
--
To view, visit http://gerrit.cloudera.org:8080/20915
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4
Gerrit-Change-Number: 20915
Gerrit-PatchSet: 17
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Mon, 05 Feb 2024 16:31:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#17). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh A testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 13 files changed, 556 insertions(+), 60 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/17 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 17 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20915 ) Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/20915/13/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/20915/13/tests/custom_cluster/test_ext_data_sources.py@250 PS13, Line 250: : @pytest.mark.execute_serially : def test_impala_ext_jdbc_tables_predicates(self, vector, unique_database): : """Run tests for external jdbc tables in Impala cluster for new predicates""" : self.run_test_case( > why remove this part? Re-Added -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 16 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Fri, 02 Feb 2024 21:54:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#15). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh A testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 12 files changed, 552 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/15 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 15 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/20915 )
Change subject: IMPALA-12503: Support date data type for predicates for
external data source table
..
Patch Set 13:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20915/13/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
File
java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java:
http://gerrit.cloudera.org:8080/#/c/20915/13/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java@36
PS13, Line 36: dateCol
> nit: rename to dateVal
Done
http://gerrit.cloudera.org:8080/#/c/20915/13/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
File
java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java:
http://gerrit.cloudera.org:8080/#/c/20915/13/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@218
PS13, Line 218: dateCol
> nit: rename to dateVal
Done
http://gerrit.cloudera.org:8080/#/c/20915/13/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
File
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test:
http://gerrit.cloudera.org:8080/#/c/20915/13/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test@49
PS13, Line 49: QUERY
: # Create external JDBC DataSource table
: DROP TABLE IF EXISTS alltypes_jdbc_datasource_2;
: CREATE TABLE alltypes_jdbc_datasource_2 (
: id INT,
: bool_col BOOLEAN,
: tinyint_col TINYINT,
: smallint_col SMALLINT,
: int_col INT,
: bigint_col BIGINT,
: float_col FLOAT,
: double_col DOUBLE,
: date_col DATE,
: string_col STRING,
: timestamp_col TIMESTAMP)
: PRODUCED BY DATA SOURCE TestJdbcDataSource(
: '{"database.type":"IMPALA",
:
"jdbc.url":"jdbc:impala://$INTERNAL_LISTEN_HOST:21050/functional",
: "jdbc.auth":"AuthMech=0",
: "jdbc.driver":"com.cloudera.impala.jdbc.Driver",
:
"driver.url":"$FILESYSTEM_PREFIX/test-warehouse/data-sources/jdbc-drivers/ImpalaJDBC42.jar",
: "dbcp.username":"impala",
: "dbcp.password":"cloudera",
: "table":"alltypes_with_date"}');
: RESULTS
: 'Table has been created.'
> still not removed
Done
--
To view, visit http://gerrit.cloudera.org:8080/20915
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4
Gerrit-Change-Number: 20915
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Fri, 02 Feb 2024 19:12:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#14). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh A testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 11 files changed, 546 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/14 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 14 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/20915 )
Change subject: IMPALA-12503: Support date data type for predicates for
external data source table
..
Patch Set 13:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20915/12/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
File
java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java:
http://gerrit.cloudera.org:8080/#/c/20915/12/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java@36
PS12, Line 36: getDateString(int dateCol);
> rename function and input parameter as getDateString(int dateVal)
Done
http://gerrit.cloudera.org:8080/#/c/20915/12/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
File
java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java:
http://gerrit.cloudera.org:8080/#/c/20915/12/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@218
PS12, Line 218: getDateString(int dateCol)
> rename function and input parameter as getDateString(int dateVal)
Done
http://gerrit.cloudera.org:8080/#/c/20915/12/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
File
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test:
http://gerrit.cloudera.org:8080/#/c/20915/12/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test@49
PS12, Line 49: QUERY
: # Create external JDBC DataSource table
: DROP TABLE IF EXISTS alltypes_jdbc_datasource_2;
: CREATE TABLE alltypes_jdbc_datasource_2 (
: id INT,
: bool_col BOOLEAN,
: tinyint_col TINYINT,
: smallint_col SMALLINT,
: int_col INT,
: bigint_col BIGINT,
: float_col FLOAT,
: double_col DOUBLE,
: date_col DATE,
: string_col STRING,
: timestamp_col TIMESTAMP)
: PRODUCED BY DATA SOURCE TestJdbcDataSource(
: '{"database.type":"IMPALA",
:
"jdbc.url":"jdbc:impala://$INTERNAL_LISTEN_HOST:21050/functional",
: "jdbc.auth":"AuthMech=0",
: "jdbc.driver":"com.cloudera.impala.jdbc.Driver",
:
"driver.url":"$FILESYSTEM_PREFIX/test-warehouse/data-sources/jdbc-drivers/ImpalaJDBC42.jar",
: "dbcp.username":"impala",
: "dbcp.password":"cloudera",
: "table":"alltypes_with_date"}');
: RESULTS
: 'Table has been created.'
:
> remove this table since it is not used
Done
http://gerrit.cloudera.org:8080/#/c/20915/12/testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test
File
testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test:
http://gerrit.cloudera.org:8080/#/c/20915/12/testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test@185
PS12, Line 185: 2009-01-09
> nit: just curious why the values of date columns in MySQL database are diff
My guess would be that the engines have different data structures for data
representation ?
--
To view, visit http://gerrit.cloudera.org:8080/20915
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4
Gerrit-Change-Number: 20915
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Thu, 01 Feb 2024 19:10:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#13). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Added tests for date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh A testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 12 files changed, 505 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/13 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 13 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20915 ) Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/20915/11/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java File java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java: http://gerrit.cloudera.org:8080/#/c/20915/11/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@220 PS11, Line 220: UTC > should be "UTC", Coordinated Universal Time. Done -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 12 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 01 Feb 2024 15:30:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20915 ) Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. Patch Set 12: (2 comments) http://gerrit.cloudera.org:8080/#/c/20915/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20915/11//COMMIT_MSG@15 PS11, Line 15: '=', '>', '<', '>=', '<=' > Please add more tests for operators '>=', '<=', '!=', 'between' Done http://gerrit.cloudera.org:8080/#/c/20915/11//COMMIT_MSG@18 PS11, Line 18: > nit: this was removed Done -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 12 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 01 Feb 2024 15:30:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#12). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Added tests for binary date predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN'. Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh A testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 12 files changed, 463 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/12 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 12 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12642 (Part-2): Fixed unit-test to verify query options for JDBC external table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20978 ) Change subject: IMPALA-12642 (Part-2): Fixed unit-test to verify query options for JDBC external table .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20978 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996c8fac038132f2b132d5e6ac36aca1dff59d72 Gerrit-Change-Number: 20978 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 31 Jan 2024 17:30:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20975 ) Change subject: IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails. .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/20975/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20975/2//COMMIT_MSG@10 PS2, Line 10: xfaile > nit: xfailed Done http://gerrit.cloudera.org:8080/#/c/20975/1/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/20975/1/tests/custom_cluster/test_ext_data_sources.py@172 PS1, Line 172: else: > nit: add a comment here like intermittent issue and need to reinstall mysql Done http://gerrit.cloudera.org:8080/#/c/20975/2/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/20975/2/tests/custom_cluster/test_ext_data_sources.py@173 PS2, Line 173: The mysql docker container creation and mysqld > nit: add more details Done -- To view, visit http://gerrit.cloudera.org:8080/20975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7829aed09d25ff3e636004f3d1f32ecc6f37299 Gerrit-Change-Number: 20975 Gerrit-PatchSet: 3 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 30 Jan 2024 20:03:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails.
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20975 to look at the new patch set (#3). Change subject: IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails. .. IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails. This patch modifies the mysql tests to be marked as xfailed if the mysql environment fails to setup successfully. Change-Id: Ib7829aed09d25ff3e636004f3d1f32ecc6f37299 --- M tests/custom_cluster/test_ext_data_sources.py 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/20975/3 -- To view, visit http://gerrit.cloudera.org:8080/20975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib7829aed09d25ff3e636004f3d1f32ecc6f37299 Gerrit-Change-Number: 20975 Gerrit-PatchSet: 3 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails.
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20975 to look at the new patch set (#2). Change subject: IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails. .. IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails. This patch modifies the mysql tests to be marked as failed if the mysql environment fails to setup successfully. Change-Id: Ib7829aed09d25ff3e636004f3d1f32ecc6f37299 --- M tests/custom_cluster/test_ext_data_sources.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/20975/2 -- To view, visit http://gerrit.cloudera.org:8080/20975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib7829aed09d25ff3e636004f3d1f32ecc6f37299 Gerrit-Change-Number: 20975 Gerrit-PatchSet: 2 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails.
gaurav singh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20975 Change subject: IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails. .. IMPALA-12471 PART-2: skip mysql ext jdbc tests if setup environment fails. This patch modifies the mysql tests to be skipped if the mysql environment fails to setup successfully. Change-Id: Ib7829aed09d25ff3e636004f3d1f32ecc6f37299 --- M tests/custom_cluster/test_ext_data_sources.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/20975/1 -- To view, visit http://gerrit.cloudera.org:8080/20975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib7829aed09d25ff3e636004f3d1f32ecc6f37299 Gerrit-Change-Number: 20975 Gerrit-PatchSet: 1 Gerrit-Owner: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#11). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Manually tested binary date predicates with operators: '=', '>', '<', 'BETWEEN'. - Modified the existing schema and add test cases for date predicate for postgres, mysql and impala databases. alltypes_impala and AllTypesCaseSensitiveNames_impala Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 11 files changed, 220 insertions(+), 81 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/11 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 11 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12378: Add jar file of JdbcDataSource library to classpath
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20971 ) Change subject: IMPALA-12378: Add jar file of JdbcDataSource library to classpath .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0daff8db6231f161ec27b45b51d78e21733d9b1f Gerrit-Change-Number: 20971 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 29 Jan 2024 18:53:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20915 ) Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. Patch Set 10: (3 comments) http://gerrit.cloudera.org:8080/#/c/20915/9/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java File java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java: http://gerrit.cloudera.org:8080/#/c/20915/9/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java@36 PS9, Line 36: dateCol) > nit: rename to dateCol Done http://gerrit.cloudera.org:8080/#/c/20915/9/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java File java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java: http://gerrit.cloudera.org:8080/#/c/20915/9/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@217 PS9, Line 217: dateCol > rename to dateCol Done http://gerrit.cloudera.org:8080/#/c/20915/9/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@219 PS9, Line 219: dateToString = > Naming convention, rename variable dateToString Done -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 29 Jan 2024 18:12:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#10). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Manually tested binary date predicates with operators: '=', '>', '<', 'BETWEEN'. - Modified the existing schema and add test cases for date predicate for postgres, mysql and impala databases. alltypes_impala and AllTypesCaseSensitiveNames_impala Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 11 files changed, 218 insertions(+), 81 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/10 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12378: Add jar file of JdbcDataSource library to classpath
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20971 ) Change subject: IMPALA-12378: Add jar file of JdbcDataSource library to classpath .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20971/1/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java File java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java: http://gerrit.cloudera.org:8080/#/c/20971/1/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@285 PS1, Line 285: FileSystemUtil.copyFileFromUriToLocal(driverUrl); Remove this copy ? -- To view, visit http://gerrit.cloudera.org:8080/20971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0daff8db6231f161ec27b45b51d78e21733d9b1f Gerrit-Change-Number: 20971 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 29 Jan 2024 18:09:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12381: Set JDBC related properties in JDBC data source
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20941 ) Change subject: IMPALA-12381: Set JDBC related properties in JDBC data source .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a0b5d9d7b06825842828c3722c2bcdb4ea8 Gerrit-Change-Number: 20941 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Fri, 26 Jan 2024 21:09:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20915 ) Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/20915/8/testdata/bin/load-ext-data-sources.sh File testdata/bin/load-ext-data-sources.sh: http://gerrit.cloudera.org:8080/#/c/20915/8/testdata/bin/load-ext-data-sources.sh@84 PS8, Line 84: alltypes_with_date > How about rename this table as alltypes_with_date? This is not jdbc data so yes makes sense. http://gerrit.cloudera.org:8080/#/c/20915/8/testdata/bin/load-ext-data-sources.sh@105 PS8, Line 105: > remove this tmp file in the end of script Done -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Fri, 26 Jan 2024 19:37:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#9). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Manually tested binary date predicates with operators: '=', '>', '<', 'BETWEEN'. - Modified the existing schema and add test cases for date predicate for postgres, mysql and impala databases. alltypes_impala and AllTypesCaseSensitiveNames_impala Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 11 files changed, 218 insertions(+), 81 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/9 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/20915 )
Change subject: IMPALA-12503: Support date data type for predicates for
external data source table
..
Patch Set 7:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/bin/setup-impala-env.sh
File testdata/bin/setup-impala-env.sh:
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/bin/setup-impala-env.sh@44
PS7, Line 44: VARCHAR(10)
> change to STRING
Done
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/bin/setup-impala-env.sh@51
PS7, Line 51: DROP TABLE IF EXISTS AllTypesCaseSensitiveNames_impala;
: CREATE TABLE AllTypesCaseSensitiveNames_impala
> Impala is case insensitive. Don't need to create this table
Done
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/bin/setup-impala-env.sh@63
PS7, Line 63: VARCHAR(10)
> change to STRING
Done
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/bin/setup-impala-env.sh@71
PS7, Line 71: # Load data to jdbc table
: cat ${IMPALA_HOME}/testdata/target/AllTypes/* >
/tmp/impala_jdbc_alltypes.csv
:
: hadoop fs -mkdir -p hdfs:///tmp/
: hadoop fs -put /tmp/impala_jdbc_alltypes.csv \
: hdfs:///tmp/impala_jdbc_alltypes >/dev/null 2>&1
:
: impala-shell -i localhost -q "use functional;load data inpath
'hdfs:///tmp/\
: impala_jdbc_alltypes' into table alltypes_impala;"
:
: hadoop fs -put /tmp/impala_jdbc_alltypes.csv
hdfs:///tmp/impala_jdbc_alltypes >\
: /dev/null 2>&1
:
: impala-shell -i localhost -q "use functional;load data inpath
'hdfs:///tmp/\
: impala_jdbc_alltypes' into table
AllTypesCaseSensitiveNames_impala;"
> load data from existing table with 'insert select' as
Done
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
File
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test:
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test@74
PS7, Line 74: AllTypesCaseSensitiveNames_impala
> still use alltypes_impala since we don't need to create AllTypesCaseSensiti
Done
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test@83
PS7, Line 83: where float_col = 0
> it was fixed after changing the way to load data from existing table with '
Done
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test@226
PS7, Line 226: limit 5
> change to 'order by id limit 5' to get result consistently
Done
http://gerrit.cloudera.org:8080/#/c/20915/7/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test@240
PS7, Line 240: limit 5
> change to 'order by id limit 5' to get result consistently
Done
--
To view, visit http://gerrit.cloudera.org:8080/20915
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4
Gerrit-Change-Number: 20915
Gerrit-PatchSet: 7
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Fri, 26 Jan 2024 18:18:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#8). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Manually tested binary date predicates with operators: '=', '>', '<', 'BETWEEN'. - Modified the existing schema and test cases for postgres and mysql databases. - Modified the impala-to-impala test by adding a new tables alltypes_impala and AllTypesCaseSensitiveNames_impala Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh M testdata/bin/setup-mysql-env.sh M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 12 files changed, 218 insertions(+), 83 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/8 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/20915 ) Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. Patch Set 7: (9 comments) http://gerrit.cloudera.org:8080/#/c/20915/6/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java File java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java: http://gerrit.cloudera.org:8080/#/c/20915/6/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@66 PS6, Line 66: long > define as 'static final long' type for this constant. Also move this line a Done http://gerrit.cloudera.org:8080/#/c/20915/6/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@214 PS6, Line 214: mm > nit: -MM-dd to keep consistent with code in the function Done http://gerrit.cloudera.org:8080/#/c/20915/6/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@220 PS6, Line 220: > nit: two more indent space Done http://gerrit.cloudera.org:8080/#/c/20915/6/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java File java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java: http://gerrit.cloudera.org:8080/#/c/20915/6/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java@28 PS6, Line 28: import org.apache.impala.extdatasource.jdbc.dao.DatabaseAccessor; > nit: keep alphabet order Done http://gerrit.cloudera.org:8080/#/c/20915/6/testdata/bin/clean-impala-env.sh File testdata/bin/clean-impala-env.sh: http://gerrit.cloudera.org:8080/#/c/20915/6/testdata/bin/clean-impala-env.sh@1 PS6, Line 1: #!/bin/bash > Don't need this script to clean up if we combine setup-impala-env.sh with t Done http://gerrit.cloudera.org:8080/#/c/20915/6/testdata/bin/setup-impala-env.sh File testdata/bin/setup-impala-env.sh: http://gerrit.cloudera.org:8080/#/c/20915/6/testdata/bin/setup-impala-env.sh@29 PS6, Line 29: # Create local impala tables > You can combine this script with testdata/bin/load-ext-data-sources.sh, whi Done http://gerrit.cloudera.org:8080/#/c/20915/6/testdata/bin/setup-impala-env.sh@32 PS6, Line 32: alltypes_impala > name the table as alltypes_jdbc_impala_datasource Done http://gerrit.cloudera.org:8080/#/c/20915/6/testdata/bin/setup-impala-env.sh@49 PS6, Line 49: cat > /tmp/impala_jdbc_alltypes_with_case_sensitive_names.sql <<__EOT__ : USE FUNCTIONAL; : DROP TABLE IF EXISTS AllTypesCaseSensitiveNames_impala; : CREATE TABLE AllTypesCaseSensitiveNames_impala : ( : id INT, : Bool_colBOOLEAN, : Tinyint_col SMALLINT, : Smallint_colSMALLINT, : Int_col INT, : Bigint_col BIGINT, : Float_col FLOAT, : Double_col DOUBLE, : Date_colDATE, : String_col VARCHAR(10), : Timestamp_col TIMESTAMP : ); : __EOT_ > Impala is case insensitive. All database names and table names are converte Done http://gerrit.cloudera.org:8080/#/c/20915/6/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/20915/6/tests/custom_cluster/test_ext_data_sources.py@209 PS6, Line 209: # create alltypes_impala and load data into it. : script = os.path.join(os.environ['IMPALA_HOME'], 'testdata/bin/setup-impala-env.sh') : run_cmd = [script] : try: : subprocess.check_call(run_cmd, close_fds=True) : except subprocess.CalledProcessError: : assert False, "Failed to setup impala env" : : @classmethod : def _remove_impala_test_env(cls): : # drop table alltypes_impala : script = os.path.join(os.environ['IMPALA_HOME'], 'testdata/bin/clean-impala-env.sh') : run_cmd = [script] : subprocess.check_call(run_cmd, close_fds=True) : try: : subprocess.check_call(run_cmd, close_fds=True) : except subprocess.CalledProcessError: : assert False, "Failed to clean impala env" > Don't need this functions if we combine setup-impala-env.sh with testdata/b Done -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#7). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for datatype date as predicates for external data sources. Testing: - Manually tested binary date predicates with operators: '=', '>', '<', 'BETWEEN'. - Modified the existing schema and test cases for postgres and mysql databases. - Modified the impala-to-impala test by adding a new tables alltypes_impala and AllTypesCaseSensitiveNames_impala Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java A testdata/bin/clean-impala-env.sh M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh A testdata/bin/setup-impala-env.sh M testdata/bin/setup-mysql-env.sh M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 14 files changed, 281 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/7 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12503: Support date data type for predicates for external data source table
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20915 to look at the new patch set (#6). Change subject: IMPALA-12503: Support date data type for predicates for external data source table .. IMPALA-12503: Support date data type for predicates for external data source table This patch adds support for date type for predicates for external data sources. Testing: - Manually tested binary date predicates with operators: '=', '>', '<', 'BETWEEN'. - Modified the existing schema and test cases for postgres and mysql databases. Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/util/QueryConditionUtil.java A testdata/bin/clean-impala-env.sh M testdata/bin/create-ext-data-source-table.sql M testdata/bin/load-ext-data-sources.sh A testdata/bin/setup-impala-env.sh M testdata/bin/setup-mysql-env.sh M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 14 files changed, 226 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/20915/6 -- To view, visit http://gerrit.cloudera.org:8080/20915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf13cbefaad812a0f78755c5791d82b24a3395e4 Gerrit-Change-Number: 20915 Gerrit-PatchSet: 6 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
