[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sun, 26 Jan 2020 18:17:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5463/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sun, 26 Jan 2020 13:29:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5524/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sun, 26 Jan 2020 13:20:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15062 to look at the new patch set (#6). Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC Add test coverage for randomly corrupt ORC files by adding orc in tests of test_scanners_fuzz.py. Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf --- M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/query_test/test_scanners_fuzz.py 3 files changed, 80 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15062/6 -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5461/ -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sun, 26 Jan 2020 11:55:35 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5461/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sun, 26 Jan 2020 07:07:04 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5522/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sun, 26 Jan 2020 03:58:02 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15062 to look at the new patch set (#5). Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC Add test coverage for randomly corrupt ORC files by adding orc in tests of test_scanners_fuzz.py. Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf --- M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/query_test/test_scanners_fuzz.py 3 files changed, 80 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15062/5 -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 4: Code-Review+1 Nice job, thanks for working on this! -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Jan 2020 12:50:20 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5482/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Jan 2020 12:37:21 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 4: (2 comments) Thanks for looking into this! I'm still running the test. No errors found in the past 2 days. http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py File tests/query_test/test_scanners_fuzz.py: http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py@278 PS3, Line 278: > flake8: E261 at least two spaces before inline comment Done http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py@281 PS3, Line 281: 2 > wrong index, should be 2 Oops... -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Jan 2020 11:54:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15062 to look at the new patch set (#4). Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC Add test coverage for randomly corrupt ORC files by adding orc in tests of test_scanners_fuzz.py. Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf --- M tests/common/impala_test_suite.py M tests/query_test/test_scanners_fuzz.py 2 files changed, 50 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15062/4 -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 3: (2 comments) Thanks for applying the changes http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py File tests/query_test/test_scanners_fuzz.py: http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py@281 PS3, Line 281: 0 wrong index, should be 2 http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py@282 PS3, Line 282: data = data[:beg] + data[end:] This might involve a lot of copying. Maybe with open(path, "wb") as f: f.write(data[:beg]) f.write(data[end:]) would be more efficient and would also require less memory. -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Jan 2020 10:29:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5476/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Jan 2020 06:37:01 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py File tests/query_test/test_scanners_fuzz.py: http://gerrit.cloudera.org:8080/#/c/15062/3/tests/query_test/test_scanners_fuzz.py@278 PS3, Line 278: flake8: E261 at least two spaces before inline comment -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Jan 2020 05:51:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15062 to look at the new patch set (#3). Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC Add test coverage for randomly corrupt ORC files by adding orc in tests of test_scanners_fuzz.py. Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf --- M tests/common/impala_test_suite.py M tests/query_test/test_scanners_fuzz.py 2 files changed, 45 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15062/3 -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/15062 )
Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15062/2/tests/query_test/test_scanners_fuzz.py
File tests/query_test/test_scanners_fuzz.py:
http://gerrit.cloudera.org:8080/#/c/15062/2/tests/query_test/test_scanners_fuzz.py@281
PS2, Line 281: LOG.info("corrupt file: Remove range [{0}, {1}) in
{0}".format(beg, end, path))
: data = data[:beg] + data[end:]
:
: with open(path, "wb") as f
> For footer-based file formats it deletes the metadata making the whole file
Good point!
--
To view, visit http://gerrit.cloudera.org:8080/15062
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf
Gerrit-Change-Number: 15062
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 21 Jan 2020 05:51:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/15062 )
Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15062/2/tests/query_test/test_scanners_fuzz.py
File tests/query_test/test_scanners_fuzz.py:
http://gerrit.cloudera.org:8080/#/c/15062/2/tests/query_test/test_scanners_fuzz.py@281
PS2, Line 281: if rng.random() < 0.4:
: truncation = rng.randint(0, len(data))
: LOG.info("corrupt file: Truncate {0} to {1}".format(path,
truncation))
: data = data[:truncation]
For footer-based file formats it deletes the metadata making the whole file
unreadable. Most likely the reader will fail during the magic bytes check, so
it doesn't add much coverage there.
Maybe we could delete parts from the front (but it will also just cause
failures during magic bytes checking). I think the best would be to delete
random parts of the file.
--
To view, visit http://gerrit.cloudera.org:8080/15062
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf
Gerrit-Change-Number: 15062
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 20 Jan 2020 14:22:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5456/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 17 Jan 2020 16:39:32 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15062 to look at the new patch set (#2). Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC We still can't pass the test_scanners_fuzz for orc format. Just share the script so others can run it to find bugs too. Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf --- M tests/common/impala_test_suite.py M tests/query_test/test_scanners_fuzz.py 2 files changed, 43 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15062/2 -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/15062 )
Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15062/1/tests/query_test/test_scanners_fuzz.py
File tests/query_test/test_scanners_fuzz.py:
http://gerrit.cloudera.org:8080/#/c/15062/1/tests/query_test/test_scanners_fuzz.py@130
PS1, Line 130: create_tbl = ("create table {0} stored as {2} as select *
from"
: " functional_{2}.{1}".format(fq_tbl_name,
orig_tbl_name,
:table_format.file_format))
> This tries to write the table with Impala, right? In that case it shouldn't
Oops... Just find this test only allows compression_codec == 'none' so only
'parquet/none' and 'orc/none' will be accepted. But we don't have 'orc/none'...
Will revert this change.
--
To view, visit http://gerrit.cloudera.org:8080/15062
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf
Gerrit-Change-Number: 15062
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Fri, 17 Jan 2020 15:13:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/15062 )
Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15062/1/tests/query_test/test_scanners_fuzz.py
File tests/query_test/test_scanners_fuzz.py:
http://gerrit.cloudera.org:8080/#/c/15062/1/tests/query_test/test_scanners_fuzz.py@130
PS1, Line 130: create_tbl = ("create table {0} stored as {2} as select *
from"
: " functional_{2}.{1}".format(fq_tbl_name,
orig_tbl_name,
:table_format.file_format))
This tries to write the table with Impala, right? In that case it shouldn't
work with Orc.
--
To view, visit http://gerrit.cloudera.org:8080/15062
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf
Gerrit-Change-Number: 15062
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 17 Jan 2020 15:00:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15062 ) Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5453/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 17 Jan 2020 14:11:29 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6772: Enable test scanners fuzz for ORC
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15062 Change subject: [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC .. [WIP] IMPALA-6772: Enable test_scanners_fuzz for ORC We still can't pass the test_scanners_fuzz for orc format. Just share the script so others can run it to find bugs too. Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf --- M tests/query_test/test_scanners_fuzz.py 1 file changed, 11 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15062/1 -- To view, visit http://gerrit.cloudera.org:8080/15062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3233e5d9f555029d954b5ddd5858ea194afc06bf Gerrit-Change-Number: 15062 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
