[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. IMPALA-7111: avoid use of boost::split in CheckPluginEnabled This is an attempt to either avoid the bug or make it easier to diagnose if it reoccurs. My suspicion is that somehow boost::split() is accessing the input string in a non-thread-safe manner, but the implementation is opaque enough that it's not obvious how. Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Reviewed-on: http://gerrit.cloudera.org:8080/10709 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-plugin-text-scanner.cc M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h 4 files changed, 48 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:46:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2679/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:46:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:38:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 3: Does someone want to +2? -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:36:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:38:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Hello Lars Volker, Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10709 to look at the new patch set (#3). Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. IMPALA-7111: avoid use of boost::split in CheckPluginEnabled This is an attempt to either avoid the bug or make it easier to diagnose if it reoccurs. My suspicion is that somehow boost::split() is accessing the input string in a non-thread-safe manner, but the implementation is opaque enough that it's not obvious how. Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 --- M be/src/exec/hdfs-plugin-text-scanner.cc M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h 4 files changed, 48 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/10709/3 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc
File be/src/exec/hdfs-plugin-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc@107
PS1, Line 107: if
(!CommaSeparatedContains(FLAGS_enabled_hdfs_text_scanner_plugins, plugin_name))
{
> Up to you, but since this isn't performance sensitive, you could use split.
I figured it would be good to avoid allocating any intermediate strings in case
this doesn't fix the issue and we need to interpret more ASAN use-after-free
warnings.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@83
PS1, Line 83: // Basic tests with string present.
> Test explicitly that this doesn't whitespace-trim the entries. I don't reme
It doesn't - you have to explicitly specify the characters to split on.
I added a test at the bottom.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@84
PS1, Line 84: EXPECT_TRUE(CommaSeparatedContains("LZO", "LZO"));
> Should we have a test with different length in string and pattern, eg. "PLU
Done
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@90
PS1, Line 90: EXPECT_FALSE(CommaSeparatedContains("", "LZO"));
> Nit: also test "," and ",,"?
Done
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:25:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
..
Patch Set 1:
(2 comments)
Thanks for looking into this. I noticed that we also have gutil/strings/split.h
in our sources. Should we use that for a simpler implementation? I don't feel
strongly about it and am ok with the custom code, too.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@84
PS1, Line 84: EXPECT_TRUE(CommaSeparatedContains("LZO", "LZO"));
Should we have a test with different length in string and pattern, eg.
"PLUGIN2", "LZO" and "A,B,C", "PLUGIN2"?
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@90
PS1, Line 90: EXPECT_FALSE(CommaSeparatedContains("", "LZO"));
Nit: also test "," and ",,"?
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:11:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc
File be/src/exec/hdfs-plugin-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc@107
PS1, Line 107: if
(!CommaSeparatedContains(FLAGS_enabled_hdfs_text_scanner_plugins, plugin_name))
{
Up to you, but since this isn't performance sensitive, you could use split.h
from gutil.
I think the following is similar.
vector x = strings::Split(FLAGS_..., ",");
if (x.find(plugin_name) != ...::npos) { ...
}
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@83
PS1, Line 83: // Basic tests with string present.
Test explicitly that this doesn't whitespace-trim the entries. I don't remember
if boost:split does or doesn't trim.
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:11:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10709 Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. IMPALA-7111: avoid use of boost::split in CheckPluginEnabled This is an attempt to either avoid the bug or make it easier to diagnose if it reoccurs. My suspicion is that somehow boost::split() is accessing the input string in a non-thread-safe manner, but the implementation is opaque enough that it's not obvious how. Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 --- M be/src/exec/hdfs-plugin-text-scanner.cc M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h 4 files changed, 38 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/10709/1 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
