[jira] [Commented] (HADOOP-17130) Configuration.getValByRegex() shouldn't update the results while fetching.
[
https://issues.apache.org/jira/browse/HADOOP-17130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183259#comment-17183259
]
Steve Loughran commented on HADOOP-17130:
-
oops, didn't push up my 3.2 patch. fixed
> Configuration.getValByRegex() shouldn't update the results while fetching.
> --
>
> Key: HADOOP-17130
> URL: https://issues.apache.org/jira/browse/HADOOP-17130
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.1.3
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
> Fix For: 3.2.2, 3.3.1, 3.1.5
>
>
> We have seen this stacktrace while using ABFS file system. After analysing
> the stack trace we can see that getValByRegex() is reading the properties and
> substituting the value in the same call. This may cause the
> ConcurrentModificationException.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.util.ConcurrentModificationException at
> java.util.concurrent.FutureTask.report(FutureTask.java:122) at
> java.util.concurrent.FutureTask.get(FutureTask.java:192) at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1877)
> ... 18 more Caused by: java.util.ConcurrentModificationException at
> java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at
> org.apache.hadoop.conf.Configuration.getValByRegex(Configuration.java:3855)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.validateStorageAccountKeys(AbfsConfiguration.java:689)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.(AbfsConfiguration.java:237)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:154)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:113)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3396) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3456) at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3424) at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:518) at
> org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17130) Configuration.getValByRegex() shouldn't update the results while fetching.
[
https://issues.apache.org/jira/browse/HADOOP-17130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159366#comment-17159366
]
Hudson commented on HADOOP-17130:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18444 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18444/])
HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the (github:
rev b21cb91c7f766b4d2920b893f756a0431f925a18)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
> Configuration.getValByRegex() shouldn't update the results while fetching.
> --
>
> Key: HADOOP-17130
> URL: https://issues.apache.org/jira/browse/HADOOP-17130
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.1.3
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
>
> We have seen this stacktrace while using ABFS file system. After analysing
> the stack trace we can see that getValByRegex() is reading the properties and
> substituting the value in the same call. This may cause the
> ConcurrentModificationException.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.util.ConcurrentModificationException at
> java.util.concurrent.FutureTask.report(FutureTask.java:122) at
> java.util.concurrent.FutureTask.get(FutureTask.java:192) at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1877)
> ... 18 more Caused by: java.util.ConcurrentModificationException at
> java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at
> org.apache.hadoop.conf.Configuration.getValByRegex(Configuration.java:3855)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.validateStorageAccountKeys(AbfsConfiguration.java:689)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.(AbfsConfiguration.java:237)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:154)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:113)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3396) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3456) at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3424) at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:518) at
> org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17130) Configuration.getValByRegex() shouldn't update the results while fetching.
[
https://issues.apache.org/jira/browse/HADOOP-17130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158531#comment-17158531
]
Steve Loughran commented on HADOOP-17130:
-
I believe it is in {{updatePropertiesWIthDeprecatedKeys()}} because that does
update the map
{code}
private void updatePropertiesWIthDeprecatedKeys(
DeprecationContext deprecations, String[] newNames) {
for (String newName : newNames) {
String deprecatedKey =
deprecations.getReverseDeprecatedKeyMap().get(newName);
if (deprecatedKey != null && !getProps().containsKey(newName)) {
String deprecatedValue = getProps().getProperty(deprecatedKey);
if (deprecatedValue != null) {
getProps().setProperty(newName, deprecatedValue); /* HERE */
}
}
}
}
{code}
> Configuration.getValByRegex() shouldn't update the results while fetching.
> --
>
> Key: HADOOP-17130
> URL: https://issues.apache.org/jira/browse/HADOOP-17130
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.1.3
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
>
> We have seen this stacktrace while using ABFS file system. After analysing
> the stack trace we can see that getValByRegex() is reading the properties and
> substituting the value in the same call. This may cause the
> ConcurrentModificationException.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.util.ConcurrentModificationException at
> java.util.concurrent.FutureTask.report(FutureTask.java:122) at
> java.util.concurrent.FutureTask.get(FutureTask.java:192) at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1877)
> ... 18 more Caused by: java.util.ConcurrentModificationException at
> java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at
> org.apache.hadoop.conf.Configuration.getValByRegex(Configuration.java:3855)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.validateStorageAccountKeys(AbfsConfiguration.java:689)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.(AbfsConfiguration.java:237)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:154)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:113)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3396) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3456) at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3424) at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:518) at
> org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
