[jira] [Created] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
Jim Brennan created HADOOP-17758:
Summary: NPE and excessive warnings after HADOOP-17728
Key: HADOOP-17758
URL: https://issues.apache.org/jira/browse/HADOOP-17758
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 3.4.0
Reporter: Jim Brennan
I'm noticing these warnings and NPE's when just running a simple pi test on a
one node cluster:
{noformat}
2021-06-09 21:51:12,334 WARN
[org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread but
it will continue to run
java.lang.NullPointerException
at
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
at java.lang.Thread.run(Thread.java:748){noformat}
This appears to be due to [HADOOP-17728].
I'm not sure I understand why that change was made? Wasn't it by design that
the remove should wait until something is queued?
[~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17486) Provide fallbacks for callqueue ipc namespace properties
Jim Brennan created HADOOP-17486:
Summary: Provide fallbacks for callqueue ipc namespace properties
Key: HADOOP-17486
URL: https://issues.apache.org/jira/browse/HADOOP-17486
Project: Hadoop Common
Issue Type: Improvement
Components: common
Affects Versions: 3.1.4
Reporter: Jim Brennan
Filing this proposal on behalf of [~daryn], based on comments he made in one of
our internal Jiras.
The following settings are currently specified per port:
{noformat}
/**
* CallQueue related settings. These are not used directly, but rather
* combined with a namespace and port. For instance:
* IPC_NAMESPACE + ".8020." + IPC_CALLQUEUE_IMPL_KEY
*/
public static final String IPC_NAMESPACE = "ipc";
public static final String IPC_CALLQUEUE_IMPL_KEY = "callqueue.impl";
public static final String IPC_SCHEDULER_IMPL_KEY = "scheduler.impl";
public static final String IPC_IDENTITY_PROVIDER_KEY =
"identity-provider.impl";
public static final String IPC_COST_PROVIDER_KEY = "cost-provider.impl";
public static final String IPC_BACKOFF_ENABLE = "backoff.enable";
public static final boolean IPC_BACKOFF_ENABLE_DEFAULT = false;
{noformat}
If one of these properties is not specified for the port, the defaults are
hard-coded.
It would be nice to provide a way to specify a fallback default property that
would be used for all ports. If the property for a specific port is not
defined, the fallback would be used, and if the fallback is not defined it
would use the hard-coded defaults.
We would likely need to make the same change for properties specified by these
classes. For example, properties used in WeightedTimeCostProvider.
The fallback properties could be specified by dropping the port from the
property name. For example, the fallback for {{ipc.8020.cost-provider.impl}}
would be {{ipc.cost-provider.impl}}.
Another option would be to use something more explicit like
{{ipc.default.cost-provider.impl}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations
[
https://issues.apache.org/jira/browse/HADOOP-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan resolved HADOOP-17408.
--
Fix Version/s: 3.4.0
3.3.1
Resolution: Fixed
Thanks for the contribution [~ahussein] and [~daryn]! I have committed this to
trunk and branch-3.3.
The patch does not apply cleanly to branch-3.2 or earlier. Please provide a
patch for 3.2 if desired.
> Optimize NetworkTopology while sorting of block locations
> -
>
> Key: HADOOP-17408
> URL: https://issues.apache.org/jira/browse/HADOOP-17408
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, net
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In {{NetworkTopology}}, I noticed that there are some hanging fruits to
> improve the performance.
> Inside {{sortByDistance}}, collections.shuffle is performed on the list
> before calling {{secondarySort}}.
> {code:java}
> Collections.shuffle(list, r);
> if (secondarySort != null) {
> secondarySort.accept(list);
> }
> {code}
> However, in different call sites, {{collections.shuffle}} is passed as the
> secondarySort to {{sortByDistance}}. This means that the shuffle is executed
> twice on each list.
> Also, logic wise, it is useless to shuffle before applying a tie breaker
> which might make the shuffle work obsolete.
> In addition, [~daryn] reported that:
> * topology is unnecessarily locking/unlocking to calculate the distance for
> every node
> * shuffling uses a seeded Random, instead of ThreadLocalRandom, which is
> heavily synchronized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17417) Reduce UGI overhead in token ops
[
https://issues.apache.org/jira/browse/HADOOP-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan resolved HADOOP-17417.
--
Resolution: Later
This needs to be done as part of a larger feature that we are not ready to put
up yet.
> Reduce UGI overhead in token ops
> -
>
> Key: HADOOP-17417
> URL: https://issues.apache.org/jira/browse/HADOOP-17417
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms, performance, rpc-server, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> {{DelegationTokenIdentifier}} has a {{ugiCache}} but
> AbstractDelegationTokenManager calls a static method {{getRemoteUser()}}
> which would bypass the cache.
> Performance analysis of the KMS revealed the RPC server layer is creating
> many and redundant UGI instances.. UGIs are not cheap to instantiate, require
> synchronization, and waste memory. Reducing instantiations will improve the
> performance of the ipc readers.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17416) Reduce synchronization in the token secret manager
[ https://issues.apache.org/jira/browse/HADOOP-17416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved HADOOP-17416. -- Resolution: Later This change is part of a larger feature that we are not ready to put up yet. > Reduce synchronization in the token secret manager > -- > > Key: HADOOP-17416 > URL: https://issues.apache.org/jira/browse/HADOOP-17416 > Project: Hadoop Common > Issue Type: Improvement > Components: common, performance, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > [~daryn] reported that Reducing synchronization in the ZK secret manager is > complicated by excessive and unnecessary global synchronization in the > AbstractDelegationTokenSecretManager. All RPC services, not just the KMS, > will benefit from the reduced synchronization. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17392) Remote exception messages should not include the exception class
[
https://issues.apache.org/jira/browse/HADOOP-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan resolved HADOOP-17392.
--
Fix Version/s: 3.2.3
3.1.5
3.4.0
3.3.1
Resolution: Fixed
Thanks [~daryn] and [~ahussein]! I have committed this to trunk, branch-3.3,
branch-3.2, and branch-3.1.
> Remote exception messages should not include the exception class
>
>
> Key: HADOOP-17392
> URL: https://issues.apache.org/jira/browse/HADOOP-17392
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> HADOOP-9844 added a change that caused some remote SASL exceptions to
> redundantly include the exception class causing the client to see "{{Class:
> Class: message}}" from an unwrapped RemoteException.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17367) Add InetAddress api to ProxyUsers.authorize
[ https://issues.apache.org/jira/browse/HADOOP-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved HADOOP-17367. -- Fix Version/s: 3.2.3 3.1.5 3.4.0 3.3.1 Resolution: Fixed Thanks [~ahussein] and [~daryn]! I've committed this to trunk, branch-3.3, branch-3.2, and branch-3.1. > Add InetAddress api to ProxyUsers.authorize > --- > > Key: HADOOP-17367 > URL: https://issues.apache.org/jira/browse/HADOOP-17367 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3 > > Time Spent: 1h > Remaining Estimate: 0h > > Improve the ProxyUsers implementation by passing the address of the remote > peer to avoid resolving the hostname. > Similarly, this requires adding InetAddress api to MachineList. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17362) Doing hadoop ls on Har file triggers too many RPC calls
[ https://issues.apache.org/jira/browse/HADOOP-17362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved HADOOP-17362. -- Fix Version/s: 3.2.3 3.1.5 3.4.0 3.3.1 Resolution: Fixed > Doing hadoop ls on Har file triggers too many RPC calls > --- > > Key: HADOOP-17362 > URL: https://issues.apache.org/jira/browse/HADOOP-17362 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > [~daryn] has noticed that Invoking hadoop ls on HAR is taking too much of > time. > The har system has multiple deficiencies that significantly impacted > performance: > # Parsing the master index references ranges within the archive index. Each > range required re-opening the hdfs input stream and seeking to the same > location where it previously stopped. > # Listing a har stats the archive index for every "directory". The per-call > cache used a unique key for each stat, rendering the cache useless and > significantly increasing memory pressure. > # Determining the children of a directory scans the entire archive contents > and filters out children. The cached metadata already stores the exact child > list. > # Globbing a har's contents resulted in unnecessary stats for every leaf path. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17306) RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
[
https://issues.apache.org/jira/browse/HADOOP-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan reopened HADOOP-17306:
--
I have reverted this commit from trunk, branch-3.3, branch-3.2, and
branch-3.2.2.
[~vinayakumarb] please address all of the unit test failures when you resubmit.
I also think we need to review references to modified times in the source base
to be sure we are not breaking things with this change. Yarn Resource
Localization is one area, but there may be others. Timestamps are sometimes
stored in state-stores, so there may be compatibility issues with this change
as well.
> RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
>
>
> Key: HADOOP-17306
> URL: https://issues.apache.org/jira/browse/HADOOP-17306
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RawLocalFileSystem's FileStatus uses {{File.lastModified()}} api from JDK.
> This api looses milliseconds due to JDK bug.
> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8177809]
> This bug fixed in JDK 10 b09 onwards and still exists in JDK 8 which is still
> being used in many productions.
> Apparently, {{Files.getLastModifiedTime()}} from java's nio package returns
> correct time.
> Use {{Files.getLastModifiedTime()}} instead of {{File.lastModified}} as
> workaround.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17342) Creating a token identifier should not do kerberos name resolution
Jim Brennan created HADOOP-17342: Summary: Creating a token identifier should not do kerberos name resolution Key: HADOOP-17342 URL: https://issues.apache.org/jira/browse/HADOOP-17342 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 2.10.1, 3.4.0 Reporter: Jim Brennan This problem was found and fixed internally for us by [~daryn]. Creating a token identifier tries to do auth_to_local short username translation. The authentication process creates a blank token identifier for deserializing the wire format. Attempting to resolve an empty username is useless work. Discovered the issue during fair call queue backoff testing. The readers are unnecessary slowed down by this bug. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17127) Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime
Jim Brennan created HADOOP-17127:
Summary: Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and
processingTime
Key: HADOOP-17127
URL: https://issues.apache.org/jira/browse/HADOOP-17127
Project: Hadoop Common
Issue Type: Improvement
Components: common
Reporter: Jim Brennan
Assignee: Jim Brennan
While making an internal change to use {{TimeUnit.MICROSECONDS}} instead of
{{TimeUnit.MILLISECONDS}} for rpc details, we found that we also had to modify
this code in DecayRpcScheduler.addResponseTime() to initialize {{queueTime}}
and {{processingTime}} with the correct units.
{noformat}
long queueTime = details.get(Timing.QUEUE, TimeUnit.MILLISECONDS);
long processingTime = details.get(Timing.PROCESSING, TimeUnit.MILLISECONDS);
{noformat}
If we change these to use {{RpcMetrics.TIMEUNIT}} it is simpler.
We also found one test case in TestRPC that was assuming the units were
milliseconds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16789) In TestZKFailoverController, restore changes from HADOOP-11149 that were dropped by HDFS-6440
Jim Brennan created HADOOP-16789: Summary: In TestZKFailoverController, restore changes from HADOOP-11149 that were dropped by HDFS-6440 Key: HADOOP-16789 URL: https://issues.apache.org/jira/browse/HADOOP-16789 Project: Hadoop Common Issue Type: Test Reporter: Jim Brennan -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16361) TestSecureLogins#testValidKerberosName fails on branch-2
Jim Brennan created HADOOP-16361:
Summary: TestSecureLogins#testValidKerberosName fails on branch-2
Key: HADOOP-16361
URL: https://issues.apache.org/jira/browse/HADOOP-16361
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.8.5, 2.9.2, 2.10.0
Reporter: Jim Brennan
This test is failing in branch-2.
{noformat}
[ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.917
s <<< FAILURE! - in org.apache.hadoop.registry.secure.TestSecureLogins
[ERROR]
testValidKerberosName(org.apache.hadoop.registry.secure.TestSecureLogins) Time
elapsed: 0.007 s <<< ERROR!
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to zookeeper/localhost
at
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:401)
at
org.apache.hadoop.registry.secure.TestSecureLogins.testValidKerberosName(TestSecureLogins.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15548) Randomize local dirs
Jim Brennan created HADOOP-15548: Summary: Randomize local dirs Key: HADOOP-15548 URL: https://issues.apache.org/jira/browse/HADOOP-15548 Project: Hadoop Common Issue Type: Bug Reporter: Jim Brennan Assignee: Jim Brennan shuffle LOCAL_DIRS, LOG_DIRS and LOCAL_USER_DIRS when launching container. Some applications will process these in exactly the same way in every container (e.g. roundrobin) which can cause disks to get unnecessarily overloaded (e.g. one output file written to first entry specified in the environment variable). There are two paths for local dir allocation, depending on whether the size is unknown or known. The unknown path already uses a random algorithm. The known path initializes with a random starting point, and then goes round-robin after that. When selecting a dir, it increments the last used by one and then checks sequentially until it finds a dir that satisfies the request. Proposal is to increment by a random value of between 1 and num_dirs - 1, and then check sequentially from there. This should result in a more random selection in all cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15357) Configuration.getPropsWithPrefix no longer does variable substitution
Jim Brennan created HADOOP-15357:
Summary: Configuration.getPropsWithPrefix no longer does variable
substitution
Key: HADOOP-15357
URL: https://issues.apache.org/jira/browse/HADOOP-15357
Project: Hadoop Common
Issue Type: Bug
Reporter: Jim Brennan
Before [HADOOP-13556], Configuration.getPropsWithPrefix() used the
Configuration.get() method to get the value of the variables. After
[HADOOP-13556], it now uses props.getProperty().
The difference is that Configuration.get() does deprecation handling and more
importantly variable substitution on the value. So if a property has a
variable specified with ${variable_name}, it will no longer be expanded when
retrieved via getPropsWithPrefix().
Was this change in behavior intentional? I am using this function in the fix
for [MAPREDUCE-7069], but we do want variable expansion to happen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15085) Output streams closed with IOUtils suppressing write errors
[
https://issues.apache.org/jira/browse/HADOOP-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan reopened HADOOP-15085:
--
Reopening so I can put up a patch for branch-2.
> Output streams closed with IOUtils suppressing write errors
> ---
>
> Key: HADOOP-15085
> URL: https://issues.apache.org/jira/browse/HADOOP-15085
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Jim Brennan
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HADOOP-15085.001.patch, HADOOP-15085.002.patch,
> HADOOP-15085.003.patch, HADOOP-15085.004.patch, HADOOP-15085.005.patch
>
>
> There are a few places in hadoop-common that are closing an output stream
> with IOUtils.cleanupWithLogger like this:
> {code}
> try {
> ...write to outStream...
> } finally {
> IOUtils.cleanupWithLogger(LOG, outStream);
> }
> {code}
> This suppresses any IOException that occurs during the close() method which
> could lead to partial/corrupted output without throwing a corresponding
> exception. The code should either use try-with-resources or explicitly close
> the stream within the try block so the exception thrown during close() is
> properly propagated as exceptions during write operations are.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
