cxzl25 commented on PR #2077:
URL:
https://github.com/apache/incubator-celeborn/pull/2077#issuecomment-1812315913
> Bump Hadoop client version from 3.2.4 to 3.3.6 will make Celeborn can not
access HDFS 2.x cluster. I think it's ok to bump hadoop version but you'll need
to test the cluster compatibility.
Test file operations such as mkdir, ls, put, get, etc.
| Hadoop Server | Hadoop client 2.x | Hadoop client 3.2.4 | Hadoop client
3.3.6 |
| ------------- | ----------------- | ------------------- |
------------------- |
| 2.2.0 | YES | Write failed | Write failed
|
| 2.3.0 | YES | YES | YES
|
| 2.6.0 | YES | YES | YES
|
| 2.10.0 | YES | YES | YES
|
### Write failed
```java
2023-11-15 17:41:58,502 WARN hdfs.DataStreamer: DataStreamer Exception
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1805)
at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1728)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713)
put: java.lang.NullPointerException
```
`nodeStorageIDs`==null NPE
org.apache.hadoop.hdfs.DataStreamer#createBlockOutputStream
```java
new Sender(out).writeBlock(blockCopy, nodeStorageTypes[0],
accessToken,
dfsClient.clientName, nodes, nodeStorageTypes, null, bcs,
nodes.length, block.getNumBytes(), bytesSent, newGS,
checksum4WriteBlock, cachingStrategy.get(), isLazyPersistFile,
(targetPinnings != null && targetPinnings[0]), targetPinnings,
nodeStorageIDs[0], nodeStorageIDs);
```
Storage ID was introduced in 2.3.0, causing an error when writing to HDFS
2.2.0 if it is greater than or equal to 3.0 (HDFS-9807).
HDFS-2832: Enable support for heterogeneous storages in HDFS
https://issues.apache.org/jira/browse/HDFS-2832
Fix Version/s: 2.3.0
HDFS-9807. Add an optional StorageID to writes.
https://issues.apache.org/jira/browse/HDFS-9807
Fix Version/s:3.0.0-alpha4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
