[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809261#comment-17809261
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903380159
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 36s | | trunk passed |
| +1 :green_heart: | compile | 2m 49s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 45s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | compile | 2m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 50s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 188m 55s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 294m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5ba014e5ba65 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 96da2c85a6d951d1897c0d994f5379e2ec1db9e2 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/14/testReport/ |
| Max. process+thread count | 4223 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/14/console |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809236#comment-17809236 ] ASF GitHub Bot commented on HDFS-17293: --- hfutatzhanghb commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903178049 > Committed to trunk. Thanks for your contribution @hfutatzhanghb . @zhangshuyan0 Sir, Thanks a lot for your reviewing. > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809235#comment-17809235 ] ASF GitHub Bot commented on HDFS-17293: --- zhangshuyan0 commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903143954 Committed to trunk. Thanks for your contribution @hfutatzhanghb . > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809234#comment-17809234 ] ASF GitHub Bot commented on HDFS-17293: --- zhangshuyan0 merged PR #6368: URL: https://github.com/apache/hadoop/pull/6368 > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809229#comment-17809229
]
ASF GitHub Bot commented on HDFS-17293:
---
hfutatzhanghb commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903059289
> > Sir, very nice catch. I think below code may resolve the problem you
found. Please take a look when you are free. I will submit another PR to fix it
and Add UT.
> > ```java
> > if (!getStreamer().getAppendChunk()) {
> > int psize = 0;
> > if (blockSize == getStreamer().getBytesCurBlock()) {
> > psize = writePacketSize;
> > } else if (blockSize - getStreamer().getBytesCurBlock() +
PacketHeader.PKT_MAX_HEADER_LEN
> > < writePacketSize ) {
> > psize = (int)(blockSize - getStreamer().getBytesCurBlock()) +
PacketHeader.PKT_MAX_HEADER_LEN;
> > } else {
> > psize = (int) Math
> > .min(blockSize - getStreamer().getBytesCurBlock(),
writePacketSize);
> > }
> > computePacketChunkSize(psize, bytesPerChecksum);
> > }
> > ```
>
> Thank you very much for investing your time in fixing these bugs. The
above fixes did not take `ChecksumSize` into account, and it would be better
for us to discuss this issue in the new PR. Please check if the failed tests
are related to the modification of this PR. Thanks again.
@zhangshuyan0 Sir, Agree with you, let's discuss this issue in the new PR.
The failed tests were all passed in my local.
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809221#comment-17809221
]
ASF GitHub Bot commented on HDFS-17293:
---
zhangshuyan0 commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903015641
> Sir, very nice catch. I think below code may resolve the problem you
found. Please take a look when you are free. I will submit another PR to fix it
and Add UT.
>
> ```java
> if (!getStreamer().getAppendChunk()) {
> int psize = 0;
> if (blockSize == getStreamer().getBytesCurBlock()) {
> psize = writePacketSize;
> } else if (blockSize - getStreamer().getBytesCurBlock() +
PacketHeader.PKT_MAX_HEADER_LEN
> < writePacketSize ) {
> psize = (int)(blockSize - getStreamer().getBytesCurBlock()) +
PacketHeader.PKT_MAX_HEADER_LEN;
> } else {
> psize = (int) Math
> .min(blockSize - getStreamer().getBytesCurBlock(),
writePacketSize);
> }
> computePacketChunkSize(psize, bytesPerChecksum);
> }
> ```
Thank you very much for investing your time in fixing these bugs. The above
fixes did not take `ChecksumSize` into account, and it would be better for us
to discuss this issue in the new PR.
Please check if the failed tests are related to the modification of this PR.
Thanks again.
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808923#comment-17808923
]
ASF GitHub Bot commented on HDFS-17293:
---
hfutatzhanghb commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1902139525
> This PR has corrected the size of the first packet in a new block, which
is great. However, due to the original logical problem in
`adjustChunkBoundary`, the calculation of the size of the last packet in a
block is still problematic, and I think we need a new PR to solve it.
>
>
https://github.com/apache/hadoop/blob/27ecc23ae7c5cafba6a5ea58d4a68d25bd7507dd/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L531-L543
>
>
> Line540, when we pass `blockSize - getStreamer().getBytesCurBlock()` to
`computePacketChunkSize` as the first parameter, `computePacketChunkSize` is
likely to cause the data that could have been sent in one data packet to be
split into two data packets and sent.
Sir, very nice catch. I think below code may resolve the problem you found.
Please take a look~ I will submit another PR to fix it and Add UT.
```java
if (!getStreamer().getAppendChunk()) {
int psize = 0;
if (blockSize == getStreamer().getBytesCurBlock()) {
psize = writePacketSize;
} else if (blockSize - getStreamer().getBytesCurBlock() +
PacketHeader.PKT_MAX_HEADER_LEN
< writePacketSize ) {
psize = (int)(blockSize - getStreamer().getBytesCurBlock()) +
PacketHeader.PKT_MAX_HEADER_LEN;
} else {
psize = (int) Math
.min(blockSize - getStreamer().getBytesCurBlock(),
writePacketSize);
}
computePacketChunkSize(psize, bytesPerChecksum);
}
```
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808912#comment-17808912
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1902116701
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 41s | | trunk passed |
| +1 :green_heart: | compile | 2m 50s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 42s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 2m 50s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 50s | | the patch passed |
| +1 :green_heart: | compile | 2m 40s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 34s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 188m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 293m 47s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestDFSShell |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a5e118489e2b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68292c7eed326eb3d653c66b885502f9befcd371 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808901#comment-17808901
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1902069369
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 49s | | trunk passed |
| +1 :green_heart: | compile | 3m 17s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 6s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 3m 8s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 8s | | the patch passed |
| +1 :green_heart: | compile | 3m 9s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 3m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 36s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/12/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 2 new + 28 unchanged - 0 fixed =
30 total (was 28) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 45s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 193m 27s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 314m 4s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDFSShell |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b9d8820b9876 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / da1ea49921d46c85ca76b8c3c42110fde18125d2 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808749#comment-17808749
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1900802717
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 14s | | trunk passed |
| +1 :green_heart: | compile | 2m 52s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 49s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 2m 43s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 43s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/11/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/11/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 2 new + 28 unchanged - 0 fixed =
30 total (was 28) |
| +1 :green_heart: | mvnsite | 1m 5s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 196m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 301m 3s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestReconstructStripedFile |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 11babc894195 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808642#comment-17808642 ] ASF GitHub Bot commented on HDFS-17293: --- hfutatzhanghb commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1900355060 @zhangshuyan0 Sir, Have modified according to your review opinions. Please take a look when you have free time. Thanks a lot~ > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808496#comment-17808496
]
ASF GitHub Bot commented on HDFS-17293:
---
hfutatzhanghb commented on code in PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#discussion_r1458465700
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:
##
@@ -184,6 +186,40 @@ public void testPreventOverflow() throws IOException,
NoSuchFieldException,
runAdjustChunkBoundary(configuredWritePacketSize, finalWritePacketSize);
}
+ @Test(timeout=6)
+ public void testFirstPacketSizeInNewBlocks() throws IOException {
+final long blockSize = 1L * 1024 * 1024;
+final int numDataNodes = 3;
+final Configuration dfsConf = new Configuration();
+dfsConf.setLong(DFS_BLOCK_SIZE_KEY, blockSize);
+MiniDFSCluster dfsCluster = null;
+dfsCluster = new
MiniDFSCluster.Builder(dfsConf).numDataNodes(numDataNodes).build();
+dfsCluster.waitActive();
+
+DistributedFileSystem fs = dfsCluster.getFileSystem();
+Path fileName = new Path("/testfile.dat");
+FSDataOutputStream fos = fs.create(fileName);
+DataChecksum crc32c =
DataChecksum.newDataChecksum(DataChecksum.Type.CRC32C, 512);
+
+long loop = 0;
+Random r = new Random();
+byte[] buf = new byte[1 * 1024 * 1024];
+r.nextBytes(buf);
+fos.write(buf);
+fos.hflush();
+
+while (loop < 20) {
+ r.nextBytes(buf);
+ fos.write(buf);
+ fos.hflush();
+ loop++;
+ Assert.assertNotEquals(crc32c.getBytesPerChecksum() +
crc32c.getChecksumSize(),
Review Comment:
Sir, thanks for this valuable suggestion. Will fix it soon.
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808488#comment-17808488
]
ASF GitHub Bot commented on HDFS-17293:
---
hfutatzhanghb commented on code in PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#discussion_r1458453315
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:
##
@@ -184,6 +186,40 @@ public void testPreventOverflow() throws IOException,
NoSuchFieldException,
runAdjustChunkBoundary(configuredWritePacketSize, finalWritePacketSize);
}
+ @Test(timeout=6)
+ public void testFirstPacketSizeInNewBlocks() throws IOException {
+final long blockSize = 1L * 1024 * 1024;
+final int numDataNodes = 3;
+final Configuration dfsConf = new Configuration();
+dfsConf.setLong(DFS_BLOCK_SIZE_KEY, blockSize);
+MiniDFSCluster dfsCluster = null;
+dfsCluster = new
MiniDFSCluster.Builder(dfsConf).numDataNodes(numDataNodes).build();
+dfsCluster.waitActive();
+
+DistributedFileSystem fs = dfsCluster.getFileSystem();
+Path fileName = new Path("/testfile.dat");
+FSDataOutputStream fos = fs.create(fileName);
+DataChecksum crc32c =
DataChecksum.newDataChecksum(DataChecksum.Type.CRC32C, 512);
+
+long loop = 0;
+Random r = new Random();
+byte[] buf = new byte[1 * 1024 * 1024];
Review Comment:
Very nice suggestion, Thanks a lot sir. I will fix them laterly
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808450#comment-17808450 ] ASF GitHub Bot commented on HDFS-17293: --- zhangshuyan0 commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1899635293 This PR has corrected the size of the first packet in a new block, which is great. However, due to the original logical problem in `adjustChunkBoundary`, the calculation of the size of the last packet in a block is still problematic, and I think we need a new PR to solve it. https://github.com/apache/hadoop/blob/27ecc23ae7c5cafba6a5ea58d4a68d25bd7507dd/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L531-L543 Line540, when we pass `blockSize - getStreamer().getBytesCurBlock()` to `computePacketChunkSize` as the first parameter, `computePacketChunkSize` is likely to split the data that could have been sent in one packet into two packets for sending. > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808444#comment-17808444
]
ASF GitHub Bot commented on HDFS-17293:
---
zhangshuyan0 commented on code in PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#discussion_r1458246249
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:
##
@@ -184,6 +186,40 @@ public void testPreventOverflow() throws IOException,
NoSuchFieldException,
runAdjustChunkBoundary(configuredWritePacketSize, finalWritePacketSize);
}
+ @Test(timeout=6)
+ public void testFirstPacketSizeInNewBlocks() throws IOException {
+final long blockSize = 1L * 1024 * 1024;
+final int numDataNodes = 3;
+final Configuration dfsConf = new Configuration();
+dfsConf.setLong(DFS_BLOCK_SIZE_KEY, blockSize);
+MiniDFSCluster dfsCluster = null;
+dfsCluster = new
MiniDFSCluster.Builder(dfsConf).numDataNodes(numDataNodes).build();
+dfsCluster.waitActive();
+
+DistributedFileSystem fs = dfsCluster.getFileSystem();
+Path fileName = new Path("/testfile.dat");
+FSDataOutputStream fos = fs.create(fileName);
+DataChecksum crc32c =
DataChecksum.newDataChecksum(DataChecksum.Type.CRC32C, 512);
+
+long loop = 0;
+Random r = new Random();
+byte[] buf = new byte[1 * 1024 * 1024];
+r.nextBytes(buf);
+fos.write(buf);
+fos.hflush();
+
+while (loop < 20) {
+ r.nextBytes(buf);
+ fos.write(buf);
+ fos.hflush();
+ loop++;
+ Assert.assertNotEquals(crc32c.getBytesPerChecksum() +
crc32c.getChecksumSize(),
Review Comment:
It is more appropriate to precisely specify the expected `packetSize` here.
Outside the `while loop`:
```
int chunkSize = crc32c.getBytesPerChecksum() + crc32c.getChecksumSize();
int packetContentSize = (dfsConf.getInt(DFS_CLIENT_WRITE_PACKET_SIZE_KEY,
DFS_CLIENT_WRITE_PACKET_SIZE_DEFAULT) -
PacketHeader.PKT_MAX_HEADER_LEN)/chunkSize*chunkSize;
```
And here:
```
Assert.assertEquals(((DFSOutputStream) fos.getWrappedStream()).packetSize,
packetContentSize);
```
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:
##
@@ -184,6 +186,40 @@ public void testPreventOverflow() throws IOException,
NoSuchFieldException,
runAdjustChunkBoundary(configuredWritePacketSize, finalWritePacketSize);
}
+ @Test(timeout=6)
+ public void testFirstPacketSizeInNewBlocks() throws IOException {
+final long blockSize = 1L * 1024 * 1024;
+final int numDataNodes = 3;
+final Configuration dfsConf = new Configuration();
+dfsConf.setLong(DFS_BLOCK_SIZE_KEY, blockSize);
+MiniDFSCluster dfsCluster = null;
+dfsCluster = new
MiniDFSCluster.Builder(dfsConf).numDataNodes(numDataNodes).build();
+dfsCluster.waitActive();
+
+DistributedFileSystem fs = dfsCluster.getFileSystem();
+Path fileName = new Path("/testfile.dat");
+FSDataOutputStream fos = fs.create(fileName);
+DataChecksum crc32c =
DataChecksum.newDataChecksum(DataChecksum.Type.CRC32C, 512);
+
+long loop = 0;
+Random r = new Random();
+byte[] buf = new byte[1 * 1024 * 1024];
Review Comment:
`byte[] buf = new byte[(int) blockSize];`
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17803904#comment-17803904 ] ASF GitHub Bot commented on HDFS-17293: --- hfutatzhanghb commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1879953128 @Hexiaoqiao @zhangshuyan0 @slfan1989 Sir, have updated this PR, added an unit test. could you please help me review this pr when you are free? Thanks~ > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802606#comment-17802606 ] Shilun Fan commented on HDFS-17293: --- Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a blocker. Retarget 3.5.0. > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801024#comment-17801024
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1871299187
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 24s | | trunk passed |
| +1 :green_heart: | compile | 2m 52s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 45s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 49s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 49s | | the patch passed |
| +1 :green_heart: | compile | 2m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 185m 47s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 290m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 61c8745edeb6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 38d52f6b6cda73a6e6a22832d92d561c5e376df3 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800966#comment-17800966
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1871056405
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 51s | | trunk passed |
| +1 :green_heart: | compile | 3m 14s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 8s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 2m 50s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 50s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 36s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/5/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 27 unchanged - 0 fixed =
28 total (was 27) |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 48s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 187m 59s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 299m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6778fb6d9b24 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bad42b71d5ce0804f6647bfd8b522fa556316323 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/5/testReport/ |
| Max. process+thread count |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800938#comment-17800938 ] ASF GitHub Bot commented on HDFS-17293: --- hfutatzhanghb commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1870932818 @Hexiaoqiao @zhangshuyan0 Hi, sir. Have updated this PR, added an unit test. could you please help me review this pr when you are free? Thanks~ > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800925#comment-17800925
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1870878942
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 17s | | trunk passed |
| +1 :green_heart: | compile | 3m 14s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 0s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 3m 7s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 7s | | the patch passed |
| +1 :green_heart: | compile | 2m 59s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 59s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 43s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/4/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 27 unchanged - 0 fixed =
28 total (was 27) |
| +1 :green_heart: | mvnsite | 1m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 47s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 185m 25s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 302m 58s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d41eab329ead 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800918#comment-17800918
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1870826943
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 38s | | trunk passed |
| +1 :green_heart: | compile | 2m 49s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 2m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 2m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 191m 51s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 297m 44s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e9decbf40c8e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 87ae30afd0c2473239720b86308b8e3c5050320a |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800866#comment-17800866
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1870587809
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 54s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 32s | | trunk passed |
| +1 :green_heart: | compile | 2m 52s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 43s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 2m 49s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 49s | | the patch passed |
| +1 :green_heart: | compile | 2m 43s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 43s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/2/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 37s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/2/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 27 unchanged - 0 fixed =
28 total (was 27) |
| +1 :green_heart: | mvnsite | 1m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 52s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 188m 13s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 299m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 989383600e7e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800196#comment-17800196
]
ASF GitHub Bot commented on HDFS-17293:
---
hfutatzhanghb commented on code in PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#discussion_r1435849521
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:
##
@@ -536,8 +536,13 @@ protected void adjustChunkBoundary() {
}
if (!getStreamer().getAppendChunk()) {
- final int psize = (int) Math
- .min(blockSize - getStreamer().getBytesCurBlock(), writePacketSize);
+ int psize = 0;
+ if (blockSize == getStreamer().getBytesCurBlock()) {
Review Comment:
@Hexiaoqiao Sir, thanks for your replying. I will add unit tests soonly when
i am avaliable.
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800151#comment-17800151
]
ASF GitHub Bot commented on HDFS-17293:
---
Hexiaoqiao commented on code in PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#discussion_r1435791414
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:
##
@@ -536,8 +536,13 @@ protected void adjustChunkBoundary() {
}
if (!getStreamer().getAppendChunk()) {
- final int psize = (int) Math
- .min(blockSize - getStreamer().getBytesCurBlock(), writePacketSize);
+ int psize = 0;
+ if (blockSize == getStreamer().getBytesCurBlock()) {
Review Comment:
Not think carefully, but it seems good improvement for my first feeling. cc
@zhangshuyan0 Would you mind to take another review?
@hfutatzhanghb it is better to add unit tests to cover this case. Thanks.
> First packet data + checksum size will be set to 516 bytes when writing to a
> new block.
> ---
>
> Key: HDFS-17293
> URL: https://issues.apache.org/jira/browse/HDFS-17293
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> First packet size will be set to 516 bytes when writing to a new block.
> In method computePacketChunkSize, the parameters psize and csize would be
> （0, 512）
> when writting to a new block. It should better use writePacketSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[
https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798002#comment-17798002
]
ASF GitHub Bot commented on HDFS-17293:
---
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1859520554
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 44s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 21s | | The patch does not
generate ASF License warnings. |
| | | 94m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8c15c6920909 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9a4a6ba47a04776162c095813f64280fc17842d9 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/1/testReport/ |
| Max. process+thread count | 554 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17797992#comment-17797992 ] ASF GitHub Bot commented on HDFS-17293: --- hfutatzhanghb commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1859452294 @Hexiaoqiao @tomscut @ayushtkn @zhangshuyan0 Sir, could you please help me review this code when you have free time？Thanks ahead. > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17293) First packet data + checksum size will be set to 516 bytes when writing to a new block.
[ https://issues.apache.org/jira/browse/HDFS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17797991#comment-17797991 ] ASF GitHub Bot commented on HDFS-17293: --- hfutatzhanghb opened a new pull request, #6368: URL: https://github.com/apache/hadoop/pull/6368 ### Description of PR First packet size(data length + checksum length) will be set to 516 bytes when writing to a new block. In method computePacketChunkSize, the parameters psize and csize would be （0, 512）when writting to a new block. It should better use writePacketSize to improve the speed of sending data. > First packet data + checksum size will be set to 516 bytes when writing to a > new block. > --- > > Key: HDFS-17293 > URL: https://issues.apache.org/jira/browse/HDFS-17293 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > > First packet size will be set to 516 bytes when writing to a new block. > In method computePacketChunkSize, the parameters psize and csize would be > （0, 512） > when writting to a new block. It should better use writePacketSize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
