subject:"\[jira\] \[Work logged\] \(HADOOP\-17895\) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3"

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-13 Thread ASF GitHub Bot (Jira)

[
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=650279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650279
]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 14/Sep/21 00:10
Start Date: 14/Sep/21 00:10
Worklog Time Spent: 10m
Work Description: cnauroth commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-918677709

@majdyz , thanks for refining the test case. Thanks also to @steveloughran
and @ayushtkn for their testing.

I can see the repro now. Interestingly, even though it repros on
branch-3.3, it does not repro on branch-3.2. I suspect though that's not
because branch-3.2 has correct Unicode handling, but rather that both the
`mkdir` and `chmod` steps of this logic are doing the wrong thing, but they are
at least in agreement on doing the same wrong thing. It looks like the end
result on branch-3.2 is a successful call, but the new directory doesn't have
the correct name.

Here is the reason it's happening. The call performs the `mkdir` using Java
classes and then performs the `chmod` through native code. I traced through
the Java and native layers. I confirmed that the string passes correctly
through all Java frames, and then inside the native layer it gets garbled.
This is because the string data is fetched through the JNI `GetStringUTFChars`
function, which is documented to return the JVM's internal "modified UTF-8"
representation. The `chmod` call would work fine with true UTF-8, but there
are edge cases where the modified UTF-8 would differ from the standard and not
work correctly.

The JavaDocs of
[`DataInputFormat`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/DataInput.html)
discuss the differences between standard UTF-8 and modified UTF-8. This is
the key point relevant to this bug:

> Only the 1-byte, 2-byte, and 3-byte formats are used.

The character in your test case is an emoji that has a 4-byte standard UTF-8
representation, but modified UTF-8 chooses to represent it as 2 surrogate pairs
of 3 bytes each. That encoding won't be compatible with the C APIs that we're
calling.

This problem is likely not unique to the `chmod` call. We use
`GetStringUTFChars` in multiple places.

Interestingly, it's possible that this bug doesn't repro at all on Windows,
where the JNI code path uses `GetStringChars` instead to fetch the UTF-16
representation and convert to Windows wide-character string types. I no longer
have easy access to a Windows test environment to confirm though.

Now what might we do about this? Some ideas:
1. Explore a possible migration from native `chmod` to the new Java NIO File
APIs, e.g.
[`Files#setPosixFilePermissions`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/file/Files.html#setPosixFilePermissions(java.nio.file.Path,java.util.Set)).
These native code paths were introduced before the JDK introduced APIs for
permission changes.
2. Convert the code to use `GetStringChars` to fetch the UTF-16 bytes and
pass them back to the JDK encoding conversion methods. That would give us a
way to get to true UTF-8. This introduces a performance risk from extra buffer
copying and extra transitions over the Java-JNI boundary. It's also probably a
more confusing call flow.
3. Use `GetStringChars` to get UTF-16, but do the conversion to UTF-8
directly within the JNI layer using something like
[`iconv`](https://linux.die.net/man/3/iconv). That would avoid some of the
weirdness and performance penalties of transitioning back over the Java
boundary.
4. Change the code to convert to a UTF-8 `byte[]` at the Java layer and pass
that as-is to JNI. (Stay away from `jstring` entirely.) This would re-raise
the debate of whether it's considered a backward-incompatible change if the
interface between the Java layer and the JNI layer changes.

It's probably natural for everyone to ask "isn't there some simpler way to
get true UTF-8 out of a `jstring`?" I'm pretty sure there is no convenient
function for that, but I'd love if someone proved me wrong. All of the
solutions I can think of are fairly large in scope. Then, there is testing
effort too of course.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Issue Time Tracking
---

Worklog Id: (was: 650279)
Time Spent: 2h (was: 1h 50m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
>

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=650159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650159
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 13/Sep/21 17:37
Start Date: 13/Sep/21 17:37
Worklog Time Spent: 10m 
  Work Description: majdyz edited a comment on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-918209856


   @cnauroth @ayushtkn Thank you for trying to reproduce the issue on your end, 
I have now updated the current test to also fail in the provided docker image 
started with `./start-build-env.sh`.
   I was using my local MacOS machine and didn't provide the randomized folder 
which makes the test runs smoothly in the second run. The CI is also breaking 
now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 650159)
Time Spent: 1h 50m  (was: 1h 40m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> The failure was happening when doing the native `chmod` command to the file 
> (the `mkdir` command itself is working).
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=650102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650102
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 13/Sep/21 16:06
Start Date: 13/Sep/21 16:06
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-918344994


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 56s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ branch-3.3.1 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  32m 55s |  |  branch-3.3.1 passed  |
   | +1 :green_heart: |  compile  |  18m 10s |  |  branch-3.3.1 passed  |
   | +1 :green_heart: |  checkstyle  |   0m 48s |  |  branch-3.3.1 passed  |
   | +1 :green_heart: |  mvnsite  |   1m 31s |  |  branch-3.3.1 passed  |
   | +1 :green_heart: |  javadoc  |   1m 32s |  |  branch-3.3.1 passed  |
   | +1 :green_heart: |  spotbugs  |   2m 23s |  |  branch-3.3.1 passed  |
   | +1 :green_heart: |  shadedclient  |  20m 38s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 57s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 58s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |  17m 59s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 47s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 32s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   1m 30s |  |  the patch passed  |
   | +1 :green_heart: |  spotbugs  |   2m 44s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  20m 27s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |  17m  7s | 
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3391/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 47s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 141m 11s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir 
|
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3391/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3391 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux de5d180a444f 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | branch-3.3.1 / e0161043c1cdb86ea1f654c3e56c593b85cf6593 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3391/2/testReport/ |
   | Max. process+thread count | 1236 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3391/2/console |
   | versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 650102)
Time Spent: 1h 40m  (was: 1.5h)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
>

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=650026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650026
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 13/Sep/21 13:48
Start Date: 13/Sep/21 13:48
Worklog Time Spent: 10m 
  Work Description: majdyz commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-918209856


   @cnauroth @ayushtkn Thank you for trying to reproduce the issue on your end, 
I have now updated the current test to also fail in the provided docker image 
started with `./start-build-env.sh`.
   I was using my local MacOS machine and didn't provide the randomized folder 
which makes the test runs smoothly in the second run.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 650026)
Time Spent: 1.5h  (was: 1h 20m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> The failure was happening when doing the native `chmod` command to the file 
> (the `mkdir` command itself is working).
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=649626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649626
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 11/Sep/21 20:12
Start Date: 11/Sep/21 20:12
Worklog Time Spent: 10m 
  Work Description: ayushtkn commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-917471563


   Not reproducing for me either. 
   FYI, the jenkins run the test with -Pnative flag and it passed:
   
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3391/1/testReport/org.apache.hadoop.fs.contract.rawlocal/TestRawlocalContractMkdir/testMkDirRmDirUnicode/
   ```
Running unit tests
   13:34:45  

   13:34:45  

   13:34:45  
   13:34:45  
   13:34:48  cd 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3391/src/hadoop-common-project/hadoop-common
   13:34:48  /usr/bin/mvn --batch-mode 
-Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3391/yetus-m2/hadoop-branch-3.3.1-patch-1
 -Ptest-patch -Pparallel-tests -P!shelltest -Pnative -Drequire.fuse 
-Drequire.openssl -Drequire.snappy -Drequire.valgrind -Drequire.zstd 
-Drequire.test.libhadoop -Pyarn-ui clean test -fae > 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3391/out/patch-unit-hadoop-common-project_hadoop-common.txt
 2>&1
   13:52:58  Elapsed:  18m 25s
   13:52:58  
   13:52:58  hadoop-common in the patch passed.
   ```
   https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3391/1/console
   
   What OS are you using?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 649626)
Time Spent: 1h 20m  (was: 1h 10m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> The failure was happening when doing the native `chmod` command to the file 
> (the `mkdir` command itself is working).
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=649333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649333
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 10/Sep/21 17:23
Start Date: 10/Sep/21 17:23
Worklog Time Spent: 10m 
  Work Description: cnauroth commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-917076931


   @majdyz , yes, I'm running your branch with `mvn -Pnative 
-Dtest=TestRawlocalContractMkdir` inside the development Docker image started 
with `./start-build-env.sh`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 649333)
Time Spent: 1h 10m  (was: 1h)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> The failure was happening when doing the native `chmod` command to the file 
> (the `mkdir` command itself is working).
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=649133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649133
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 10/Sep/21 10:14
Start Date: 10/Sep/21 10:14
Worklog Time Spent: 10m 
  Work Description: majdyz commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-916792991


   Just to confirm, have you already used `-Pnative` for that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 649133)
Time Spent: 1h  (was: 50m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> The failure was happening when doing the native `chmod` command to the file 
> (the `mkdir` command itself is working).
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=648911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648911
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 09/Sep/21 22:57
Start Date: 09/Sep/21 22:57
Worklog Time Spent: 10m 
  Work Description: cnauroth commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-916495654


   ```
   Path dir = path("testMkDirRmDirUnicode/dir-\\U0001f63b");
   ```
   
   Something looks off with the test.  That looks like `\\U`, so the 
double-slash escapes to just one slash, followed by the rest of the string 
literal.  A Unicode character escape needs `\u` (single slash followed by 
lower-case u).
   
   Still, I tried changing the test locally to use a Unicode escape, and I 
still couldn't reproduce a failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 648911)
Time Spent: 50m  (was: 40m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> The failure was happening when doing the native `chmod` command to the file 
> (the `mkdir` command itself is working).
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=647395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647395
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 07/Sep/21 14:32
Start Date: 07/Sep/21 14:32
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-914357168


   the core mkdir() works, it's the setPermission call after that is failing...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 647395)
Time Spent: 40m  (was: 0.5h)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=647369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647369
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 07/Sep/21 13:33
Start Date: 07/Sep/21 13:33
Worklog Time Spent: 10m 
  Work Description: majdyz commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-914310362


   Thank you for the response.
   The test setup doesn't seem to be using native is there a way to enable it 
through the code? 
   I'm running it with `-Pnative` profile. The failing was indeed in the chmod 
command, what do you mean by the double invocation? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 647369)
Time Spent: 0.5h  (was: 20m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=647313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647313
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 07/Sep/21 11:30
Start Date: 07/Sep/21 11:30
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-914225427


   (or: test setup isn't using nativeio).
   
   Looking @ the code, it's not native mkdir() failing, it's chmod...which has 
recalculated pathToFile(). Maybe double invocation is the problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 647313)
Time Spent: 20m  (was: 10m)

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

2021-09-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=647311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647311
 ]

ASF GitHub Bot logged work on HADOOP-17895:
---

Author: ASF GitHub Bot
Created on: 07/Sep/21 11:26
Start Date: 07/Sep/21 11:26
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-914223315


   hmm. test works.
   
   What FS are you seeing the problem with? macos, extfs, ...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 647311)
Remaining Estimate: 0h
Time Spent: 10m

> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
> 
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with 
> NativeIO enabled.
> Stacktrace:
> {{ENOENT: No such file or directory  ENOENT: No such file or directory at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
>  at 
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>  
> *To reproduce:*
>  * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
>  * Sample: [https://github.com/apache/hadoop/pull/3391]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3

12 matches

Site Navigation

Mail list logo

Footer information