steveloughran commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2093178178
> Got it, i was planning to embed the logic as part of this PR sometime
early next week but separate PR sounds more manageable!
it's a bit blurred as there are now options in
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2091630129
> I'm doing a quick PR of the design; @HarshitGupta11 and I discussed it.
Got it, i was planning to embed the logic as part of this PR sometime early
next week but separate PR
steveloughran commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2091119335
I'm doing a quick PR of the design; @HarshitGupta11 and I discussed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2089805333
As of today, `fs.s3a.create.performance` is mandatory option while creating
file:
```
builder
.create()
.overwrite(true)
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2089769498
The above proposal of providing list of optimization flags sounds impressive.
Please let me know if this summary looks good:
As part of this Jira:
- Add
steveloughran commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2088443377
I'm going to propose we change how the options are done, and do something
similar for ABFS.
I think we need something like C/C++ optimisers where you pass in a -O list
of things
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2077150464
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks
virajjasani commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1573386499
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##
@@ -124,7 +138,32 @@ public Boolean execute() throws IOException {
mukund-thakur commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1572754351
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##
@@ -124,7 +138,32 @@ public Boolean execute() throws IOException {
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2046223066
> Don't think I have forgotten about this -I have just been very distracted.
me too, not a problem at all :)
--
This is an automated message from the Apache Git Service.
To
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-204647
> I'm wondering if we should provide a list of options to "optimise", e.g
"create, mkdir, delete" and the specific optimisations those get turned on.
Yes, it is suspiciously like
steveloughran commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2046062412
Don't think I have forgotten about this -I have just been very distracted.
I'm wondering if we should provide a list of options to "optimise", e.g
"create, mkdir, delete" and
virajjasani commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1554648819
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##
@@ -124,7 +132,32 @@ public Boolean execute() throws IOException {
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2036356955
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2036347165
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2036214685
Tested against `us-west-2`, looks good for this change.
Though found a separate issue with scale tests using `noaa-cors-pds` bucket
for my local endpoint/region setup. It's
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2036162917
> But if we're doing this for a whole directory, for all applications, I
think that is a bit too risky.
I see your point.
Let me run the whole suite with the latest
steveloughran commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1549880873
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##
@@ -124,7 +132,32 @@ public Boolean execute() throws IOException {
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2031428896
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2031166900
> ok, this is getting over complex.
>
> proposed: copy the superclass code but remove the expectation of failures,
retaining only setup and validation.
sounds good,
steveloughran commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2025732364
reviewed, i'm just wondering how to make the test the cleanest.
Going to invite reviews from @shameersss1 @ahmarsuhail @HarshitGupta11
@mukund-thakur as they've been looking
steveloughran commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1543344114
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdirWithCreatePerf.java:
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2025582097
@steveloughran could you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2020614729
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
shameersss1 commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-1951717665
> @shameersss1 what do you think here? actually, maybe under magic paths we
skip trying to create dirs at all, at least on the in-memory mode. no files to
look for after all so all
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-1948129199
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-1948121472
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
virajjasani commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1492090102
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##
@@ -124,7 +132,32 @@ public Boolean execute() throws IOException {
virajjasani commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1492088874
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdirWithCreatePerf.java:
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the
steveloughran commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1490982845
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##
@@ -73,11 +79,13 @@ public MkdirOperation(
final
steveloughran commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1490959395
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java:
##
@@ -144,7 +147,7 @@ public void
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-1943188871
Tested against `us-west-2`:
`mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch`
--
This is an automated message from the Apache Git Service.
To
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-1935424401
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-1935301858
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
34 matches
Mail list logo
