[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036723#comment-16036723
]
Dongjoon Hyun commented on SPARK-15731:
---
Hi, All.
I simply tested on HDFS with the latest versions (1.6.3 and 2.1.1) at HDP.
This issue seems to be resolved now.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354851#comment-15354851
]
Ran Haim commented on SPARK-15731:
--
I have tested it under 1.6.1, and writing the data under the local file system
the permissions are OK, but when writing the data under HDFS the problem occurs.
I don't see a point on testing it under 2.x.
thanks.
Ran.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351012#comment-15351012
]
Sean Owen commented on SPARK-15731:
---
What do you mean? I'm trying to bridge the gap between what you're seeing and
what Kevin is seeing. Since I am not sure anyone's going to work on this, I am
trying to guide you to narrow it down if you're interested. 1.6.2 will be out
shortly, and I expect a 1.6.3 eventually. But first we have to figure out if
this is even a problem, if it was fixed somewhere, if it can be back-ported,
etc.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351026#comment-15351026
]
Ran Haim commented on SPARK-15731:
--
Oh, OK - I will try to do this on 1.6 locally and then on 2.x locally.
Thanks.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350996#comment-15350996
]
Ran Haim commented on SPARK-15731:
--
Yes, I will try it locally.
But this might be the problem, I am saving the orcs on HDFS.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351003#comment-15351003
]
Ran Haim commented on SPARK-15731:
--
wait a minute, you are not addressing any issues on 1.6.1 which is the latest
stable?
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350987#comment-15350987
]
Sean Owen commented on SPARK-15731:
---
Are you addressing Kevin Yu? He's clearly using a local file system and
standalone mode. Can you run that on Spark 2.x locally to start?
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350983#comment-15350983
]
Ran Haim commented on SPARK-15731:
--
You did not answer any of my questions, it will help me build a scenario for
reproducing this.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350977#comment-15350977
]
Sean Owen commented on SPARK-15731:
---
We still have a can't-reproduce here, and you're not using upstream Spark, or
the latest version. It's not necessarily invalid, just don't see that it will
be actionable unless you're submitting a PR. I'll close this later if there is
no follow up.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315887#comment-15315887
]
Ran Haim commented on SPARK-15731:
--
Fine, you can close it - if it works.
I will use a work around.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315885#comment-15315885
]
Sean Owen commented on SPARK-15731:
---
[~ran.h...@optimalplus.com] no, I removed "Target Version". Only committers set
that, and it doesn't make sense to target a version released already. Given
[~kevinyu98] sees different behavior on what I presume is a later version, I
think it's probably fixed in a more up to date version.
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315884#comment-15315884
]
Ran Haim commented on SPARK-15731:
--
I am using Mapr distribution and they provide spark 1.5.1 :/ - so not really.
I did specify the version, but you removed it a few days ago :)
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315782#comment-15315782
]
Ran Haim commented on SPARK-15731:
--
Hi Kevin.
it is pretty much the same, the only difference I see is that I have created a
dataframe with a given schema using createDataFrame(Rdd, StructType).
I am using spark 1.5.1 - what version did you use?
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15731) orc writer directory permissions
[
https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314364#comment-15314364
]
kevin yu commented on SPARK-15731:
--
Hi Ran: I tried on my machine for orc file, the partition directories has the x
permission. Did I do anything different than yours?
scala> spark.createDataFrame(data).toDF("a",
"b").write.format("orc").mode("append").partitionBy("a").save("/Users/qianyangyu/sparkcp/spark-15731orc2")
Qianyangs-MBP:sparkcp qianyangyu$ ls -al spark-15731orc2/
total 8
drwxr-xr-x 10 qianyangyu staff 340 Jun 3 09:11 .
drwxr-xr-x 15 qianyangyu staff 510 Jun 3 09:11 ..
-rw-r--r-- 1 qianyangyu staff8 Jun 3 09:11 ._SUCCESS.crc
-rw-r--r-- 1 qianyangyu staff0 Jun 3 09:11 _SUCCESS
drwxr-xr-x 4 qianyangyu staff 136 Jun 3 09:11 a=1
drwxr-xr-x 4 qianyangyu staff 136 Jun 3 09:11 a=3
drwxr-xr-x 4 qianyangyu staff 136 Jun 3 09:11 a=5
drwxr-xr-x 4 qianyangyu staff 136 Jun 3 09:11 a=7
drwxr-xr-x 4 qianyangyu staff 136 Jun 3 09:11 a=9
drwxr-xr-x 12 qianyangyu staff 408 Jun 3 09:11 a=__HIVE_DEFAULT_PARTITION__
Qianyangs-MBP:sparkcp qianyangyu$
> orc writer directory permissions
>
>
> Key: SPARK-15731
> URL: https://issues.apache.org/jira/browse/SPARK-15731
> Project: Spark
> Issue Type: Bug
>Reporter: Ran Haim
>
> When saving orc files with partitions, the partition directories created do
> not have x permission (even tough umask is 002), then no other users can get
> inside those directories to read the orc file.
> When writing parquet files there is no such issue.
> code example:
> datafrmae.write.format("orc").mode("append").partitionBy("date").save("/path")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
