[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929496#comment-16929496
]
Leah commented on AIRFLOW-5418:
---
Apologies for the incorrect terminology. I'll update the title of the ticket to
reflect that
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screen Shot 2019-09-13 at 12.35.33 PM.png, Screen Shot
> 2019-09-13 at 12.36.40 PM.png, Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929491#comment-16929491
]
Kamil Bregula commented on AIRFLOW-5418:
In gs://us-central1-leah-playground-*bucket/dags/trigger_response_dag.py
gs:// is a protocol
us-central1-leah-playground-*bucket is a bucket name. It doesn't contain slash.
dags/trigger_response_dag.py is a object name.
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screen Shot 2019-09-13 at 12.35.33 PM.png, Screen Shot
> 2019-09-13 at 12.36.40 PM.png, Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929486#comment-16929486
]
Kamil Bregula commented on AIRFLOW-5418:
dags/ data/ is a object
us-central-*-bucket is a bucket name.
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screen Shot 2019-09-13 at 12.35.33 PM.png, Screen Shot
> 2019-09-13 at 12.36.40 PM.png, Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929481#comment-16929481
]
Leah commented on AIRFLOW-5418:
---
Looking at your example of the invalid request, I suspect that if the name of
the object in the bucket contains a slash on the end, it isn't valid (e.g.
`gs://test-slash/sss`). But, if there is a slash in the middle, like
`gs://test-slash/sss/my-file.py`, it IS valid
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screen Shot 2019-09-13 at 12.35.33 PM.png, Screen Shot
> 2019-09-13 at 12.36.40 PM.png, Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929480#comment-16929480
]
Leah commented on AIRFLOW-5418:
---
I'm attaching two screenshots - one of them shows the path to an Airflow DAG I
have inside of cloud composer - it's in a bucket with a pathname that contains
slashes to mimic subdirectories. The second shows how they are rendered as
subdirectories within GCS.
!Screen Shot 2019-09-13 at 12.35.33 PM.png!!Screen Shot 2019-09-13 at 12.36.40
PM.png!
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screen Shot 2019-09-13 at 12.35.33 PM.png, Screen Shot
> 2019-09-13 at 12.36.40 PM.png, Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925961#comment-16925961
]
Kamil Bregula commented on AIRFLOW-5418:
Can you show proof that a bucket can contain slash?
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925960#comment-16925960
]
Kamil Bregula commented on AIRFLOW-5418:
Bucket is a "root directory"
Object is a directory and files in bucket. Its may contain names.
Bucket is not object.
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925956#comment-16925956
]
Kamil Bregula commented on AIRFLOW-5418:
Bucket cannot contain slash. !Screenshot 2019-09-09 at 19.46.26.png!
Objects may contain slashes.
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
> Attachments: Screenshot 2019-09-09 at 19.46.26.png
>
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925950#comment-16925950
]
Leah commented on AIRFLOW-5418:
---
No - this is still an issue. Slash names in buckets are valid, and using this
operator, users cannot download from a bucket with a slash name
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925948#comment-16925948
]
Kamil Bregula commented on AIRFLOW-5418:
Yes. Even more interesting is when the name contains two characters "/" next to
each other. Then an empty name is displayed in the interface. This is very
confusing.
Can I close this ticket?
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925859#comment-16925859
]
Leah commented on AIRFLOW-5418:
---
Hey [~kamil.bregula]! Yeah - that top part is kind of misleading. If you scroll
down a little further to the [Object Naming
Considerations|https://cloud.google.com/storage/docs/naming#object-considerations]
it talks about the use of slashes. Objects can have slashes in the name, and
in the UI they render as if they were subdirectories, though under the hood the
object exists in the bucket and just has a name with slashes in it.
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5418) bucket names with a slash in them cause an error with the GCS Download Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925226#comment-16925226
]
Kamil Bregula commented on AIRFLOW-5418:
Slash in a bucket name is restricted?
{quote}Bucket names must contain only lowercase letters, numbers, dashes
({{-}}), underscores ({{_}}), and dots ({{.}}). Names containing dots require
[verification|https://cloud.google.com/storage/docs/domain-name-verification].
{quote}
[https://cloud.google.com/storage/docs/naming]
> bucket names with a slash in them cause an error with the GCS Download
> Operator
> ---
>
> Key: AIRFLOW-5418
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5418
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2,
> 1.10.3, 1.10.4, 1.10.5
>Reporter: Leah
>Priority: Minor
>
> If you try to use the GCS Downlaod Operator on a bucket that has a slash in
> the name, for example, my-bucket/dags, the operation results in a "Bad
> Request" error. I replicated this error consistently in Cloud Composer (GCP
> hosted managed Airflow) and was able to get it to work on a bucket name
> without a slash in it. I was not able to use a backslash to escape the slash
> when passing in the bucket name either.
>
>
> {{[2019-09-05 23:51:53,951] \{models.py:1796} ERROR - requesting
> https://www.googleapis.com/storage/v1/b/my-bucket/dags/o/gcs_download.py?alt=media
> returned "Bad Request">}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
