So I believe the client library, when using the private endpoint, is making multiple calls, the first one is on the private endpoint and that's why you get a 404 when setting https://*storage*-euwest1.p.googleapis.com/*storage*/v1/ As you can see, it's making a call to a wrong URL (twice /storage/v1): https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable This is why you get the 404. When you set the right endpoint you don't get the 404 but later on the client library has to make a call to googleapis.com (if I recall correctly this part) and that's why you got the unknown host exception.
Le mer. 8 nov. 2023 à 16:52, Pierre Villard <[email protected]> a écrit : > I believe that if you temporarily enable debug logs on the GCP client > library (via logback.xml on com.google.cloud.storage) you'd be able to > confirm this. > > Le mer. 8 nov. 2023 à 16:43, <[email protected]> a écrit : > >> Hi Pierre, >> >> I confirm, it works with Proxy. >> >> But how we are sure that the data are transfert from the "private >> connection" and not through the proxy... >> >> Regards >> >> >> *Envoyé:* mercredi 8 novembre 2023 à 16:36 >> *De:* [email protected] >> *À:* [email protected] >> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2 >> Hi Pierre, >> >> But it seems weird because when I put the wrong "storage api url >> : https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"; >> >> We got the wrong url but has the correct hostname >> >> Not Found POST >> https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable >> >> *Envoyé:* mercredi 8 novembre 2023 à 16:29 >> *De:* "Pierre Villard" <[email protected]> >> *À:* [email protected] >> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2 >> IIRC, but not in a position to confirm right now, the right API to set >> would be https://*storage*-euwest1.p.googleapis.com but your host should >> still be able to resolve googleapis.com (I believe this can work through >> proxy as well). >> >> Le mer. 8 nov. 2023 à 16:16, <[email protected]> a écrit : >> >>> Hi >>> >>> I have checked the DNS entry for "*storage*-euwest1.p.googleapis.com >>> <https://deref-gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F>" >>> it seems OK >>> From the VM nifi, in the terminal, I could reach the port 443 it works >>> fine. >>> >>> I check with the devops google team also. >>> >>> Thanks >>> >>> *Envoyé:* mercredi 8 novembre 2023 à 15:40 >>> *De:* "Joe Witt" <[email protected]> >>> *À:* [email protected] >>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2 >>> Ok sounds good. I do think this is the correct path to be taking then. >>> >>> I think though there might be a dns configuration needed. Im not quite >>> sure but I am aware of users that rely on this value heavily. >>> >>> Thanks >>> >>> On Wed, Nov 8, 2023 at 7:34 AM <[email protected]> wrote: >>> >>>> Hello Joe >>>> >>>> Until now we leave the "storage api url" empty by default, but we use >>>> "proxy configuration service" because we are behind the corporate network. >>>> >>>> Since some month ago we have configured "private connection" with >>>> google. >>>> So I have setup "storage api url" with the private url and leave empty >>>> "proxy configuration service" >>>> >>>> Regards >>>> >>>> >>>> *Envoyé:* mercredi 8 novembre 2023 à 14:31 >>>> *De:* "Joe Witt" <[email protected]> >>>> *À:* [email protected] >>>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2 >>>> Minh >>>> >>>> Do you need to override that value? Did Google specify a different URL >>>> for you or you have a private connection? >>>> >>>> If you dont set that value what happens? >>>> >>>> Thanks >>>> >>>> On Wed, Nov 8, 2023 at 4:57 AM <[email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm trying to user the overwriting storage url parameter in the nifi >>>>> version 1.23.2. >>>>> >>>>> Do you know what is the correct parameter to set for the "Storage API >>>>> URL" in the processor PutGCSObject ? >>>>> >>>>> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"; >>>>> It got this error : >>>>> >>>>> >>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload >>>>> flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not >>>>> FoundPOST >>>>> https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot >>>>> Found: com.google.cloud.storage.StorageException: 404 Not Found >>>>> >>>>> >>>>> If I setup "https://*storage*-euwest1.p.googleapis.com"; >>>>> It got this error : >>>>> >>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload >>>>> flowfile=test/minh.txt bucket=xxx key=test/minh.txt >>>>> reason=www.googleapis.com: com.google.cloud.storage.StorageException: >>>>> www.googleapis.com- Caused by: java.net.UnknownHostException: >>>>> www.googleapis.com >>>>> >>>>> >>>>> Thanks for help >>>>> >>>>> Minh >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> >
