Re: Bug : PutGCSObject : Version NIFI 1.23.2

Mon, 13 Nov 2023 02:26:15 -0800

Hello,
 
Seems not working, I don't have any logs concerning google/putgcsObjects in the nifi-app.log 
 
Missing something ?
 
Minh
 
Envoyé: vendredi 10 novembre 2023 à 15:36
De: "Pierre Villard" <[email protected]>
À: [email protected]
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
You'd just need to add
<logger name="com.google.cloud" level="DEBUG" />
 
in the file, for example next to the one like
<logger name="net.schmizz.sshj" level="WARN" />
 
And you should see the debug logs in nifi-app.log
 
 
Le ven. 10 nov. 2023 à 15:31, <[email protected]> a écrit :
Hello Pierre,
 
I'm not developper, so it is a bit confuse for me.
 
I'm trying to temporarily push this config in the logback.xml but it is not working.
 
 
.
.
.
<logger name="com.google.cloud.storage" level="DEBUG" additivity="false">
<appender-ref ref="APP_FILE"/>
</logger>
 
<root level="INFO">
<appender-ref ref="APP_FILE"/>
</root>
.
.
 
It is missing something ?
 
Thanks a lot for your help
 
Regards 
 
Minh
Envoyé: mercredi 8 novembre 2023 à 16:52
De: "Pierre Villard" <[email protected]>
À: [email protected]
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
I believe that if you temporarily enable debug logs on the GCP client library (via logback.xml on com.google.cloud.storage) you'd be able to confirm this.
 
Le mer. 8 nov. 2023 à 16:43, <[email protected]> a écrit :
Hi Pierre,
 
I confirm, it works with Proxy. 
 
But how we are sure that the data are transfert from the "private connection" and not through the proxy...
 
Regards
 
 
Envoyé: mercredi 8 novembre 2023 à 16:36
De: [email protected]
À: [email protected]
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
Hi Pierre,
 
But it seems weird because when I put the wrong "storage api url : https://storage-euwest1.p.googleapis.com/storage/v1/"
 
We got the wrong url but has the correct hostname
 
 
Envoyé: mercredi 8 novembre 2023 à 16:29
De: "Pierre Villard" <[email protected]>
À: [email protected]
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
IIRC, but not in a position to confirm right now, the right API to set would be https://storage-euwest1.p.googleapis.com but your host should still be able to resolve googleapis.com (I believe this can work through proxy as well).
 
Le mer. 8 nov. 2023 à 16:16, <[email protected]> a écrit :
Hi 
 
I have checked the DNS entry for "storage-euwest1.p.googleapis.com" it seems OK
From the VM nifi, in the terminal, I could reach the port 443 it works fine.
 
I check with the devops google team also.
 
Thanks
 
Envoyé: mercredi 8 novembre 2023 à 15:40
De: "Joe Witt" <[email protected]>
À: [email protected]
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
Ok sounds good. I do think this is the correct path to be taking then.
 
I think though there might be a dns configuration needed.  Im not quite sure but I am aware of users that rely on this value heavily.
 
Thanks
 
On Wed, Nov 8, 2023 at 7:34 AM <[email protected]> wrote:
Hello Joe
 
Until now we leave the "storage api url" empty by default, but we use "proxy configuration service" because we are behind the corporate network.
 
Since some month ago we have configured "private connection" with google.
So I have setup "storage api url" with the private url and leave empty "proxy configuration service"
 
Regards 
 
 
Envoyé: mercredi 8 novembre 2023 à 14:31
De: "Joe Witt" <[email protected]>
À: [email protected]
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
Minh
 
Do you need to override that value?  Did Google specify a different URL for you or you have a private connection?
 
If you dont set that value what happens?
 
Thanks
 
On Wed, Nov 8, 2023 at 4:57 AM <[email protected]> wrote:
Hello,
 
I'm trying to user the overwriting storage url parameter in the nifi version 1.23.2.
 
Do you know what is the correct parameter to set for the "Storage API URL" in the processor PutGCSObject ?
 
If I setup "https://storage-euwest1.p.googleapis.com/storage/v1/"
It got this error :
 
PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot Found: com.google.cloud.storage.StorageException: 404 Not Found


 





If I setup "https://storage-euwest1.p.googleapis.com"



It got this error :









PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com- Caused by: java.net.UnknownHostException: www.googleapis.com







 



Thanks for help



 



Minh 
 
 
 
 
 
 
 
 
 
 

Reply via email to