connectivity
Hi Team, Can anyone provide the sample code to connect to azure to connect to ADLS using azure key vault(user managed key). Qatar Airways - Going Places Together Disclaimer:- This message (including attachments) is intended solely for the addressee named above. It may be confidential, privileged, subject to copyright, trade secret, or other legal rules and may not be forwarded without the author's permission. If you are not the addressee you must not read, copy or disseminate this message. If you have received it in error please notify the sender immediately and delete the message from all storage devices. Any opinions expressed in this message do not necessarily represent the official positions of Qatar Airways. Any agreements (including any warranties, representations, or offers) concluded with Qatar Airways by using electronic correspondence shall only come into existence if an authorized representative of Qatar Airways has explicitly approved such contract formation. To the fullest extent permissible by law, Qatar Airways disclaim all liability for loss or damage to person or property arising from this message being infected by computer virus or other contamination.
RE: [External]Re: error while connecting to azure blob storage
I sent you a fake token and key
From: Julien Laurenceau
Sent: 23 August 2019 6:06 PM
To: Krishna Chandran Nair
Cc: user@spark.apache.org
Subject: [External]Re: error while connecting to azure blob storage
Hi
Did you just publicly disclosed a real token to tour blob storage ?
If so, please be aware that you share the data and maybe write permission...
Refresh tout token !
Regards
Le ven. 23 août 2019 à 08:33, Krishna Chandran Nair
mailto:kcn...@qatarairways.com.qa>> a écrit :
Hi Team,
I have written a small code to connect to azure blob storage but go error. I
have attached the error log. Please help
Calling command -- ./spark-submit stg.py --jars
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
Code
vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import DataFrameReader
from pyspark.sql import SparkSession
session = SparkSession.builder.getOrCreate()
#session.conf.set("fs.azure",
"org.apache.hadoop.fs.azure.NativeAzureFileSystem")
#session.conf.set("fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem")
#session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
#session.conf.set(
#
"fs.azure.sas.snowflakestrg.blob.core.windows.net/test<http://fs.azure.sas.snowflakestrg.blob.core.windows.net/test>",
#
"?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
#)
session.conf.set(
"fs.azure.account.key.snowflakestrg.blob.core.windows.net<http://fs.azure.account.key.snowflakestrg.blob.core.windows.net>",
"LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv<http://t...@snowflakestrg.blob.core.windows.net/users.csv>")
df.show(5)
Qatar Airways - Going Places Together
[OW LOGO]
Disclaimer:- This message (including attachments) is intended solely for the
addressee named above. It may be confidential, privileged, subject to
copyright, trade secret, or other legal rules and may not be forwarded without
the author's permission. If you are not the addressee you must not read, copy
or disseminate this message. If you have received it in error please notify the
sender immediately and delete the message from all storage devices. Any
opinions expressed in this message do not necessarily represent the official
positions of Qatar Airways. Any agreements (including any warranties,
representations, or offers) concluded with Qatar Airways by using electronic
correspondence shall only come into existence if an authorized representative
of Qatar Airways has explicitly approved such contract formation. To the
fullest extent permissible by law, Qatar Airways disclaim all liability for
loss or damage to person or property arising from this message being infected
by computer virus or other contamination.
RE: [External]Re: error while connecting to azure blob storage
Please find the attached error
From: Roland Johann
Sent: 23 August 2019 10:51 AM
To: Krishna Chandran Nair
Cc: user@spark.apache.org
Subject: [External]Re: error while connecting to azure blob storage
Hi Krishna,
there seems to be no attachment.
In addition, you should NEVER post private credentials to public forums. Please
renew the credentials of your storage account as soon as possible!
Best Regards
Roland Johann
Software Developer/Data Engineer
phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany
Mobil: +49 172 365 26 46
Mail: roland.joh...@phenetic.io<mailto:roland.joh...@phenetic.io>
Web: phenetic.io<http://phenetic.io>
Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann
Am 23.08.2019 um 08:33 schrieb Krishna Chandran Nair
mailto:kcn...@qatarairways.com.qa>>:
Hi Team,
I have written a small code to connect to azure blob storage but go error. I
have attached the error log. Please help
Calling command -- ./spark-submit stg.py --jars
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
Code
vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import DataFrameReader
from pyspark.sql import SparkSession
session = SparkSession.builder.getOrCreate()
#session.conf.set("fs.azure",
"org.apache.hadoop.fs.azure.NativeAzureFileSystem")
#session.conf.set("fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem")
#session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
#session.conf.set(
# "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
#
"?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
#)
session.conf.set(
"fs.azure.account.key.snowflakestrg.blob.core.windows.net",
"LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv")
df.show(5)
Qatar Airways - Going Places Together
[OW LOGO]
Disclaimer:- This message (including attachments) is intended solely for the
addressee named above. It may be confidential, privileged, subject to
copyright, trade secret, or other legal rules and may not be forwarded without
the author's permission. If you are not the addressee you must not read, copy
or disseminate this message. If you have received it in error please notify the
sender immediately and delete the message from all storage devices. Any
opinions expressed in this message do not necessarily represent the official
positions of Qatar Airways. Any agreements (including any warranties,
representations, or offers) concluded with Qatar Airways by using electronic
correspondence shall only come into existence if an authorized representative
of Qatar Airways has explicitly approved such contract formation. To the
fullest extent permissible by law, Qatar Airways disclaim all liability for
loss or damage to person or property arising from this message being infected
by computer virus or other contamination.
Error
--
citus@azcitusclient:~/spark/spark-2.3.3-bin-hadoop2.7/bin$ ./spark-submit
stg.py --jars
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.apache.hadoop.security.authentication.util.KerberosUtil
(file:/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-auth-2.7.3.jar)
to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of
org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
19/08/19 08:55:31 DEBUG NativeCodeLoader: Trying to load the custom-built
native-hadoop library...
19/08/19 08:55:31 DEBUG NativeCodeLoader: Failed to load native-hadoop with
error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path:
[/usr/java/packages/lib, /usr/lib/x86_64-linux-gnu/jni, /lib/x86_64-linux-gnu,
/usr/lib/x86_64-linux-gnu, /usr/lib/jni, /lib, /usr/lib]
19/08/19 08:55:31 DEBUG NativeCodeLoader:
java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
19/08/19 08:55:31 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
19/08/19 08:55:31 INFO Spa
RE: error while connecting to azure blob storage
Hi Team,
I have written a small code to connect to azure blob storage but go error. I
have attached the error log. Please help
Calling command -- ./spark-submit stg.py --jars
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
Code
vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import DataFrameReader
from pyspark.sql import SparkSession
session = SparkSession.builder.getOrCreate()
#session.conf.set("fs.azure",
"org.apache.hadoop.fs.azure.NativeAzureFileSystem")
#session.conf.set("fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem")
#session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
#session.conf.set(
# "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
#
"?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
#)
session.conf.set(
"fs.azure.account.key.snowflakestrg.blob.core.windows.net",
"LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv")
df.show(5)
Qatar Airways - Going Places Together
Disclaimer:- This message (including attachments) is intended solely for the
addressee named above. It may be confidential, privileged, subject to
copyright, trade secret, or other legal rules and may not be forwarded without
the author's permission. If you are not the addressee you must not read, copy
or disseminate this message. If you have received it in error please notify the
sender immediately and delete the message from all storage devices. Any
opinions expressed in this message do not necessarily represent the official
positions of Qatar Airways. Any agreements (including any warranties,
representations, or offers) concluded with Qatar Airways by using electronic
correspondence shall only come into existence if an authorized representative
of Qatar Airways has explicitly approved such contract formation. To the
fullest extent permissible by law, Qatar Airways disclaim all liability for
loss or damage to person or property arising from this message being infected
by computer virus or other contamination.
