Re: Custom JdbcConnectionProvider

2020-10-28 Thread rafaelkyrdan
Thank you Gabor Somogyi for highlighting the issue. I cannot compile the example because 3.1.0-SNAPSHOT is not available in maven. What resolvers do you use? I tried one which suggested in SBT documentation SBT documentation b

Re: Custom JdbcConnectionProvider

2020-10-28 Thread Mich Talebzadeh
I think you can pickup your custom build driver from the command line itself Here I am using a custom build third-party driver to access Oracle Table on-premisses from cloud val jdbUrl = "jdbc:datadirect:ddhybrid://"+HybridServer+":"+HybridPort+";hybridDataPipelineDataSource="+ hybridDataPipeline

Re: Custom JdbcConnectionProvider

2020-10-28 Thread Gabor Somogyi
Please use the latest snapshot as dependency + update force the snapshots. On Wed, Oct 28, 2020 at 5:42 PM rafaelkyrdan wrote: > I cloned your example and it is also cannot be compiled > < > http://apache-spark-user-list.1001560.n3.nabble.com/file/t10977/Screenshot_2020-10-28_182213.png> > > > [

Re: Custom JdbcConnectionProvider

2020-10-28 Thread rafaelkyrdan
I cloned your example and it is also cannot be compiled [ERROR] C:\Users\Rafael_Kyrdan\projects\spark-jdbc-connection-provider\src\main\scala\com\spark\ExampleJdbcConnectionProvider.scala:23: error

Re: Custom JdbcConnectionProvider

2020-10-28 Thread rafaelkyrdan
Thank you for a quick response. Your example is clear but I wasn't able to extend *JdbcConnectionProvider * in my application. I'm getting compile errors: /[error] C:\Users\Rafael_Kyrdan\projects\tdm_engine2\src\main\scala\com\tdm\engine\connection\BasicConnectionProvider.scala:10:39: not found:

Re: spark-submit parameters about two keytab files to yarn and kafka

2020-10-28 Thread Gabor Somogyi
Hi, Cross-realm trust must be configured. One can find several docs on how to do that. BR, G On Wed, Oct 28, 2020 at 8:21 AM big data wrote: > Hi, > > We want to submit spark streaming job to YARN and consume Kafka topic. > > YARN and Kafka are in two different clusters, and they have the > d

spark-submit parameters about two keytab files to yarn and kafka

2020-10-28 Thread big data
Hi, We want to submit spark streaming job to YARN and consume Kafka topic. YARN and Kafka are in two different clusters, and they have the different kerberos authentication. We have two keytab files for YARN and Kafka. And my questions is how to add parameter for spark-submit command for th