Re: note.json or interpreter.json format change?

2017-08-28 Thread Jeff Zhang 
Yeah, that bug happens in around May. But it should only happen in the
snapshot version, not in any release version.


Ruslan Dautkhanov 于2017年8月29日周二 下午1:01写道：

> This problem only happens when trying to upgraded Zeppelin (yes, from
> yesterday's snapshot) and then switching back
> to May-old Zeppelin.
> Date format has changed in all fields
> - "dateCreated", "dateStarted", "dateFinished"
> Impact limited to only notebooks modified in the newer version of Zeppelin
> - they'll not show up in list of available notebooks.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Mon, Aug 28, 2017 at 6:07 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
>
>>
>> Do you use the latest zeppelin master branch ? I see this issue before,
>> but believe it has been fixed.
>>
>>
>> Best Regard,
>> Jeff Zhang
>>
>>
>> From: Ruslan Dautkhanov 
>> Reply-To: "users@zeppelin.apache.org" 
>> Date: Tuesday, August 29, 2017 at 1:50 AM
>> To: Zeppelin Users 
>> Subject: Re: note.json or interpreter.json format change?
>>
>> There is a date format change
>>
>> current zeppelin snapshot note.json date format example
>>   "dateUpdated": "2017-08-27 19:56:22.229",
>>
>> ~May-1st zeppelin note.json date format example
>>   "dateUpdated": "May 5, 2017 2:55:34 PM",
>>
>> This breaks note.json compatibility.
>> Can somebody please point me to PR / jria for this change?
>> Any workarounds that would make an upgrade easier?
>> Also, this change makes reverting zeppelin upgrades impossible.
>>
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Mon, Aug 28, 2017 at 11:35 AM, Ruslan Dautkhanov > > wrote:
>>
>>> I guess exception in the log might be why- see [1].
>>> Any fixes/ workarounds for this issue?
>>>
>>>
>>> [1]
>>>
>>> ERROR [2017-08-28 11:25:52,628] ({main} VFSNotebookRepo.java[list]:151)
>>> - Can't read note file:///home/rdautkha/zeppelin/notebooks/2CJW01020
>>> com.google.gson.JsonSyntaxException: 2017-08-27 19:56:22.229
>>> at
>>> com.google.gson.internal.bind.DateTypeAdapter.deserializeToDate(DateTypeAdapter.java:81)
>>> at
>>> com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeAdapter.java:66)
>>> at
>>> com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeAdapter.java:41)
>>> at
>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
>>> at
>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
>>> at
>>> com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
>>> at
>>> com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:81)
>>> at
>>> com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:60)
>>> at
>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
>>> at
>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
>>> at com.google.gson.Gson.fromJson(Gson.java:791)
>>> at com.google.gson.Gson.fromJson(Gson.java:757)
>>> at com.google.gson.Gson.fromJson(Gson.java:706)
>>> at com.google.gson.Gson.fromJson(Gson.java:678)
>>> at org.apache.zeppelin.notebook.Note.fromJson(Note.java:898)
>>> at
>>> org.apache.zeppelin.notebook.repo.VFSNotebookRepo.getNote(VFSNotebookRepo.java:178)
>>> at
>>> org.apache.zeppelin.notebook.repo.VFSNotebookRepo.getNoteInfo(VFSNotebookRepo.java:201)
>>> at
>>> org.apache.zeppelin.notebook.repo.VFSNotebookRepo.list(VFSNotebookRepo.java:146)
>>> at
>>> org.apache.zeppelin.notebook.repo.NotebookRepoSync.list(NotebookRepoSync.java:158)
>>> at
>>> org.apache.zeppelin.notebook.Notebook.loadAllNotes(Notebook.java:553)
>>> at
>>> org.apache.zeppelin.notebook.Notebook.(Notebook.java:124)
>>> at
>>> org.apache.zeppelin.server.ZeppelinServer.(ZeppelinServer.java:158)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>> at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>> at
>>> org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createSingletonInstance(CXFNonSpringJaxrsServlet.java:382)
>>> at
>>> org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createApplicationInstance(CXFNonSpringJaxrsServlet.java:454)
>>> at
>>>

Re: note.json or interpreter.json format change?

2017-08-28 Thread Ruslan Dautkhanov 
This problem only happens when trying to upgraded Zeppelin (yes, from
yesterday's snapshot) and then switching back
to May-old Zeppelin.
Date format has changed in all fields
- "dateCreated", "dateStarted", "dateFinished"
Impact limited to only notebooks modified in the newer version of Zeppelin
- they'll not show up in list of available notebooks.



-- 
Ruslan Dautkhanov

On Mon, Aug 28, 2017 at 6:07 PM, Jianfeng (Jeff) Zhang <
jzh...@hortonworks.com> wrote:

>
> Do you use the latest zeppelin master branch ? I see this issue before,
> but believe it has been fixed.
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Ruslan Dautkhanov 
> Reply-To: "users@zeppelin.apache.org" 
> Date: Tuesday, August 29, 2017 at 1:50 AM
> To: Zeppelin Users 
> Subject: Re: note.json or interpreter.json format change?
>
> There is a date format change
>
> current zeppelin snapshot note.json date format example
>   "dateUpdated": "2017-08-27 19:56:22.229",
>
> ~May-1st zeppelin note.json date format example
>   "dateUpdated": "May 5, 2017 2:55:34 PM",
>
> This breaks note.json compatibility.
> Can somebody please point me to PR / jria for this change?
> Any workarounds that would make an upgrade easier?
> Also, this change makes reverting zeppelin upgrades impossible.
>
>
>
>
> --
> Ruslan Dautkhanov
>
> On Mon, Aug 28, 2017 at 11:35 AM, Ruslan Dautkhanov 
> wrote:
>
>> I guess exception in the log might be why- see [1].
>> Any fixes/ workarounds for this issue?
>>
>>
>> [1]
>>
>> ERROR [2017-08-28 11:25:52,628] ({main} VFSNotebookRepo.java[list]:151)
>> - Can't read note file:///home/rdautkha/zeppelin/notebooks/2CJW01020
>> com.google.gson.JsonSyntaxException: 2017-08-27 19:56:22.229
>> at com.google.gson.internal.bind.DateTypeAdapter.deserializeToD
>> ate(DateTypeAdapter.java:81)
>> at com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeA
>> dapter.java:66)
>> at com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeA
>> dapter.java:41)
>> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1
>> .read(ReflectiveTypeAdapterFactory.java:93)
>> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$A
>> dapter.read(ReflectiveTypeAdapterFactory.java:172)
>> at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.
>> read(TypeAdapterRuntimeTypeWrapper.java:40)
>> at com.google.gson.internal.bind.CollectionTypeAdapterFactory$A
>> dapter.read(CollectionTypeAdapterFactory.java:81)
>> at com.google.gson.internal.bind.CollectionTypeAdapterFactory$A
>> dapter.read(CollectionTypeAdapterFactory.java:60)
>> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1
>> .read(ReflectiveTypeAdapterFactory.java:93)
>> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$A
>> dapter.read(ReflectiveTypeAdapterFactory.java:172)
>> at com.google.gson.Gson.fromJson(Gson.java:791)
>> at com.google.gson.Gson.fromJson(Gson.java:757)
>> at com.google.gson.Gson.fromJson(Gson.java:706)
>> at com.google.gson.Gson.fromJson(Gson.java:678)
>> at org.apache.zeppelin.notebook.Note.fromJson(Note.java:898)
>> at org.apache.zeppelin.notebook.repo.VFSNotebookRepo.getNote(VF
>> SNotebookRepo.java:178)
>> at org.apache.zeppelin.notebook.repo.VFSNotebookRepo.getNoteInf
>> o(VFSNotebookRepo.java:201)
>> at org.apache.zeppelin.notebook.repo.VFSNotebookRepo.list(VFSNo
>> tebookRepo.java:146)
>> at org.apache.zeppelin.notebook.repo.NotebookRepoSync.list(Note
>> bookRepoSync.java:158)
>> at org.apache.zeppelin.notebook.Notebook.loadAllNotes(Notebook.
>> java:553)
>> at org.apache.zeppelin.notebook.Notebook.(Notebook.java:1
>> 24)
>> at org.apache.zeppelin.server.ZeppelinServer.(ZeppelinSer
>> ver.java:158)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>> ConstructorAccessorImpl.java:57)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>> legatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:5
>> 26)
>> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.create
>> SingletonInstance(CXFNonSpringJaxrsServlet.java:382)
>> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.create
>> ApplicationInstance(CXFNonSpringJaxrsServlet.java:454)
>> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.create
>> ServerFromApplication(CXFNonSpringJaxrsServlet.java:432)
>> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.init(C
>> XFNonSpringJaxrsServlet.java:93)
>> at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletH
>> older.java:616)
>> at

Re: note.json or interpreter.json format change?

2017-08-28 Thread Ruslan Dautkhanov 
There is a date format change

current zeppelin snapshot note.json date format example
  "dateUpdated": "2017-08-27 19:56:22.229",

~May-1st zeppelin note.json date format example
  "dateUpdated": "May 5, 2017 2:55:34 PM",

This breaks note.json compatibility.
Can somebody please point me to PR / jria for this change?
Any workarounds that would make an upgrade easier?
Also, this change makes reverting zeppelin upgrades impossible.




-- 
Ruslan Dautkhanov

On Mon, Aug 28, 2017 at 11:35 AM, Ruslan Dautkhanov 
wrote:

> I guess exception in the log might be why- see [1].
> Any fixes/ workarounds for this issue?
>
>
> [1]
>
> ERROR [2017-08-28 11:25:52,628] ({main} VFSNotebookRepo.java[list]:151) -
> Can't read note file:///home/rdautkha/zeppelin/notebooks/2CJW01020
> com.google.gson.JsonSyntaxException: 2017-08-27 19:56:22.229
> at com.google.gson.internal.bind.DateTypeAdapter.
> deserializeToDate(DateTypeAdapter.java:81)
> at com.google.gson.internal.bind.DateTypeAdapter.read(
> DateTypeAdapter.java:66)
> at com.google.gson.internal.bind.DateTypeAdapter.read(
> DateTypeAdapter.java:41)
> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$
> 1.read(ReflectiveTypeAdapterFactory.java:93)
> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$
> Adapter.read(ReflectiveTypeAdapterFactory.java:172)
> at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.
> read(TypeAdapterRuntimeTypeWrapper.java:40)
> at com.google.gson.internal.bind.CollectionTypeAdapterFactory$
> Adapter.read(CollectionTypeAdapterFactory.java:81)
> at com.google.gson.internal.bind.CollectionTypeAdapterFactory$
> Adapter.read(CollectionTypeAdapterFactory.java:60)
> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$
> 1.read(ReflectiveTypeAdapterFactory.java:93)
> at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$
> Adapter.read(ReflectiveTypeAdapterFactory.java:172)
> at com.google.gson.Gson.fromJson(Gson.java:791)
> at com.google.gson.Gson.fromJson(Gson.java:757)
> at com.google.gson.Gson.fromJson(Gson.java:706)
> at com.google.gson.Gson.fromJson(Gson.java:678)
> at org.apache.zeppelin.notebook.Note.fromJson(Note.java:898)
> at org.apache.zeppelin.notebook.repo.VFSNotebookRepo.getNote(
> VFSNotebookRepo.java:178)
> at org.apache.zeppelin.notebook.repo.VFSNotebookRepo.
> getNoteInfo(VFSNotebookRepo.java:201)
> at org.apache.zeppelin.notebook.repo.VFSNotebookRepo.list(
> VFSNotebookRepo.java:146)
> at org.apache.zeppelin.notebook.repo.NotebookRepoSync.list(
> NotebookRepoSync.java:158)
> at org.apache.zeppelin.notebook.Notebook.loadAllNotes(
> Notebook.java:553)
> at org.apache.zeppelin.notebook.Notebook.(Notebook.java:124)
> at org.apache.zeppelin.server.ZeppelinServer.(
> ZeppelinServer.java:158)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.
> createSingletonInstance(CXFNonSpringJaxrsServlet.java:382)
> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.
> createApplicationInstance(CXFNonSpringJaxrsServlet.java:454)
> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.
> createServerFromApplication(CXFNonSpringJaxrsServlet.java:432)
> at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.init(
> CXFNonSpringJaxrsServlet.java:93)
> at org.eclipse.jetty.servlet.ServletHolder.initServlet(
> ServletHolder.java:616)
> at org.eclipse.jetty.servlet.ServletHolder.initialize(
> ServletHolder.java:396)
> at org.eclipse.jetty.servlet.ServletHandler.initialize(
> ServletHandler.java:871)
> at org.eclipse.jetty.servlet.ServletContextHandler.startContext(
> ServletContextHandler.java:298)
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(
> WebAppContext.java:1349)
> at org.eclipse.jetty.webapp.WebAppContext.startContext(
> WebAppContext.java:1342)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doStart(ContextHandler.java:741)
> at org.eclipse.jetty.webapp.WebAppContext.doStart(
> WebAppContext.java:505)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.
> start(AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.
> start(ContainerLifeCycle.java:132)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.
> doStart(ContainerLifeCycle.java:114)
> at org.eclipse.jetty.server.handler.AbstractHandler.
>

Re: note.json or interpreter.json format change?

2017-08-28 Thread Ruslan Dautkhanov 
I guess exception in the log might be why- see [1].
Any fixes/ workarounds for this issue?


[1]

ERROR [2017-08-28 11:25:52,628] ({main} VFSNotebookRepo.java[list]:151) -
Can't read note file:///home/rdautkha/zeppelin/notebooks/2CJW01020
com.google.gson.JsonSyntaxException: 2017-08-27 19:56:22.229
at
com.google.gson.internal.bind.DateTypeAdapter.deserializeToDate(DateTypeAdapter.java:81)
at
com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeAdapter.java:66)
at
com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeAdapter.java:41)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at
com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
at
com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:81)
at
com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:60)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:791)
at com.google.gson.Gson.fromJson(Gson.java:757)
at com.google.gson.Gson.fromJson(Gson.java:706)
at com.google.gson.Gson.fromJson(Gson.java:678)
at org.apache.zeppelin.notebook.Note.fromJson(Note.java:898)
at
org.apache.zeppelin.notebook.repo.VFSNotebookRepo.getNote(VFSNotebookRepo.java:178)
at
org.apache.zeppelin.notebook.repo.VFSNotebookRepo.getNoteInfo(VFSNotebookRepo.java:201)
at
org.apache.zeppelin.notebook.repo.VFSNotebookRepo.list(VFSNotebookRepo.java:146)
at
org.apache.zeppelin.notebook.repo.NotebookRepoSync.list(NotebookRepoSync.java:158)
at
org.apache.zeppelin.notebook.Notebook.loadAllNotes(Notebook.java:553)
at org.apache.zeppelin.notebook.Notebook.(Notebook.java:124)
at
org.apache.zeppelin.server.ZeppelinServer.(ZeppelinServer.java:158)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createSingletonInstance(CXFNonSpringJaxrsServlet.java:382)
at
org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createApplicationInstance(CXFNonSpringJaxrsServlet.java:454)
at
org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createServerFromApplication(CXFNonSpringJaxrsServlet.java:432)
at
org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.init(CXFNonSpringJaxrsServlet.java:93)
at
org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:616)
at
org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:396)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:871)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:387)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at

note.json or interpreter.json format change?

2017-08-28 Thread Ruslan Dautkhanov 
Testing a newer Zeppelin version from yesterday's zeppelin snapshot.
Noticed it doesn't see majority of my notebooks.
Our previous Zeppelin version is ~end of April snapshot of Zeppelin.
Did the format of note.json or interpreter.json expected format change
in a way that made those note.json notebooks not show up?


Thanks,
Ruslan Dautkhanov

Re: zeppelin build fails with DependencyConvergence error

2017-08-28 Thread Ruslan Dautkhanov 
Thanks for prompt response Jeff.

Yep, I was able to build Zeppelin without -Dhadoop.version=2.6.0-cdh5.12.0
.. interesting. I had to use this previously as
zeppelin wouldn't build without that (on older versions).

---
Might be unrelated, but now to start a spark interperter I have to have a
valid kerberos ticket.
SPARK_SUBMIT_OPTIONS's --keytab & --principal was always sufficient.
That might be a Spark 2.2 change though and not Zeppelin.

I had to have a valid local (on the interpreter's servers) kerberos ticket.
Also might be related - https://issues.apache.org/jira/browse/SPARK-19038
I see this would be very inconvenient for users.



[1]

ERROR [2017-08-27 19:46:17,597] ({pool-2-thread-5}
Logging.scala[logError]:91) - Error initializing SparkContext.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation
Token can be issued only with kerberos or web authentication
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7501)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:548)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:663)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:981)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)

at org.apache.hadoop.ipc.Client.call(Client.java:1502)
at org.apache.hadoop.ipc.Client.call(Client.java:1439)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:928)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy13.getDelegationToken(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1085)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1508)
at
org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:546)
at
org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:524)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2292)
at
org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:53)
at
org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:50)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
at
org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider.obtainCredentials(HadoopFSCredentialProvider.scala:50)
at
org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:82)
at
org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:80)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at
scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at