Cloudera's default configuration for the HDFSDirectoryFactory <https://github.com/cloudera/lucene-solr/blob/cdh6.3.3/cloudera/solrconfig.xml#L118> is very similar to yours in solrconfig.xml. The solr.hdfs.home property is provided as a java property during Solr startup and we haven't seen the ":" issue yet.
Hope it helps gp On Wed, Aug 26, 2020 at 9:17 AM Prashant Jyoti <jtprash...@gmail.com> wrote: > Hi Joe, > Yes I had made these changes for getting HDFS to work with Solr. Below are > config changes which I carried out: > > -------------------------------- Changes in solr.in.cmd > -------------------------------- > > set SOLR_OPTS=%SOLR_OPTS% -Dsolr.directoryFactory=HdfsDirectoryFactory > set SOLR_OPTS=%SOLR_OPTS% -Dsolr.lock.type=hdfs > set SOLR_OPTS=%SOLR_OPTS% -Dsolr.hdfs.home=hdfs:// > > hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data > > > -------------------------------- Changes in solrconfig.xml > -------------------------------- > > <directoryFactory name="DirectoryFactory" > class="solr.HdfsDirectoryFactory"> > <str name="solr.hdfs.home">hdfs:// > > hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data > </str> > <str name="solr.hdfs.confdir">/etc/hadoop/conf</str> > <bool name="solr.hdfs.blockcache.enabled">true</bool> > <int name="solr.hdfs.blockcache.slab.count">1</int> > <bool > name="solr.hdfs.blockcache.direct.memory.allocation">true</bool> > <int name="solr.hdfs.blockcache.blocksperbank">16384</int> > <bool name="solr.hdfs.blockcache.read.enabled">true</bool> > <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool> > <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int> > <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int> > </directoryFactory> > > Let me know if you have any comments? > > Thanks! > > On Mon, Aug 24, 2020 at 10:23 PM Joe Obernberger < > joseph.obernber...@gmail.com> wrote: > > > Are you running with solr.lock.type=hdfs > > ? > > > > Have you defined your DirectoryFactory - something like: > > > > <directoryFactory name="DirectoryFactory" > > class="solr.HdfsDirectoryFactory"> > > <bool name="solr.hdfs.blockcache.enabled">true</bool> > > <bool name="solr.hdfs.blockcache.global">true</bool> > > <int name="solr.hdfs.blockcache.slab.count">43</int> > > <bool > > name="solr.hdfs.blockcache.direct.memory.allocation">true</bool> > > <int name="solr.hdfs.blockcache.blocksperbank">16384</int> > > <bool name="solr.hdfs.blockcache.read.enabled">true</bool> > > <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool> > > <int > name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">128</int> > > <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">1024</int> > > <str > name="solr.hdfs.home">hdfs://nameservice1:8020/solr8.2.0</str> > > <str > name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</str> > > </directoryFactory> > > > > -Joe > > On 8/20/2020 2:30 AM, Prashant Jyoti wrote: > > > > Hi Joe, > > These are the errors I am running into: > > > > org.apache.solr.common.SolrException: Error CREATEing SolrCore > > 'newcollsolr2_shard1_replica_n1': Unable to create core > > [newcollsolr2_shard1_replica_n1] Caused by: Illegal char <:> at index 4: > > hdfs:// > > > hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\ > <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C> > > < > http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C > > > > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1256) > > at > > > org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93) > > at > > > org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362) > > at > > > org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397) > > at > > > org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181) > > at > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) > > at > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842) > > at > > > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808) > > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596) > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > > at > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > at > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) > > at > > > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177) > > at > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > at > > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > at org.eclipse.jetty.server.Server.handle(Server.java:500) > > at > > > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) > > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) > > at > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) > > at > > org.eclipse.jetty.io > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > > at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > > at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > > at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > > at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > > at > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: org.apache.solr.common.SolrException: Unable to create core > > [newcollsolr2_shard1_replica_n1] > > at > > > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1327) > > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1217) > > ... 47 more > > Caused by: org.apache.solr.common.SolrException: Illegal char <:> at > index > > 4: hdfs:// > > > hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\ > <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C> > > < > http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C > > > > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) > > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) > > at > > > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1306) > > ... 48 more > > Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index > > 4: hdfs:// > > > hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\ > <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C> > > < > http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C > > > > at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) > > at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) > > at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) > > at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94) > > at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255) > > at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53) > > at org.apache.solr.core.SolrCore.initUpdateLogDir(SolrCore.java:1380) > > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:958) > > ... 50 more > > > > Attaching it as a file too. Thanks! > > > > On Wed, Aug 19, 2020 at 9:37 PM Joe Obernberger < > > joseph.obernber...@gmail.com> wrote: > > > >> Your exception didn't come across - can you paste it in? > >> > >> -Joe > >> > >> On 8/19/2020 10:50 AM, Prashant Jyoti wrote: > >> > You're right Andrew. Even I read about that. But there's a use case > for > >> > which we want to configure the said case. > >> > > >> > Are you also aware of what feature we are moving towards instead of > >> HDFS? > >> > Will you be able to help me with the error that I'm running into? > >> > > >> > Thanks in advance! > >> > > >> > > >> > On Wed, 19 Aug, 2020, 5:24 pm Andrew MacKay, < > andrew.mac...@superna.net > >> > > >> > wrote: > >> > > >> >> I believe HDFS support is being deprecated in Solr. Not sure you > want > >> to > >> >> continue configuration if support will disappear. > >> >> > >> >> On Wed, Aug 19, 2020 at 7:52 AM Prashant Jyoti <jtprash...@gmail.com > > > >> >> wrote: > >> >> > >> >>> Hi all, > >> >>> Hope you are healthy and safe. > >> >>> > >> >>> Need some help with HDFS configuration. > >> >>> > >> >>> Could anybody of you share an example of the configuration with > which > >> you > >> >>> are running Solr with HDFS in any of your production/dev > environments? > >> >>> I am interested in the parts of SolrConfig.xml / Solr.in.cmd/sh > which > >> you > >> >>> may have modified. Obviously with the security parts obfuscated. > >> >>> > >> >>> I am stuck at an error and unable to move ahead. Attaching the > >> exception > >> >>> log if anyone is interested to look at the error. > >> >>> > >> >>> Thanks! > >> >>> > >> >>> -- > >> >>> Regards, > >> >>> Prashant. > >> >>> > >> >> -- > >> >> CONFIDENTIALITY NOTICE: The information contained in this email is > >> >> privileged and confidential and intended only for the use of the > >> >> individual > >> >> or entity to whom it is addressed. If you receive this message in > >> error, > >> >> please notify the sender immediately at 613-729-1100 and destroy the > >> >> original message and all copies. Thank you. > >> >> > >> > > >> > > > > > > -- > > Regards, > > Prashant. > > > > > > < > http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > > www.avg.com > > < > http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient > > > > <#m_-6899573471279532520_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > > > > > -- > Regards, > Prashant. >