Re: SolrCloud on HDFS empty tlog hence doesn't replay after Solr process crash and restart

2014-08-01 Thread Tom Chen 
I wonder if there's any update on this. Should we create a JIRA to track
this?

Thanks,
Tom


On Mon, Jul 21, 2014 at 12:18 PM, Mark Miller markrmil...@gmail.com wrote:

 It’s on my list to investigate.

 --
 Mark Miller
 about.me/markrmiller

 On July 21, 2014 at 10:26:09 AM, Tom Chen (tomchen1...@gmail.com) wrote:
  Any thought about this issue: Solr on HDFS generate empty tlog when add
  documents without commit.
 
  Thanks,
  Tom
 
 
  On Fri, Jul 18, 2014 at 12:21 PM, Tom Chen wrote:
 
   Hi,
  
   This seems a bug for Solr running on HDFS.
  
   Reproduce steps:
   1) Setup Solr to run on HDFS like this:
  
   java -Dsolr.directoryFactory=HdfsDirectoryFactory
   -Dsolr.lock.type=hdfs
   -Dsolr.hdfs.home=hdfs://host:port/path
  
   For the purpose of this testing, turn off the default auto commit in
   solrconfig.xml, i.e. comment out autoCommit like this:
  
  
   2) Add a document without commit:
   curl http://localhost:8983/solr/collection1/update?commit=false; -H
   Content-type:text/xml; charset=utf-8 --data-binary @solr.xml
  
   3) Solr generate empty tlog file (0 file size, the last one ends with
 6):
   [hadoop@hdtest042 exampledocs]$ hadoop fs -ls
   /path/collection1/core_node1/data/tlog
   Found 5 items
   -rw-r--r-- 1 hadoop hadoop 667 2014-07-18 08:47
   /path/collection1/core_node1/data/tlog/tlog.001
   -rw-r--r-- 1 hadoop hadoop 67 2014-07-18 08:47
   /path/collection1/core_node1/data/tlog/tlog.003
   -rw-r--r-- 1 hadoop hadoop 667 2014-07-18 08:47
   /path/collection1/core_node1/data/tlog/tlog.004
   -rw-r--r-- 1 hadoop hadoop 0 2014-07-18 09:02
   /path/collection1/core_node1/data/tlog/tlog.005
   -rw-r--r-- 1 hadoop hadoop 0 2014-07-18 09:02
   /path/collection1/core_node1/data/tlog/tlog.006
  
   4) Simulate Solr crash by killing the process with -9 option.
  
   5) restart the Solr process. Observation is that uncommitted document
 are
   not replayed, files in tlog directory are cleaned up. Hence uncommitted
   document(s) is lost.
  
   Am I missing anything or this is a bug?
  
   BTW, additional observations:
   a) If in step 4) Solr is stopped gracefully (i.e. without -9 option),
   non-empty tlog file is geneated and after re-starting Solr, uncommitted
   document is replayed as expected.
  
   b) If Solr doesn't run on HDFS (i.e. on local file system), this issue
 is
   not observed either.
  
   Thanks,
   Tom
  
 


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SolrCloud on HDFS empty tlog hence doesn't replay after Solr process crash and restart

2014-07-21 Thread Tom Chen 
Any thought about this issue: Solr on HDFS generate empty tlog when add
documents without commit.

Thanks,
Tom


On Fri, Jul 18, 2014 at 12:21 PM, Tom Chen tomchen1...@gmail.com wrote:

 Hi,

 This seems a bug for Solr running on HDFS.

 Reproduce steps:
 1) Setup Solr to run on HDFS like this:

 java -Dsolr.directoryFactory=HdfsDirectoryFactory
  -Dsolr.lock.type=hdfs
  -Dsolr.hdfs.home=hdfs://host:port/path

 For the purpose of this testing, turn off the default auto commit in
 solrconfig.xml, i.e. comment out autoCommit like this:
 !--
 autoCommit
maxTime${solr.autoCommit.maxTime:15000}/maxTime
openSearcherfalse/openSearcher
  /autoCommit
 --

 2) Add a document without commit:
 curl http://localhost:8983/solr/collection1/update?commit=false; -H
 Content-type:text/xml; charset=utf-8 --data-binary @solr.xml

 3) Solr generate empty tlog file (0 file size, the last one ends with 6):
 [hadoop@hdtest042 exampledocs]$ hadoop fs -ls
 /path/collection1/core_node1/data/tlog
 Found 5 items
 -rw-r--r--   1 hadoop hadoop667 2014-07-18 08:47
 /path/collection1/core_node1/data/tlog/tlog.001
 -rw-r--r--   1 hadoop hadoop 67 2014-07-18 08:47
 /path/collection1/core_node1/data/tlog/tlog.003
 -rw-r--r--   1 hadoop hadoop667 2014-07-18 08:47
 /path/collection1/core_node1/data/tlog/tlog.004
 -rw-r--r--   1 hadoop hadoop  0 2014-07-18 09:02
 /path/collection1/core_node1/data/tlog/tlog.005
 -rw-r--r--   1 hadoop hadoop  0 2014-07-18 09:02
 /path/collection1/core_node1/data/tlog/tlog.006

 4) Simulate Solr crash by killing the process with -9 option.

 5) restart the Solr process. Observation is that uncommitted document are
 not replayed, files in tlog directory are cleaned up. Hence uncommitted
 document(s) is lost.

 Am I missing anything or this is a bug?

 BTW, additional observations:
 a) If in step 4) Solr is stopped gracefully (i.e. without -9 option),
 non-empty tlog file is geneated and after re-starting Solr, uncommitted
 document is replayed as expected.

 b) If Solr doesn't run on HDFS (i.e. on local file system), this issue is
 not observed either.

 Thanks,
 Tom

SolrCloud on HDFS empty tlog hence doesn't replay after Solr process crash and restart

2014-07-18 Thread Tom Chen 
Hi,

This seems a bug for Solr running on HDFS.

Reproduce steps:
1) Setup Solr to run on HDFS like this:

java -Dsolr.directoryFactory=HdfsDirectoryFactory
 -Dsolr.lock.type=hdfs
 -Dsolr.hdfs.home=hdfs://host:port/path

For the purpose of this testing, turn off the default auto commit in
solrconfig.xml, i.e. comment out autoCommit like this:
!--
autoCommit
   maxTime${solr.autoCommit.maxTime:15000}/maxTime
   openSearcherfalse/openSearcher
 /autoCommit
--

2) Add a document without commit:
curl http://localhost:8983/solr/collection1/update?commit=false; -H
Content-type:text/xml; charset=utf-8 --data-binary @solr.xml

3) Solr generate empty tlog file (0 file size, the last one ends with 6):
[hadoop@hdtest042 exampledocs]$ hadoop fs -ls
/path/collection1/core_node1/data/tlog
Found 5 items
-rw-r--r--   1 hadoop hadoop667 2014-07-18 08:47
/path/collection1/core_node1/data/tlog/tlog.001
-rw-r--r--   1 hadoop hadoop 67 2014-07-18 08:47
/path/collection1/core_node1/data/tlog/tlog.003
-rw-r--r--   1 hadoop hadoop667 2014-07-18 08:47
/path/collection1/core_node1/data/tlog/tlog.004
-rw-r--r--   1 hadoop hadoop  0 2014-07-18 09:02
/path/collection1/core_node1/data/tlog/tlog.005
-rw-r--r--   1 hadoop hadoop  0 2014-07-18 09:02
/path/collection1/core_node1/data/tlog/tlog.006

4) Simulate Solr crash by killing the process with -9 option.

5) restart the Solr process. Observation is that uncommitted document are
not replayed, files in tlog directory are cleaned up. Hence uncommitted
document(s) is lost.

Am I missing anything or this is a bug?

BTW, additional observations:
a) If in step 4) Solr is stopped gracefully (i.e. without -9 option),
non-empty tlog file is geneated and after re-starting Solr, uncommitted
document is replayed as expected.

b) If Solr doesn't run on HDFS (i.e. on local file system), this issue is
not observed either.

Thanks,
Tom

Solr and Maven

2014-07-04 Thread Tom Chen 
Hi,

The default tool to build Solr is ant ( plus ivy), while Maven support is
provided.

Regarding building with Maven,  some questions:

1) Is there any difference between the build created by ant and that
created by Maven?
2) Any plan for Solr to use Maven as the default building tool?

Regards,
Tom

Solr and Kerberos

2014-06-10 Thread Tom Chen 
Hi,

I wonder how to secure Solr with Kerberos.

We can Kerberos secure Solr by configuring the AuthenticationFilter from
the hadoop-auth.jar that is packaged in solr.war in the web.xml

But after we do that,

1) How does a SolrJ client connect to the secured Solr server?
2) In SolrCloud environment, how one Solr node connect to other secured
Solr node?

Thanks,
Tom

Test case HardAutoCommitTest.testCommitWithin fail

2014-05-26 Thread Tom Chen 
 was: ?xml version=1.0 encoding=UTF-8?
response
lst name=responseHeaderint name=status0/intint
name=QTime2528/int/lstresult name=response numFound=1
start=0docint name=id529/intint
name=range_facet_si529/intarr
name=range_facet_llong529/long/arrarr
name=range_facet_sllong529/long/arrarr name=field_tstrwhat's
inside?/str/arrarr name=subjectstrinfo/str/arrlong
name=_version_1468909952844169216/longdate
name=timestamp2014-05-23T16:14:55.142Z/dateint
name=intDefault42/intarr
name=multiDefaultstrmuLti-Default/str/arr/doc/result
/response

  request was:rows=20qt=standardq=id:529start=0version=2.2
   at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:689)
   ... 41 more



Any idea?

Tom Chen