On 13/12/2016 18:05, Gupta, Rajiv wrote:
After I create directory by myself I'm getting this error:
Which directory do you try to create? I wouldn't try to make manual changes inside Lucy's index directory. This will only make things worse.
Can't open '/u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1/seg_fd/lexicon-7.ixix': Invalid argument 20161211 182109 [] * LUCY_FSFolder_Local_Open_FileHandle_IMP at core/Lucy/Store/FSFolder.c line 118 20161211 182109 [] * LUCY_Folder_Local_Open_In_IMP at core/Lucy/Store/Folder.c line 101 20161211 182109 [] * LUCY_Folder_Open_In_IMP at core/Lucy/Store/Folder.c line 75 There are two more failures they also failed due so similar reasons rename from '/u/smoke/presub/logs/cit-fg-adr-ndo-rtp.rajivg.1481473039.49384_cmode_1of1/.lucyindex/1/schema.temp' to '/u/smoke/presub/logs/cit-fg-adr-ndo-rtp.rajivg.1481473039.49384_cmode_1of1/.lucyindex/1/schema_e4.json' failed: No such file or directory Can't delete 'lexicon-3.ix' I believe all three are related to race condition while doing parallel indexing and should go away with retries. However my retries started failing with different error which is strange to me as if directory already exists shouldn't it skip from create attempt. 20161211 182109 [] * FAIL: [FAILED]: Retrying to add doc at path: /u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1 : Couldn't create directory '/u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1': No such file or directory 20161211 182109 [] * LUCY_FSFolder_Initialize_IMP at core/Lucy/Store/FSFolder.c line 102 So my all retry attempts were also failed.
These errors still look like multiple processes are modifying the index at the same time. Are you really sure that every indexer is created with an IndexManager and that every IndexManager is created with a `host` argument that is unique to each machine?
All these errors mean that there's something fundamentally wrong with your code or that you hit a bug in Lucy. The only type of error where it makes sense to retry is LockErr. All other errors are mostly fatal and could result in index corruption. Retrying will only mask an underlying problem in this case.
Unfortunately, I'm unable to help unless you provide some kind of self-contained, reproducible test case. I'm aware that this isn't easy, especially with multiple clients writing to a shared volume.
As I already hinted at, you might want to reconsider your architecture and use some kind of search server that uses an index on a local filesystem. There are ready-made platforms on top of Lucy like Dezi, but it isn't too hard to roll your own solution. This should result in better performance and makes behavior of your code more predictable.
Nick
