[IGFS][Semaphore] Write file concurrently from several nodes

[cristic83]

Hi guys,

We have the following usecase involving IGFS: we have an IGFS configured for
our cluster and we want that each time a node in the cluster is started a
file (describing the operations provided by the node) is written in the IGFS
by that node. If several nodes provide the same operation, the same file is
to be written in IGFS. The nodes can overwrite the file that is written.
Although this seemed to be quite a simple task, we've got stuck into several
problems that we couldn't figure out, due to the fact that one node can't
flush the data, while the other can't open the file to write as it has
already been opened for writing by another node.

We have implemented the writing to the IGFS like this in a component called
FileManager:

  @Override
    public void uploadFile(FileKey fileKey, byte[] content) {
       IgfsPath filePath = ...
        try (IgfsOutputStream igfsOutputStream = fileSystem.create(filePath,
true)){ //line 42: overwrite the file if it already exists
            igfsOutputStream.write(content);
        } catch (IOException e) { //line 44
            throw new RuntimeException("Failed to write file to the file
system", e);
        }
    }

Then, in each node we have the code performing the write uses a distributed
semaphore with a count of 1, to ensure that only one node at a time can
perform the write:

IgniteSemaphore journeySema =  ignite.semaphore(journeyNameAndVersion, 1,
true,  true);
journeySema.acquire();
 try{
                     igfsDocManager.uploadFile(fileDescriptor);
 } finally {
                     journeySema.release();

  }
  });

The problem is that every time we start the cluster we get exceptions on
both nodes of the cluster: one of the nodes can't perform the flushing and
throws the exception: 

Caused by: java.io.IOException: File was concurrently deleted:
/doc/change-settings/1.0/documentation.yaml

    at
org.apache.ignite.internal.processors.igfs.IgfsOutputStreamImpl.flush(IgfsOutputStreamImpl.java:283)

    at
org.apache.ignite.internal.processors.igfs.IgfsOutputStreamAdapter.close(IgfsOutputStreamAdapter.java:182)

    at
solutions.theglue.base.common.igfs.IgfsManager.uploadFile(IgfsManager.java:44)

while the other node complains that it cannot overwrite the file during
creation because the file is already opened for writing:

Exception in thread "main" class org.apache.ignite.igfs.IgfsException:
Failed to overwrite file (file is opened for writing)
[fileName=documentation.yaml,
fileId=deb08dc9551-184c9036-ab49-4ff4-845e-c74da751bfa3,
lockId=beb08dc9551-e9edb720-fd7a-4df5-b2fd-ff79acdf30eb]

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

    at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

    at
org.apache.ignite.internal.processors.igfs.IgfsUtils.newIgfsException(IgfsUtils.java:100)

    at
org.apache.ignite.internal.processors.igfs.IgfsUtils.toIgfsException(IgfsUtils.java:75)

    at
org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1940)

    at
org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:997)

    at
org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:958)

    at
solutions.theglue.base.common.igfs.IgfsManager.uploadFile(IgfsManager.java:42)

The filesystem is configured in PRIMARY mode, like below:

  @Bean
    @Qualifier(IGFS_CACHE)
    public CacheConfiguration<Object, Object> igfsCacheConfiguration() {
        CacheConfiguration<Object, Object> igfsCacheConfiguration = new
CacheConfiguration<>();
        igfsCacheConfiguration.setName(IGFS_META);
        igfsCacheConfiguration.setCacheMode(CacheMode.REPLICATED);
       
igfsCacheConfiguration.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
        return igfsCacheConfiguration;
    }

    @Bean
    @Qualifier(IGFS_DATA_CACHE)
    public CacheConfiguration<Object, Object> igfsDataCacheConfiguration() {
        CacheConfiguration<Object, Object> igfsDataCacheConfiguration = new
CacheConfiguration<>();
        igfsDataCacheConfiguration.setName(IGFS_DATA);
       
igfsDataCacheConfiguration.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
        igfsDataCacheConfiguration.setAffinityMapper(new
IgfsGroupDataBlocksKeyMapper(DEFAULT_GRP_SIZE));
        return igfsDataCacheConfiguration;
    }

    @Bean
    public FileSystemConfiguration
fileSystemConfiguration(@Qualifier(IGFS_BEAN_NAME) String igfsName) {
        FileSystemConfiguration fileSystemConfiguration = new
FileSystemConfiguration();
        fileSystemConfiguration.setName(igfsName);
        fileSystemConfiguration.setMetaCacheName(IGFS_META);
        fileSystemConfiguration.setDataCacheName(IGFS_DATA);
        fileSystemConfiguration.setDefaultMode(IgfsMode.PRIMARY);
        return fileSystemConfiguration;
    }

The version of ignite that we use is 1.5.9.

Any idea what might be wrong?

Thanks for your support,
Cristi




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/IGFS-Semaphore-Write-file-concurrently-from-several-nodes-tp6009.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.