!ret.isUsable
On September 4, 2017 at 11:23:09, Otto Fowler ([email protected])
wrote:
> It looks like if the SourceHandler has a problem with it’s output stream,
> it will never recover.
> The handler will be in the map and continue to be used, but it will
> continue to throw exceptions.
>
> Is there a reason why we don’t try to recover and recreate the
> SourceHandler, such as:
>
> synchronized SourceHandler getSourceHandler(String sourceType, String
> stellarResult, WriterConfiguration config) throws IOException {
> SourceHandlerKey key = new SourceHandlerKey(sourceType, stellarResult);
> SourceHandler ret = sourceHandlerMap.get(key);
>
> // BEGIN FAKE CODE PART
> if(ret != null && !ret.isUsableOrIsOpenOrSomething()) {
> try {
> ret.close();
> } catch(Exception e){
> // NO LOGGER IN THIS CLASS
> }
> sourceHandlerMap.remove(key);
> ret = null;
> }
> // END FAKE CODE PART
>
> if(ret == null) {
> if(sourceHandlerMap.size() >= maxOpenFiles) {
> throw new IllegalStateException("Too many HDFS files open!");
> }
> ret = new SourceHandler(rotationActions,
> rotationPolicy,
> syncPolicyCreator.create(sourceType, config),
> new
> PathExtensionFileNameFormat(key.getStellarResult(), fileNameFormat),
> new SourceHandlerCallback(sourceHandlerMap, key));
> sourceHandlerMap.put(key, ret);
> }
> return ret;
> }
>
>
>
> It seems strange for something that is supposed to be a long long running
> process, writing to a distributed network store.
>
