Dear Wiki user, You have subscribed to a wiki page or wiki category on "Tika Wiki" for change notification.
The "BaysianMimeTypeSelector" page has been changed by Lukeliush: https://wiki.apache.org/tika/BaysianMimeTypeSelector?action=diff&rev1=4&rev2=5 The following shows the initial conditional probability values used in the default setting of feature in Tika. + {{{ - // conditional probability: probability of the type estimated by the Magic test given that the type is the type predicted by the magic test. + /* conditional probability: probability of the type estimated by the Magic test given that the type is the type predicted by the magic test. */ private static final float DEFAULT_MAGIC_TRUST = 0.9f; + }}} + {{{ + /* conditional probability: probability of the type estimated by the metadata hint test given that the type is the type predicted by the metadata hint test.*/ private static final float DEFAULT_META_TRUST = 0.8f; + }}} + {{{ + /* conditional probability: probability of the type estimated by the extension test given that the type is the type predicted by the extension test.*/ private static final float DEFAULT_EXTENSION_TRUST = 0.8f; + }}} + {{{ - - private static final float DEFAULT_MAGIC_TRUST = 0.9f; - - // conditional probability: probability of the type estimated by the metadata hint test given that the type is the type predicted by the metadata hint test. - - private static final float DEFAULT_META_TRUST = 0.8f; - - // conditional probability: probability of the type estimated by the extension test given that the type is the type predicted by the extension test. - - . private static final float DEFAULT_EXTENSION_TRUST = 0.8f; - - // conditional probability: probability of the type Not estimated by the Magic test given that the type is not the type predicted by the magic test. + /* conditional probability: probability of the type Not estimated by the Magic test given that the type is not the type predicted by the magic test.*/ 0.9f + }}} + {{{ - - 0.9f - - // conditional probability: probability of the type Not estimated by the meadata test given that the type is not the type predicted by the metadata test. + /* conditional probability: probability of the type Not estimated by the meadata test given that the type is not the type predicted by the metadata test.*/ 0.8f + }}} + {{{ - - 0.8f - - // conditional probability: probability of the type Not estimated by the extension test given that the type is not the type predicted by the extension test. + /* conditional probability: probability of the type Not estimated by the extension test given that the type is not the type predicted by the extension test.*/ 0.8f + }}} - - 0.8f - {{https://lh4.googleusercontent.com/wUrj6keimRl2xjZjxutax0jXyhIbHBkOtNZTwLoHmU2q_VFq_fVxLBP1YRv29teUZdbdvXunj7xiTtC3GLtCErsg_yz2NaftfVO9YF9rmdD18QRgaHoYJ21k-kEb2WrOw6NST8U||height="381px;",width="563px;"}} The above tells that the file extension and contentMetadata matching method fail to detect a type (i.e. they both return byte-stream as the type), and only the magic test returns a non-byte-stream type, then the type estimated by magic test has a higher posterior probability 90%; @@ -66, +60 @@ Kindly note, usually the metadata hint is present in the http response header, if we bluntly or directly give a file to the feature in Tika, the meta data hint usually is absent. + ---------- - ---------------------------------------------------------------------------------------------------------- - How to use this feature in Tika (proposed use) If a user wanted to use this feature, the following code would be needed. @@ -90, +83 @@ } }); }}} - The idea is simple that we overwrite the getDefaultDetector() by providing the DefaultProbDetector which extends the CompositeDetector, a CompositeDetector is one (whose supertype is Detector) that takes a list of detectors, and when its detect() method gets called, each detector in the list is called sequentially one after another. The original implemenation of getDefaultDetector() in TikaConfig returns an instance of “DefaultDetector” that also extends the CompositeDetector by providing a list of detectors that includes “MimeTypes” which is the native implemenation with 3 detectors (i.e. magic bytes, extension and metadatahint). However, DefaultProbDetector replaces this MimeTypes with ProbabilisticMimeDetectionSelector. + . The idea is simple that we overwrite the getDefaultDetector() by providing the DefaultProbDetector which extends the CompositeDetector, a CompositeDetector is one (whose supertype is Detector) that takes a list of detectors, and when its detect() method gets called, each detector in the list is called sequentially one after another. The original implemenation of getDefaultDetector() in TikaConfig returns an instance of “DefaultDetector” that also extends the CompositeDetector by providing a list of detectors that includes “MimeTypes” which is the native implemenation with 3 detectors (i.e. magic bytes, extension and metadatahint). However, DefaultProbDetector replaces this MimeTypes with ProbabilisticMimeDetectionSelector. In order to set the preferential weights, an instance of ProbabilisticMimeDetectionSelector can be created in the above example snippet. Alternatively, if we dont want to go with the default settings with ProbabilisticMimeDetectionSelector, it is ok to just ignore the arguments by calling only “return new DefaultProbDetector();”. + - Alternatively, if we dont want to write some extra code, the following can also be used. + . Alternatively, if we dont want to write some extra code, the following can also be used. {{{ /* @@ -109, +103 @@ }}} '''TIKA-detector-sample.xml''' - <?xml version="1.0" encoding="UTF-8"?> + <?xml version="1.0" encoding="UTF-8"?> <!-- + + . Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. + - <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - --> - <properties> + --> <properties> + - <detectors> + . <detectors> - <detector class="org.apache.tika.detect.DefaultProbDetector"/> + . <detector class="org.apache.tika.detect.DefaultProbDetector"/> - </detectors> + </detectors> + </properties>
