Looking at the other field types I have come up with this IP field type.
I am not sure what needs to be done to include it into the project
though. Are there any other files that need to be modified to add the
field type into the core or anything?
This field type should convert IP addresses into a long which can then
be searched with a query specifying a range of IPs. Example...
IP:[192.168.0.1 TO 192.168.0.255]
Thanks,
Colton McInroy
* Director of Security Engineering
Phone
(Toll Free)
_US_ (888)-818-1344 Press 2
_UK_ 0-800-635-0551 Press 2
My Extension 101
24/7 Support [email protected] <mailto:[email protected]>
Email [email protected] <mailto:[email protected]>
Website http://www.dosarrest.com
On 10/21/2013 5:43 PM, Aaron McCurry wrote:
Possibly, I will add an issue to take a look at it.
https://issues.apache.org/jira/browse/BLUR-283
On Mon, Oct 21, 2013 at 3:53 PM, Colton McInroy <[email protected]>wrote:
Oh... strict is still showing up as false, even with -s -p strictTypes
true and adding a first mutate.
Is this a bug?
Thanks,
Colton McInroy
* Director of Security Engineering
Phone
(Toll Free)
_US_ (888)-818-1344 Press 2
_UK_ 0-800-635-0551 Press 2
My Extension 101
24/7 Support [email protected] <mailto:[email protected]>
Email [email protected] <mailto:[email protected]>
Website http://www.dosarrest.com
On 10/21/2013 12:50 PM, Colton McInroy wrote:
I just tested, yes, that does appear to be the case...
blur (default)> mutate Program_sshd 1 1 event Date:20131013124823
blur (default)> schema Program_sshd
Schema(table:Program_sshd,
families:{event={Date=**ColumnDefinition(family:event,
columnName:Date, subColumnName:null, fieldLessIndexed:false,
fieldType:date, properties:{dateFormat=**yyyyMMddHHmmss})}})
Program_sshd
family : event
column : Date
fieldType : date
prop : dateFormat=yyyyMMddHHmmss
Makes it kind of hard to look at the current schema before committing to
start pushing data into it.
Thanks,
Colton McInroy
* Director of Security Engineering
Phone
(Toll Free)
_US_ (888)-818-1344 Press 2
_UK_ 0-800-635-0551 Press 2
My Extension 101
24/7 Support [email protected] <mailto:[email protected]>
Email [email protected] <mailto:[email protected]>
Website http://www.dosarrest.com
On 10/21/2013 12:28 PM, Aaron McCurry wrote:
Yes it might. I will take a look this evening.
On Mon, Oct 21, 2013 at 3:20 PM, Colton McInroy <[email protected]
wrote:
Hmmm... Is this expected? Does the schema only show up once I have added
something? Also, strictTypes doesn't appear to be getting changed to
true.
blur (default)> create -c 10 -l file:///tmp/blur/tables/****Program_sshd
-t
Program_sshd -s
blur (default)> definecolumn Program_sshd event Date date -p dateFormat
yyyyMMddHHmmss
blur (default)> schema Program_sshd
Schema(table:Program_sshd, families:{})
Program_sshd
blur (default)> describe Program_sshd
cluster : default
name : Program_sshd
enabled : true
tableUri : file:///tmp/blur/tables/****
Program_sshd
shardCount : 10
readOnly : false
columnPreCache : null
blockCaching : true
blockCachingFileTypes : null
tableProperties : null
strictTypes : false
defaultMissingFieldType : text
defaultMissingFieldLessIndexin****g : true
defaultMissingFieldProps : null
blur (default)> disable Program_sshd
blur (default)> remove Program_sshd
blur (default)> create -c 10 -l file:///tmp/blur/tables/****Program_sshd
-t
Program_sshd -s -p strictTypes true
blur (default)> describe Program_sshd
cluster : default
name : Program_sshd
enabled : true
tableUri : file:///tmp/blur/tables/****
Program_sshd
shardCount : 10
readOnly : false
columnPreCache : null
blockCaching : true
blockCachingFileTypes : null
tableProperties : null
strictTypes : false
defaultMissingFieldType : text
defaultMissingFieldLessIndexin****g : true
defaultMissingFieldProps : null
blur (default)> definecolumn Program_sshd event Date date -p dateFormat
yyyyMMddHHmmss
blur (default)> schema Program_sshd
Schema(table:Program_sshd, families:{})
Program_sshd
blur (default)>
Thanks,
Colton McInroy
* Director of Security Engineering
Phone
(Toll Free)
_US_ (888)-818-1344 Press 2
_UK_ 0-800-635-0551 Press 2
My Extension 101
24/7 Support [email protected] <mailto:[email protected]>
Email [email protected] <mailto:[email protected]>
Website http://www.dosarrest.com
On 10/21/2013 12:08 PM, Colton McInroy wrote:
Thanks,
Colton McInroy
* Director of Security Engineering
Phone
(Toll Free)
_US_ (888)-818-1344 Press 2
_UK_ 0-800-635-0551 Press 2
My Extension 101
24/7 Support [email protected] <mailto:[email protected]>
Email [email protected] <mailto:[email protected]>
Website http://www.dosarrest.com
On 10/21/2013 11:41 AM, Aaron McCurry wrote:
Yes, however with the API changes that have/are being discussed in
others
threads (Document vs. Record, Document Collection vs.Row, etc) I want
to
change the value portion of the Column to have a Value type that will
be
a
union in Thrift instead of a struct. This would allow us to have all
the
basic types be defined in separate fields. This stringValue for
string
types, textValue for text types, intValue for int types, etc that way
when
a table is not in strict mode it could better guess the correct type
instead of blindly choosing text.
Makes sense, for my purposes this should work well enough though. I
will
want strictTypes=true and define column types for the column I know
about,
and let automatic determination take care of the unspecified fields,
although that shouldn't be very many, since I should know all field
types
as I define them in my interface. Kinda sucks that they cannot be
changed
though, I'll have to build that into my GUI and make sure to take note
of
that.
Ok so this code is to merely add the type to be available as a type
that
can be used. After this runs your new type will act just like "text",
"int", "date", or any other built in type. After the type is
registered
in
the system, either by table or system wide you will still need to call
definecolumn to make use of the new type.
So in the example.
tableDescriptor.****putToTableProperties("blur.****fieldtype.customfield1",
"org.apache.blur.analysis.****type.ExampleType1");
"blur.fieldtype." is the important part for the loader. The prefix
tells
the TableContext that this property is a new field type. So it takes
the
value "org.apache.blur.analysis.****type.ExampleType1" for example
and
tries to
load the class via the Class.forName method. If successfully and if
it's a
FieldTypeDefinition it will register is in the BaseFieldManager. Then
the
type is available for use.
So "customfield1" is not even used. It's only there to makes the
property
be a unique name.
Hope this helps.
Ahh... *sings* I can see clearly now the sun is out *sings*, I was
misunderstanding what you meant first, that makes a lot more sense,
Thanks.
/**
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.blur.analysis.type;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.blur.thrift.generated.Blur;
import org.apache.blur.thrift.generated.Column;
import org.apache.hadoop.conf.Configuration;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.document.LongField;
import org.apache.lucene.document.StoredField;
import org.apache.lucene.document.StringField;
import org.apache.lucene.document.Field.Store;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.NumericRangeQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
public class IPFieldTypeDefinition extends NumericFieldTypeDefinition {
public static final String NAME = "ip";
private FieldType _typeNotStored = LongField.TYPE_NOT_STORED;
@Override
public String getName() {
return NAME;
}
private long getLongIP(String ip) {
try {
byte[] value = InetAddress.getByName(ip).getAddress();
final long longIP = ((value[0] & 0xFFl) << (3*8)) +
((value[1] & 0xFFl) << (2*8)) + ((value[2] & 0xFFl) << (1*8)) + (value[3] &
0xFFl);
return longIP;
} catch (UnknownHostException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return 0;
}
@Override
public void configure(String fieldNameForThisInstance, Map<String, String>
properties, Configuration configuration) {
}
@Override
public Iterable<? extends Field> getFieldsForColumn(String family, Column
column) {
String name = getName(family, column.getName());
List<Field> fields = new ArrayList<Field>();
StoredField storedField = new StoredField(name, column.getValue());
long longIP = getLongIP(column.getValue());
LongField field = new LongField(name, longIP, _typeNotStored);
fields.add(field);
fields.add(storedField);
return fields;
}
@Override
public Iterable<? extends Field> getFieldsForSubColumn(String family, Column
column, String subName) {
String name = getName(family, column.getName(), subName);
long longIP = getLongIP(column.getValue());
return makeIterable(new LongField(name, longIP, _typeNotStored));
}
@Override
public Query getNewRangeQuery(String field, String part1, String part2,
boolean startInclusive, boolean endInclusive) {
long p1 = getLongIP(part1);
long p2 = getLongIP(part2);
return NumericRangeQuery.newLongRange(field, _precisionStep, p1, p2,
startInclusive, endInclusive);
}
}
